scala - Move file from local to HDFS -

-

my environment uses spark, pig , hive.

i having trouble write code in scala (or other language compatible environment) copy file local file system hdfs.

does have advices on how should proceed?

you write scala job using hadoop filesystem api.
, use ioutils apache commons copy data inputstream outputstream

import org.apache.hadoop.conf.configuration; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path;  import org.apache.commons.io.ioutils;    val hadoopconf = new configuration(); val fs = filesystem.get(hadoopconf);  //create output stream hdfs file val outfilestream = fs.create(new path("hedf://<namenode>:<port>/<filename>))  //create input stream local file val instream = fs.open(new path("file://<input_file>"))  ioutils.copy(instream, outfilestream)  //close both files instream.close() outfilestream.close()

Comments

Post a Comment

Popular posts from this blog

mysql - FireDac error 314 - but DLLs are in program directory -

-
i getting when trying access mysql database : [firedac][phys][mysql]-314. cannot load vendor library [libmysql.dll or libmysqlld.dll] this did not happen (unchanged) code, however, have upgraded windows 10 , had reinstall delphi xe8, system configuration matter. in order try solve problem, copied both of files c:\windows\sysytem32. when did not seem work, copied them \win32\debug, generated .exe resides. i imagine doing rather stupid, can't see what. the proper solution place driver file (eg., libmysql.dll ) in application's folder, or place installation location in fddrivers.ini file: [mysql] vendor=<folder>\libmysql.dll (recent versions of documentation seem use vendorlib instead of vendor in ini file.) see rad studio documentation topics configuring drivers (firedac) , connect mysql server (firedac) more information.
Read more

git - How to list all releases of public repository with GitHub API V3 -

-
so per documentation https://developer.github.com/v3/repos/releases/#list-releases-for-a-repository get /repos/:owner/:repo/releases should list relases , https://api.github.com/repos/jquery/jquery/releases should list releases in jquery project , not , why ? because repository doesn't have releases. has tags github presenting in releases page. for more clear example, see: https://github.com/hashicorp/terraform/releases which has both releases , tags showing on page, api shows releases: https://api.github.com/repos/hashicorp/terraform/releases
Read more

c++ - Getting C2512 "no default constructor" for `ClassA` error on the first parentheses of constructor for `ClassB`? -

-
my situation follows. let there header file classdeclarations.h , have following classes declared: class classa; class classb; class classa { public: classa(int x); classa(double y); int returninteger(void) { return m_x; } double returndouble(void) { return m_y; } private: int m_x; double m_y; } class classb { public: classb(int x); int returninteger(void) { return m_x; } private: int m_x; classa m_instancea_1; } let there *.cpp file called classdefns.cpp , define methods of classes declared in classdeclarations.h : #include classdeclarations.h classa::classa(int x) : m_x(x) { } classa::classa(double y) : m_y(y) { } classb::classb(int x) : m_x(x) { // line build error occurs m_instancea_1 = classa(0); } note have commented line compiler says there error: error c2512: 'classa' : no appropriate default constructor avai...
Read more