scala - Move file from local to HDFS -

-

my environment uses spark, pig , hive.

i having trouble write code in scala (or other language compatible environment) copy file local file system hdfs.

does have advices on how should proceed?

you write scala job using hadoop filesystem api.
, use ioutils apache commons copy data inputstream outputstream

import org.apache.hadoop.conf.configuration; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path;  import org.apache.commons.io.ioutils;    val hadoopconf = new configuration(); val fs = filesystem.get(hadoopconf);  //create output stream hdfs file val outfilestream = fs.create(new path("hedf://<namenode>:<port>/<filename>))  //create input stream local file val instream = fs.open(new path("file://<input_file>"))  ioutils.copy(instream, outfilestream)  //close both files instream.close() outfilestream.close()

Comments

Post a Comment

Popular posts from this blog

symfony - InvalidConfigurationException in SecurityExtension.php line 429: No authentication listener registered for firewall "secured_area" -

-
i'm new symfony user , follow tutorial . current task add logout script unfortunately gives me error: invalidconfigurationexception in securityextension.php line 429: no authentication listener registered firewall "secured_area". my security.yml file looks this: security: # http://symfony.com/doc/current/book/security.html#where-do-users-come-from-user-providers providers: in_memory: memory: users: ryan: password: $2a$12$lcy0mefviec3typhv9snnuzofyr2p/axigoqjeds4am4jwhnz/jli roles: 'role_user' admin: password: $2a$12$cytwee9kpq1pjqkfiwuzfucrpwvyazwm4xzmz1qpufl7/flcm3v0g roles: 'role_admin' firewalls: secured_area: # ... logout: path: /logout target: / # disables authentication assets , prof...
Read more

angular ui router - 10 digest iterations reached in angularjs when using $state.go -

-
i using angular ionic. i not want user have login again in case logged in , did not logout last time used application. reason, using local storage check if user logged in. tried putting code in run block. $rootscope.$on('$statechangestart', function (event, tostate, toparams, fromstate, fromparams) { if($localstorage.userinfo){ $state.go('deployment'); //this line 270 of app.js event.preventdefault(); return; } }) but gives error says: rangeerror: maximum call stack size exceeded @ scope.$broadcast (ionic.bundle.js:23414) @ object.transitionto (ionic.bundle.js:40804) @ object.go (ionic.bundle.js:40671) @ app.js:270 @ scope.$broadcast (ionic.bundle.js:23412) @ object.transitionto (ionic.bundle.js:40804) @ object.go (ionic.bundle.js:40671) @ app.js:270 @ scope.$broadcast (ionic.bundle.js:23412) @ object.transitionto (ionic.bundle.js:40804) also, error: [$rootscope...
Read more

javascript - Handling constructor related functions while testing with mocha and sinon -

-
i testing application in declare specific variable within constructor , assign return of function imported class. how code looks like: var someclass; function constructor() { this.someclasshandler = null; } constructor.prototype.newsomeclass = function() { someclass = require('./someclass.js'); return new someclass(); } constructor.prototype.getfunction = function() { this.someclasshandler = this.newsomeclass(); } constructor.prototype.callsomefunction = function() { this.someclasshandler.somefunction(); } when try test code whether function 'somefunction' imported class 'someclass' called within method'callsomefunction' receive failed test responses variable 'someclasshandler' still equals null (which makes sense not call method 'getfunction' assigns class variable). i use mocha , sinon tests, bit helpless whether mock or stub class in order assign variable 'this.someclasshandler' correctly. the m...
Read more