scala - Spark driver disassociated and removed by the master -

-

i have cluster made 2 slaves , 1 master , set , submit jar (scala) spark master (192.168.1.64):

spark-submit --master spark://spark-master:7077 --class tests.elements target/scala-2.10/zzz-project_2.10-1.0.jar

after quite sometime running fine stops abruptly last lines on terminal being

... 15/08/19 17:45:24 info scheduler.taskschedulerimpl: adding task set 411292.0 6 tasks 15/08/19 17:45:24 warn scheduler.tasksetmanager: stage 411292 contains task of large size (2762 kb). maximum recommended task size 100 kb. 15/08/19 17:45:24 info scheduler.tasksetmanager: starting task 2.0 in stage 411292.0 (tid 1832, 192.168.1.64, process_local, 2828792 bytes) 15/08/19 17:45:24 info scheduler.tasksetmanager: starting task 0.0 in stage 411292.0 (tid 1833, 192.168.1.62, process_local, 2310009 bytes) 15/08/19 17:45:24 info scheduler.tasksetmanager: starting task 3.0 in stage 411292.0 (tid 1834, 192.168.1.64, process_local, 2669188 bytes) 15/08/19 17:45:24 info scheduler.tasksetmanager: starting task 1.0 in stage 411292.0 (tid 1835, 192.168.1.62, process_local, 2295676 bytes) 15/08/19 17:45:24 info scheduler.tasksetmanager: starting task 4.0 in stage 411292.0 (tid 1836, 192.168.1.64, process_local, 2847786 bytes) 15/08/19 17:45:24 info scheduler.tasksetmanager: starting task 5.0 in stage 411292.0 (tid 1837, 192.168.1.64, process_local, 2913528 bytes) killed

and error occurring @ master log following:

... 15/08/19 16:09:49 info master.master: launching executor app-20150819160949-0001/0 on worker worker-20150819160925-192.168.1.64-51640 15/08/19 16:09:49 info master.master: launching executor app-20150819160949-0001/1 on worker worker-20150819160938-192.168.1.62-38007 15/08/19 16:15:44 info master.master: akka.tcp://sparkdriver@192.168.1.64:46823 got disassociated, removing it. 15/08/19 16:15:44 info master.master: removing app app-20150819160949-0001 15/08/19 16:15:44 warn remote.reliabledeliverysupervisor: association remote system [akka.tcp://sparkdriver@192.168.1.64:46823] has failed, address gated [5000] ms. reason is: [disassociated]. 15/08/19 16:15:44 warn master.master: application testpagerank still in progress, may terminated abnormally. ...

both workers have in logs this

... 15/08/19 16:15:49 info worker.worker: executor app-20150819160949-0001/0 finished state exited message command exited code 1 exitstatus 1 15/08/19 16:15:50 warn remote.reliabledeliverysupervisor: association remote system [akka.tcp://sparkexecutor@192.168.1.64:54799] has failed, address gated [5000] ms. reason is: [disassociated].

and 

... 15/08/19 16:15:43 info worker.worker: executor app-20150819160949-0001/1 finished state exited message command exited code 1 exitstatus 1 15/08/19 16:15:43 warn remote.reliabledeliverysupervisor: association remote system [akka.tcp://sparkexecutor@192.168.1.62:53325] has failed, address gated [5000] ms. reason is: [disassociated].

respectively. work/app files contain this

... 15/08/19 16:15:41 info executor.executor: finished task 1.0 in stage 387758.0 (tid 1803). 1911 bytes result sent driver 15/08/19 16:15:41 info executor.executor: finished task 4.0 in stage 387758.0 (tid 1806). 1911 bytes result sent driver 15/08/19 16:15:41 info storage.blockmanager: found block rdd_1206_5 locally 15/08/19 16:15:41 info executor.executor: finished task 5.0 in stage 387758.0 (tid 1807). 1911 bytes result sent driver 15/08/19 16:15:41 info storage.blockmanager: found block rdd_1206_3 locally 15/08/19 16:15:41 info executor.executor: finished task 3.0 in stage 387758.0 (tid 1805). 1911 bytes result sent driver 15/08/19 16:15:44 error executor.coarsegrainedexecutorbackend: driver 192.168.1.64:46823 disassociated! shutting down. 15/08/19 16:15:44 warn remote.reliabledeliverysupervisor: association remote system [akka.tcp://sparkdriver@192.168.1.64:46823] has failed, address gated [5000] ms. reason is: [disassociated]. 15/08/19 16:15:45 info storage.diskblockmanager: shutdown hook called 15/08/19 16:15:46 info util.utils: shutdown hook called

and

... 15/08/19 16:15:41 info storage.blockmanager: found block rdd_1206_0 locally 15/08/19 16:15:41 info executor.executor: finished task 2.0 in stage 387758.0 (tid 1804). 1911 bytes result sent driver 15/08/19 16:15:41 info executor.executor: finished task 0.0 in stage 387758.0 (tid 1802). 1911 bytes result sent driver 15/08/19 16:15:42 error executor.coarsegrainedexecutorbackend: driver 192.168.1.64:46823 disassociated! shutting down. 15/08/19 16:15:42 info storage.diskblockmanager: shutdown hook called 15/08/19 16:15:42 warn remote.reliabledeliverysupervisor: association remote system [akka.tcp://sparkdriver@192.168.1.64:46823] has failed, address gated [5000] ms. reason is: [disassociated]. 15/08/19 16:15:42 info util.utils: shutdown hook called

respectively. there seem no other error in hdfs or spark.

i suspecting error lies in master log, third line (15/08/19 16:15:44 info master.master: akka.tcp://sparkdriver@192.168.1.64:46823 got disassociated, removing it.) can't figure out why. tried changing spark.akka.heartbeat.interval 100 suggested in posts no luck. know why happens , how solve this? much.

as mentioned in similar question here warn reliabledeliverysupervisor: association remote system has failed, address gated [5000] ms. reason: [disassociated]

the problem lack of memory. adding more memory (or in case more nodes) should solve problem.

(alternately, needing less memory should work of course).


Comments

Post a Comment

Popular posts from this blog

html - Firefox flex bug applied to buttons? -

-
i'm trying apply flex button on firefox i'm having problem on chrome doesn't happens. this css: .button { display: inline-flex; align-items: center; } as can see in fiddle, "caret" word moved new line, happens if element <button> tag, <div> works expected. http://codepen.io/anon/pen/xblmgx is firefox bug or there css i'm missing? nevermind, looks "mr firefox" doesn't idea... <button> not implementable (by browsers) in pure css, bit of black box, perspective of css. means don't react in same way e.g. <div> would. this isn't specific flexbox -- e.g. don't render scrollbars if put "overflow:scroll" on button, , don't render table if put "display:table" on it. stepping further, isn't specific <button> . consider <fieldset> , <table> have special rendering behavior: data:text/html,<fieldset style="d...
Read more

html - Missing border-right in select on Firefox -

-
Image
when put select div width , padding-left, options have no border on right. more, happend if select have width 118px , parent have 15px of padding left! have idea, going on? my code: <!doctype html> <html> <head lang="en"> </head> <body> <div style="width: 1000px; padding-left: 15px;"> <select style="width: 118px;"> <option>a</option> <option>b</option> <option>c</option> </select> </div> </body> </html> and result: here plunker: click it happend on firefox on windows. tested on 40.0.2 version -webkit-box-sizing: border-box; -moz-box-sizing: border-box; -ms-box-sizing: border-box; box-sizing: border-box; source: https://stackoverflow.com/a/11390432/4431269 source of problem: https://bugzilla.mozilla.org/show_bug.cgi?id=924068 and use of google ! edit: (source of problem, explained) after rea...
Read more

python - build a suggestions list using fuzzywuzzy -

-
i building suggestion engine database. i'm using fuzzywuzzy module match user input string existing database shown below. if request.method == "get": display_dict = {} user_input = request.get["user_input"] # user input match_dict = {} food_items_obj = food_items.objects.all() # fetch objects table items in food_items_obj : match = fuzz.ratio(user_input.lower(),items.name.lower()) # try match user input existing names match_dict[match] = items.name # put 1 dictionary matched_dict = ordereddict(sorted(match_dict.items(),reverse = true)) # sort dictionary if max(matched_dict.keys()) == 100: # if exact match put in dict , return display_dict[100] = matched_dict[100] else : # if not found try best matching words.. ###########this part need help! ############# k,v in matched_dict.items(): if user_input in v : display_dict[k]...
Read more