python multiprocessing logging: QueueHandler with RotatingFileHandler "file being used by another process" error -

i'm converting program multiprocessing , need able log single rotating log main process subprocesses. i'm trying use 2nd example in python cookbook logging single file multiple processes, starts logger_thread running part of main process, picking log messages off queue subprocesses add to. example works is, , works if switch rotatingfilehandler.

however if change start logger_thread before subprocesses (so can log main process well), log rotates, subsequent logging generates traceback windowserror: [error 32] process cannot access file because being used process.

in other words change code 2nd example

workers = [] in range(5):     wp = process(target=worker_process, name='worker %d' % (i + 1), args=(q,))     workers.append(wp)     wp.start() logging.config.dictconfig(d) lp = threading.thread(target=logger_thread, args=(q,)) lp.start() 

to this:

logging.config.dictconfig(d) lp = threading.thread(target=logger_thread, args=(q,)) lp.start() workers = [] in range(5):     wp = process(target=worker_process, name='worker %d' % (i + 1), args=(q,))     workers.append(wp)     wp.start() 

and swap out logging.filehandler logging.handlers.rotatingfilehandler (with small maxbytes testing) , hit error.

i'm using windows , python 2.7. queuehandler not part of stdlib til python 3.2 i've copied source code gist, says safe do.

i don't understand why starting listener first make difference, nor understand why process other main attempting access file.

you should never start any threads before subprocesses. when python forks, threads , ipc state not copied properly.

there several resources on this, google fork , threads. people claim can it, it's not clear me can ever work properly.

just start processes first.

example additional information:

status of mixing multiprocessing , threading in python

https://stackoverflow.com/a/6079669/4279

in case, might copied open file handle problem, still should start subprocesses before threads (and before open files later want destroy).

some rules of thumb, summarized fantabolous comments:

• subprocesses must started before threads created same process.

• multiprocessing.pool creates both subprocesses , threads, 1 mustn't create additional processes or pools after first one.

• files should not open @ time process or pool created. (this ok in cases, not, e.g. if file deleted later.)

• subprocesses can create own threads , processes, same rules above applying.

• starting processes first easiest way this