I believe the complicated nature between fork(), threading, etc. in general, not to mention throwing Python’s various multiprocessing/threading modules on top, is a large reason why the Celery project have their fork of multiprocessing called billiard.
Prior to billiard a large codebase and Celery using project I was on would experience all kinds of issues around fork()ing due to the COW nature of it and all the copied mutexes, thread handles, etc. Thankfully billiard resolved nearly all of those and the rest we were able to work around with more resilient code and monitoring to detect when it looked like things were going to go sideways.
Never heard of billiard before, looks like a ripe place to mine unknown fixes for forking quirks. I guess it’s preaching to the choir on lobste.rs, but here is another long yet hopelessly incomplete list of reasons you should avoid fork like the plague. There’s a reason newer OS designs like NT never publicly supported it.
I believe the complicated nature between fork(), threading, etc. in general, not to mention throwing Python’s various multiprocessing/threading modules on top, is a large reason why the Celery project have their fork of multiprocessing called billiard.
Prior to billiard a large codebase and Celery using project I was on would experience all kinds of issues around fork()ing due to the COW nature of it and all the copied mutexes, thread handles, etc. Thankfully billiard resolved nearly all of those and the rest we were able to work around with more resilient code and monitoring to detect when it looked like things were going to go sideways.
billiard looks really interesting, thanks!
Posix is quite clear that after calling fork in a threaded program, the only thing you’re allowed to do next is call execve.
Never heard of billiard before, looks like a ripe place to mine unknown fixes for forking quirks. I guess it’s preaching to the choir on lobste.rs, but here is another long yet hopelessly incomplete list of reasons you should avoid fork like the plague. There’s a reason newer OS designs like NT never publicly supported it.
I think that’s an abuse of
Pool. The point of it is to avoid forking off a process for every task.The
whileloop is just to make the race condition happen consistently; in real code you wouldn’t loop, so you’d only get the deadlock intermittently.