I think a lot of the work around removing the GIL from various languages has more to do with not feeling cool rather than any actual technical problem. Sure, there are problems one cannot solve in Python because it has a GIL but those probably aren’t problems one should be solving in Python. Ocaml is working hard on multicore support, which sounds nice but it’ll probably have a much more complex memory model, so is it worth the pain? Like you said, I could use shared memory and processes. I’d know more about it but, honestly, I haven’t hit a reason yet!
I’d imagine that the GIL is destined to be more of a problem on massively multicore systems: say, 1024 cores. At 4-16, you can probably use multiprocessing and a Pool and be fine, assuming that you can get near-100% utilization of one core per Process. I don’t know if the same is true when you get to “embarassingly parallel” problems on machines with a lot of cores.
Multi process doesn’t preclude the use of shared memory. I think Linus was on to something with “a thread is just a process that shares more”. This doesn’t mean that processes are prohibited from sharing too. The nice thing about running 1000 python processes is they can all run their GC in parallel, too. (Assuming the shared data is not itself made of python objects. PyObject is, in imho, a poor boxing format for the kinds of data one analyzes with 1000s of cores. But I’m not the expert in such areas.)
At this point (about ten years late imo), I’d say don’t bother. People have learned to use async or multi processes techniques as appropriate.
Every once in a while I ask myself why I can’t use parallel threads in lua, then I ignore the problem and it goes away.
I do think go has some nice answers here, but it was designed up to support a certain programming model.
I think a lot of the work around removing the GIL from various languages has more to do with not feeling cool rather than any actual technical problem. Sure, there are problems one cannot solve in Python because it has a GIL but those probably aren’t problems one should be solving in Python. Ocaml is working hard on multicore support, which sounds nice but it’ll probably have a much more complex memory model, so is it worth the pain? Like you said, I could use shared memory and processes. I’d know more about it but, honestly, I haven’t hit a reason yet!
I’d imagine that the GIL is destined to be more of a problem on massively multicore systems: say, 1024 cores. At 4-16, you can probably use multiprocessing and a Pool and be fine, assuming that you can get near-100% utilization of one core per Process. I don’t know if the same is true when you get to “embarassingly parallel” problems on machines with a lot of cores.
Multi process doesn’t preclude the use of shared memory. I think Linus was on to something with “a thread is just a process that shares more”. This doesn’t mean that processes are prohibited from sharing too. The nice thing about running 1000 python processes is they can all run their GC in parallel, too. (Assuming the shared data is not itself made of python objects. PyObject is, in imho, a poor boxing format for the kinds of data one analyzes with 1000s of cores. But I’m not the expert in such areas.)
This just in: OpenBSD core contributor to jump ship to Linux!