This came as a surprise to me. While I knew we needed mutexes (and other good multi-threading practices) in Python, I also vaguely believed the GIL helped the user with their threading in some way. It turns out, it doesn’t!
Also, I realize that there are a dozen different definitions of “data race”. I prefer the one in the Rustonomicon, but YMMV!
I’ve wasted so much time trying to explain to people in the past just that (a big one being telling people that they shouldn’t be using collections.deque for communication across threads but rather queue.Queue and company). The GIL is an implementation detail, and is there only to make the interpreter threadsafe, not Python code itself.
If it makes you feel any better, you’ve written a great example of why people still have to do proper concurrency control, and I’ll be saving it to send to people in the future.
Oof, I never thought about it before, but it makes sense that python can have data races. I’m assuming the GIL locks atomically on each instruction so if you interleave load, change, and store instructions in different threads it’s possible to end up with races. I don’t generally write multithreaded code that way in Python, but I wonder how many bugs I’ve created since I didn’t realizing this :(
I’m assuming the GIL locks atomically on each instruction
I just realized an exception to this: a lot of instructions that look primitive can actually call back into Python code (for example BINARY_ADD can call an __add__ method) so those must yield the GIL in the same way a CALL_FUNCTION instruction would. So, whether an instruction is atomic can depend on what data is passed in.
In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via sys.setswitchinterval(). Each bytecode instruction and therefore all the C implementation code reached from each instruction is therefore atomic from the point of view of a Python program.
For those of us who don’t have the Python bytecode model memorized, the standard package dis can provide some insight into what is actually atomic. Although I don’t believe there are any guarantees that the details of bytecode compilation won’t change between versions.
Gotta say I love how this blog is laid out and the way the notes sections worked. Very nice, though looking at the source it looks a bit futzy to get right if I wanted to use it for my own writing.
This came as a surprise to me. While I knew we needed mutexes (and other good multi-threading practices) in Python, I also vaguely believed the GIL helped the user with their threading in some way. It turns out, it doesn’t!
Also, I realize that there are a dozen different definitions of “data race”. I prefer the one in the Rustonomicon, but YMMV!
I’ve wasted so much time trying to explain to people in the past just that (a big one being telling people that they shouldn’t be using
collections.dequefor communication across threads but ratherqueue.Queueand company). The GIL is an implementation detail, and is there only to make the interpreter threadsafe, not Python code itself.If it makes you feel any better, you’ve written a great example of why people still have to do proper concurrency control, and I’ll be saving it to send to people in the future.
Oof, I never thought about it before, but it makes sense that python can have data races. I’m assuming the GIL locks atomically on each instruction so if you interleave load, change, and store instructions in different threads it’s possible to end up with races. I don’t generally write multithreaded code that way in Python, but I wonder how many bugs I’ve created since I didn’t realizing this :(
I just realized an exception to this: a lot of instructions that look primitive can actually call back into Python code (for example
BINARY_ADDcan call an__add__method) so those must yield the GIL in the same way aCALL_FUNCTIONinstruction would. So, whether an instruction is atomic can depend on what data is passed in.That appears to be the case. The official Python FAQ has a question, What kinds of global value mutation are thread-safe?, with the answer:
For those of us who don’t have the Python bytecode model memorized, the standard package dis can provide some insight into what is actually atomic. Although I don’t believe there are any guarantees that the details of bytecode compilation won’t change between versions.
Gotta say I love how this blog is laid out and the way the notes sections worked. Very nice, though looking at the source it looks a bit futzy to get right if I wanted to use it for my own writing.
BTW do you have an RSS feed for this blog?