1. 30

  2. 8

    A Viable Solution for Python Concurrency is the LWN story on this topic.

    1. 6

      @pushcx, could you please merge this with Python Multithreading without the GIL, an earlier story on this topic?

      1. 3

        Very good comments on this article.

        The big question I have is: if all the complexity is in making reference counting thread safe, why not abandon reference counting? Alternatively as something of this magnitude will already break most extensions, why not introduce explicit ownership semantics with its own language support to allow sharing between threads?

        1. 4

          Removing reference counting would break every C extension out there. Most of Python’s value is in its libraries, many of which have extensive native code managed as C extensions.

        2. 1

          Without in any way diminishing this, it seems like reference counting instead of other gc or memory management is a big part of the problem here. Alternative approaches seem worth investigation given that even this would be a breaking change to Python.

          1. 1

            It is such an amazing accomplishment. Think of how many times this has been talked about, and even tried a few times. But always to the tune of “It is impossible to remove the GIL”. Then Sam Gross comes along and very modestly just does it.

            1. 2

              The biggest different is that people previously wanted to make Python 100% thread-safe by default; it was never impossible to remove the GIL, it’s just that the semantics that were used led to serious performance regressions. See e.g. this (talking about the patchset for Python 1.5):

              Yah. That’s about right. The largest problem was dealing with dictionaries. Since you never knew whether a specific dictionary was shared (between threads) or not, you always had to lock the access. And since all namespaces use dictionaries…

              This approach is a bit different, where dictionaries just aren’t locked and can have race conditions, similar to e.g. Go.

              1. 1

                I saw the nogil term like few month or years back. But it seems didn’t get the popularity due to some reasons. the gil is never a blocker for me, since I’ve been constantly remodel the code to avoid gil or rewrite critical parts in other languages.

                1. 2

                  I think that’s how most works around the GIL: rethink, remodel, rewrite, etc. Imagine that was not necessary, but could be accomplished in Python :-)