Oh, this is lovely writing. People who expect ‘Python 2 is dead’ articles to be uninteresting: don’t skip this article: it’s a thoughtful and insightful essay about about Python’s languages, communities, and history, not just about a version cutover. Warmly recommmended.
A bit late to the party, but I just thought that the people and organizations who are migrating away from Python altogether are not doing that because of the backwards-incompatible changes introduced by Python 3. As has been pointed out time and time again, migrating (or porting, or whatever other term you prefer) from Python 2 to 3 involves less effort than a full reimplementation in a different language.
At the same time, a number of CPython developers have expressed that had they known how much friction this would introduce to the entire ecosystem, such a big break in compatibility would not have happened. In other words, people have learnt their lesson, and it’s extremely unlikely that a similar situation will happen to Python again. Any insinuations about not being able to trust Python with regards to forwards compatibility are more likely to be rooted in frustration than fact.
I sincerely hope that most of those who have decided to abandon Python are doing it for better reasons than out of spite, because that is not exactly a rational factor to base such a decision on. Sure, the friction involved in the upgrade has likely acted as a catalyst, a contributing circumstance, but I don’t see how it could be rationally considered the single most important factor in something like that.
This is probably a topic for another post, but I have a theory that in at least some organizations, a lateral move – adopting a new language/platform and rewriting existing code in it – can be sold to management as enabling something they care about (specific features, claims about performance, whatever) and thus offers a way to actually get to do all the maintenance the organization has deferred for years as part of the rewrite.
I appreciate the time that you took to write this post. I know that the Python community, including myself, can be aggressive and unpleasant in online discussions.
The PyPy project, for example, is currently dependent on a Python 2 interpreter to bootstrap and so will be maintaining their own either for as long as PyPy exists, or for as long as it takes to migrate to bootstrapping on Python 3 (which they seem to think is either not feasible, or not something they want to do).
This quote is emblematic of the contempt that you display towards Python users. You imply with your words here that porting RPython to Python 3 is “feasible”, but that ignores the cost of labor. Multiple projects have found that Python 3 porting is too expensive to tackle without dedicated funding and support, including not just Django and PyPy, but also Twisted and entire distros like Ubuntu and Gentoo; if those projects cannot find funding then they do not do the porting.
I don’t understand your denigration of interpreters. Producing and maintaining an interpreter is non-trivial, a language community should strive to have a heterogeneous collection of interpreters and compilers besides the reference implementation, and the big list of Python 3 libraries pledging to drop Python 2.7 is missing not just PyPy, but also Jython and IronPython. While some implementations, like MicroPython have switched exclusively to Python 3, others have not.
To paraphrase somebody else’s insight from an IRC discussion, perhaps, as with Perl and Raku, it is the case that Python’s community became too ideologically burdened, and the fork is the result of an ideology operating without constraints. After all, my observations are only weighty because of the inertia of code running on computers, and perhaps that is the only argument in favor of Python 2: That it is the way that things are being done, regardless of the loudness of the claims of Python 3 proponents. I certainly am not here to praise Python 2 either, but to bury it; in fact, were the language still in one piece, I would only have to dig one tomb and write one eulogy. I need to credit glyph for being the one to first paraphrase Marc Antony, over a decade ago at PyCon; I was so slow to understand what everybody else was trying to point out.
Though there is one thing I think gets overlooked a lot: usually, the anti-Python-3 argument is presented as the desire of a particular company, or project, or person, to stand still and buck the trend of the world to be ever-changing.
But really they’re asking for the inverse of that. Rather than being a fixed point in a constantly-changing world, what they really seem to want is to be the only ones still moving in a world that has become static around them.
I am not asking for either of these positions. First, let me dispel this idea as you’ve worded it. I understand that a language is not fixed in stone, and that it will constantly change and evolve with usage. Indeed, I believe that languages are expressive media, and that the main purpose of language is to send messages and communicate, but since our understanding of the world changes over time, our languages must change as well. (Also, explicitly, I am not wearing any corporate hats. I have many commit bits, but the only relevant plan I currently have is to remove Python entirely from all active codebases, incrementally and immediately.)
My argument, in ten words or less, is: Okay, yes, but must we have changed “2” to “3”? The massive amount of backporting done so far has revealed that no, most of the new features in Python 3 could have been put into Python 2. People had already, as you pointed out, survived having to change with to be reserved and then a keyword, and that change was far more intrusive than anything in the 2-to-3 changelist. Meanwhile, the modules that ought to have been killed off directly, most notably asyncore, were left in Python 3 despite being effectively unmaintained and long since rotting.
And I can say that, having overseen the 2 to 3 conversion for a large adtech codebase and maintained 2/3 compatible libraries. I did that because it was both challenging and required really creative, often perverse, solutions. And this was with my mental model very carefully keeping track of string/binary boundaries. It wasn’t easy and people who balanced both interpreters and didn’t collapse under the devilry of duplicate code (and subsequent bug duplication) have my admiration.
If I could move a stodgy adtech company to 3 for active projects, then he can as well.
I’m not a super coder, just someone who adds asserts and routinely retests her assumptions. If you’re not on 3, it’s either a big ball of mud that should’ve been incrementally rewritten/rearchitected (thus exposing bad design) or you expected an ecosystem to stay in stasis forever. If you’re the latter, compile from source and lockdown your environments more than ever cause you’re on your own.
Already I’ve witnessed removal of python 2 from distributions! It’s getting killed regardless of how anyone feels. Digging ones heels in is just silly.
You should work on your mind-reading. My complaint is mostly that, for over a decade, PyPy developers have been consistently marginalized by the PSF. This forced deprecation of Python 2 is merely one more brick in the wall. While I am fortunate to be independently wealthy, and I also barely contribute to PyPy, the typical PyPy developer is a hard-working academic who is trying to dramatically speed up the Python ecosystem, and the PSF’s continued snub of their work is disappointing, especially given their continued endorsement of corporate-sponsored JITs hacked on top of CPython, like Unladen Swallow, Pyston, and Pyjion.
I’ve overseen 2-to-3 conversions as well. I’ve seen companies that can do it fine, companies that are ready with 2to3, and companies that are totally hopeless. I’ve seen companies that are so behind on security patches that they cannot upgrade away from Python 2, and companies that greenfield with Python 3. It is hard but possible. The main sticking point, which folks seem to constantly paper over, is that Python 2 and Python 3 are fundamentally different langauges, with different grammar and semantics. Indeed, PSF representatives have explicitly asked me to not talk about Python 2 and Python 3 as if they are different, and when I’ve asked followup questions, they’ve indicated that it is because treating the two languages as different undermines the porting effort. This is how the system works.
It is funny that you mention “compile from source;” I use Nix, and so of course there will be CPython and PyPy for Python 2.7 available for a very long time via nixpkgs. Again, I am independently configured, and I am not a part of any core development team. But I am not “on my own;” I am one of a community which maintains the ports tree.
Python 2 and Python 3 are fundamentally different langauges, with different grammar and semantics
No they’re not. The existence of 2/3 compatible packages utterly shits on your point.
The only “breaking” change that you seem to be whining about as soooo different is the splitting of a primitive type. That’s it. Splitting a fucking type into two. That’s not a fundamental language difference and you know it. You’ve made backwards incompatible changes in your code bases and most likely have split classes and refactored. Tell me, was your program utterly grammatically different? Probably not.
You didn’t whine about long/int merging because it’s a trivial alias to add.
But splitting a type is always a lot more contentious and difficult (probably verging on the biggest type of breaking change possible).
Removing “print” so that it could be substituted with a function hardly counts as a massive grammer change. And the addition of coroutines is strictly an “add on”.
I’m not going to even bother with your “mother loved you best” vis a vis PyPy. Jython has been effectively dead for years (unless they finally got invokedynamic support to actually get usable speeds - I last checked two years ago, so sue me).
I don’t know about the others as they’ve been more like toys than have capability of supporting the ecosystem as-is. You can’t just demand equal buy in without supporting the ecosystem as-is - PyPy literally had to make c-extensions work after claiming for years it was impossible to do (which was absurd - I remember reading one of the JVM Ruby interpreters ended up supporting c-extensions through parsing the c-extension source code into compatible libraries - if ruby could do it, so can python).
Your points withstanding (I should be less overtly crude), the GP’s routine use of “contempt” as an accusative is nothing short of insulting. He paints himself as the victim of the PSF and anyone who agrees with them as doing their bidding. It is extremely unkind and quite conversation terminating.
I get that you feel you’re being fair, but it comes across as uneven enforcement amidst the unfair characterizations and portrayal of an anti-Python 2 conspiracy, when in reality it’s an interpreter hide-bound by Guido’s rules to keep it as simple as possible (thus hamstringing backwards compatibility like seen in the JVM), a hold over from the early anemic days of Python and something that’ll never change.
The existence of polyglot libraries does not “shit[] on my point;” it merely shows how far people can be convinced to go in pursuit of a satisfying end-user experience. The existence of libraries which compile under both a C compiler and also a C++, Objective-C, Zig, or etc. compiler does not invalidate the point that the languages are different, regardless of whether the program texts are identical in those cases.
Here are two short interactions with two CPython versions. I contend that they implement distinct grammars and semantics from each other.
>>> {1:2,3:4}.items()
[(1, 2), (3, 4)]
>>> {1:2,3:4}.iteritems()
<dictionary-itemiterator object at 0x7f704a55f170>
>>> nonlocal = 42
>>> class Foo: pass
...
>>> Foo.__mro__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: class Foo has no attribute '__mro__'
And the other one.
>>> {1:2,3:4}.items()
dict_items([(1, 2), (3, 4)])
>>> {1:2,3:4}.iteritems()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'iteritems'
>>> nonlocal = 42
File "<stdin>", line 1
nonlocal = 42
^
SyntaxError: invalid syntax
>>> class Foo: pass
...
>>> Foo.__mro__
(<class '__main__.Foo'>, <class 'object'>)
I am not sure why you characterize my points as “whining” and then agree with them by substantiating the needful evidence. It seems that you are angry with me because I have points, not because I don’t have points.
By using a broad brush to label various Python projects as “toys” or “dead”, you only emphasize the culture of contempt which I highlighted in my thread-starting comment. Moreover, you’ve said the quiet part out loud: The reason for this contemptuousness is a tribal belief in the power of C extensions over plain old Python code. What’s remarkable to me is that this belief isn’t actually in the service of speed or compatibility; the PyPy project produced not just PyPy, but also CFFI, in order to put those claims to rest. Instead, the belief seems to be simply one of majority rule: Because CPython is the popular standard, CPython must remain the popular standard. To quote from The Power and The Glory:
And though the hard times are really due to me, it is still in me to quell all this unrest
Things must stay, there must be no change; anyway, time to rearrange
You must believe me, that there’s been no such betrayal, all that I’ve done, I’ve done for you
Things must stay, there must be no change; anyway, time to rearrange
There must be a reason why plans have turned around! Not only in me, it must be in you too
Things must stay, there must be no change; anyway, time to rearrange
Uh, no. With the exception of nonlocal as a keyword (which btw is no different when with as a keyword was introduced), all the examples cited are properties on data objects, which is categorically not “grammar”.
Grammar pertains to parsing the language into an syntactically valid AST from which a bytecode suitable for the VM can run on. The only incompatible grammatical changes I can think of off the top of my head are:
with (python 2.4)
print to print() (Python 3)
nonlocal (Python 3) (similar to global except restricted to scopes excluding global)
That’s it.
Write a proxy for the builtins and you can support older method returns without a problem. That’s why six was did it’s job so well.
By using a broad brush to label various Python projects as “toys” or “dead”
Does it support the ecosystem? If not, then not very comparable with CPython in now, isn’t it?
As to dead, Jython still doesn’t do invokedynamic and that’s pretty critical to decent performance. Python is well known to be pretty slow. Jython is even slower. That heavily constrains use case to “we have to run a Python script on the JVM” and that’s about it. And it’s also stuck on 2. So it’s left behind in time to a shrinking ecosystem of pure python 2 modules. Talk about useful 🙄
Moreover, you’ve said the quiet part out loud: The reason for this contemptuousness is a tribal belief in the power of C extensions over plain old Python code
Actually, I didn’t. You made that up all by yourself honey.
I only cited the support of C-Extensions as part of recognizing the audience demand for being able to use PyPy in a broader case than just pure Python and rare/underused one shot ports of C libraries and C-compatible library exports (did several for previous employers) that typically used C-extensions/Cython.
You’re so wrapped up in inventing enemies that heap contempt on you, but it’s just fellow engineers raising their eyebrows at someone being overly dramatic. Lol contempt. 😂😂😂
I need to go update it for 3.8 final and things coming in 3.9, but I maintain a list of ways to “break” Python, geared at library authors who want to trigger syntax or import errors as quickly as possible when someone tries to use one of their libraries on an unsupported Python version.
It also serves as a sort of tour of changes to Python syntax over the past decade.
I don’t have any real opinion on python2/3 myself. However, if the language you use forces you to make changes which you disagree with, then it is not unreasonable to at least consider the other options out there. Maybe it’s simple enough for them to port from python 2 to 3, but will they then have to port from 3 to 4 in the future? Maybe porting from 2 to 3 isn’t a massive deal, but they weren’t super happy with python for other reasons, and it was the straw that broke the camel’s back.
It is more or less twenty years since discussion started about possible breaking changes that eventually became Python3 and 14 years since “official” start. We are not even at the first step for Python4. Based on how often most people change jobs and projects in this industry, it is very unlikely they will need to port to 4 or even worry about what that will entail for a completely imaginary version of the language. They are likely to go through far more impactful transitions before then.
These days I mostly write Python 3 and on occasion still Python 2. I find the notion that they are different languages absurd.
Yes, and it would also be less work to use Python 3 than to write an entirely new language. However, we have reasons, and so we produced Monte, a modern flavor of E. We were already leaving Python, and we were already rewriting our Python-based virtual machine in Monte.
This quote is emblematic of the contempt that you display towards Python users. You imply with your words here that porting RPython to Python 3 is “feasible”, but that ignores the cost of labor. Multiple projects have found that Python 3 porting is too expensive to tackle without dedicated funding and support, including not just Django and PyPy, but also Twisted and entire distros like Ubuntu and Gentoo; if those projects cannot find funding then they do not do the porting.
I’d like to think I know a thing or two about large projects porting, and about funding situations.
(for the uninitiated: I got my commit bit on Django in 2007, was its release manager from then up to about 2014, served multiple terms on the framework’s technical board, and am currently in my fifth year on the board of the DSF, the nonprofit foundation set up to support Django and its community, including by fundraising for, and then funding, its development)
Also, hi. I am a Python user, and I’d seriously dispute your assertion that I have “contempt” for Python users.
I don’t understand your denigration of interpreters. Producing and maintaining an interpreter is non-trivial, a language community should strive to have a heterogeneous collection of interpreters and compilers besides the reference implementation, and the big list of Python 3 libraries pledging to drop Python 2.7 is missing not just PyPy, but also Jython and IronPython. While some implementations, like MicroPython have switched exclusively to Python 3, others have not.
I don’t understand why you interpret it as “denigration”. You’ve repeatedly indicated your dislike for the Python core team’s and the PSF’s messaging that Python 2 is no longer supported, and you’ve argued that Python 2 is supported for as long as alternative Python 2 interpreters are supported. My point is that while a supported interpreter is a necessary thing for calling Python 2 “supported”, it is not sufficient. When people say they want to keep using Python 2, they mean they also want to keep using all their favorite packages and libraries, and many of those simply do not run on Python 2 anymore. And as far as I know, of non-CPython implementations still supporting Python-2-the-language, only PyPy has ever been robust enough to support some of the really important packages like NumPy/SciPy.
Anyway, this is why I bring up the question of whether you’re willing to commit to also maintaining a complete Python 2 ecosystem including the big third-party packages. The Linux distros who’ll be maintaining Python 2 interpreters for years to come have committed to that, but as far as I can tell nobody else has. And without the ecosystem I don’t think you have a sufficient “Python 2” and as I’ve told you directly and also mentioned in the post, I think you create unnecessary and unwanted burdens on those packages’ maintainers if you insist on saying “Python 2 is still supported”.
The massive amount of backporting done so far has revealed that no, most of the new features in Python 3 could have been put into Python 2.
OK, but now make a Python 2 that cleanly implements the string change and the reorganization of the builtins and the standard library and doesn’t create a massive backwards-compatibility gap to older Python 2 versions. I don’t think that’s possible, and I do think that at least the string change is an absolutely necessary thing for Python’s future.
The Python Packaging Authority (PyPA) is looking to get rid of Python 2 support, at which point I suppose you’ll shift even more of the burden to the remaining Python 2 users somehow, despite the singular package index still being squarely under PSF control.
The rest of your post fails to address my point. If you can’t stop talking past me, then please don’t bother. You can’t even look at a Python 2 user without the insinuation that they must be responsible for the entirety of their ecosystem.
Frankly, I think that you show your hand when you say “really important packages like NumPy/SciPy.” That’s the direction that you want Python to go in.
at which point I suppose you’ll shift even more of the burden to the remaining Python 2 users somehow
Well, I’m not trying to be mean about this, but: this is what the “entitlement” section of the post was about. The PyPA folks don’t owe you a Python-2-compatible packaging toolchain. The Django team doesn’t owe you a Python-2-compatible version of Django. And so on.
If those teams have decided they only want to support Python 3, they have the right to do that! And if you want to keep using those tools and libraries and packages with Python 2, you can. You just have to choose between:
Use an older version of the project with no more bugfixes or security support, or
Form a group of like-minded people to do your own bugfixes and security support
Way way back in the early days of Django when the “magic-removal” effort rewrote the ORM (between Django 0.91 and 0.95), creating a big backwards-incompatible jump for projects to get over, I and some other people took the second option: even after the core Django team wasn’t supporting Django 0.91 anymore, we maintained a Django 0.91 branch for a while in order to gain more time to migrate. It’s open source and you can do that.
You can’t even look at a Python 2 user without the insinuation that they must be responsible for the entirety of their ecosystem.
I could just as easily flip this around and say you can’t look at someone who develops on Python 3 without insinuating they owe you their time and effort to also maintain a Python 2 version of whatever they build.
When you say “Python 2 is still supported”, you seem content to mean it in the specific narrow sense of “there is at least one group still maintaining a Python 2 interpreter”. What I’ve told you many times now is that when people hear “Python 2 is still supported”, what they will expect that to mean is a wider sense of popular third-party packages still maintaining Python 2 compatibility. I have concerns about that, because I maintain third-party packages and I’d rather not have a stream of people showing up in my bug trackers and getting angry because they heard “Python 2 is still supported” and thought that would mean I’ll maintain packages for them to use on Python 2.
Frankly, I think that you show your hand when you say “really important packages like NumPy/SciPy.” That’s the direction that you want Python to go in.
NumPy and SciPy are important packages. I don’t personally use them – as I’ve mentioned now multiple times in these discussions with you, I’m a Django guy, and my main use case for Python is as a backend web development lanaguage – but I know that there’s a large segment of the Python community who rely heavily on the numeric/scientific stack to do their work. I don’t see how anyone could look at the Python community and not come to that conclusion. So regardless of whether I use them or not, NumPy and SciPy are important packages. Just as Jupyter (née IPython) notebooks are important, even though I don’t personally use them. Just as the ML/AI packages are important even though I don’t use them. Just as Flask and SQLAlchemy are important packages, even though I don’t use them. Python’s continued success as a language comes from the large community of people using it for different things. The fact that there are large numbers of people using Python for not-my-use-case with not-the-libraries-I-use is a really good thing!
Anyway. You’re angry at what you see as the marginalization of alternative Python implementations, and especially the one you personally are most involved with (PyPy). That’s very clear (in fact, you’ve even said that pretty explicitly). But you’re so angry about it that you’re lashing out at random people on the internet, and either ignoring or misinterpreting what they say.
And on one level, I get it. The alternative implementation story has been rough! Django used to support IronPython and Jython, for example, and we treated it as a release blocker if we had compatibility issues with them. But at a certain point they started falling further and further behind CPython, and eventually it was clear that it would be a long time before they caught up, if ever. That’s sad – it used to be a great way to sneak Django into a Java or Windows shop – but I don’t believe Django has the resources to keep supporting such widely-diverged platforms long-term. If and when Jython and/or IronPython catch up to CPython (and both of them seem to have Python 3 compatibility as a goal), I’d be happy to push for Django to maintain compatibility with them again.
But on another level, you’re not really helping your case when you treat people as your enemy for no longer supporting Python 2 in their packages. If I didn’t already have a long history of knowing other PyPy people, for example, I’d be coming away with a pretty negative view of the project from my interactions with you.
This document is obsolete. Twisted has been ported to Python 3 (jettisonning a few unmaintained modules in the process). The next release will require Python 3, likely 3.5+.
Twisted did take about 15 million years to port, though, so, it does speak to how difficult porting can be. They did start a long time ago, and also have pretty much finished a while ago as well, so, anyone who hasn’t started should’ve kinda seen this (the end-of-life for Python 2) coming.
Yeah, it was a huge project, and particularly difficult because Twisted is exactly the sort of code that is most difficult to port: it sits at the divide between bytes and str. Something like a Django app is much easier to port — both because of the solid groundwork laid by Django itself, and also because the bulk of a web app’s logic is only concerned with text.
I do think that the OP undersells how unevenly the pain of the Python 3 transition has been distributed. It has been particularly rough for library maintainers, who often want to retain compatibility for a long time to enable their users to migrate. The end result has been a slow and painful transition with the corresponding toll on the Python community.
I would really like to see a blameless post-mortem of the Python 3 transition. Surely we can learn some lessons from this process that apply to development going forward.
I do think that the OP undersells how unevenly the pain of the Python 3 transition has been distributed. It has been particularly rough for library maintainers, who often want to retain compatibility for a long time to enable their users to migrate.
Assuming by “OP” you mean me (author of the blog post this whole thread is about), I don’t think I’m “underselling”. I was just trying to write about a bunch of different thoughts and perspectives that I think either haven’t been covered, or haven’t been covered enough, in the copious literature on the Python 2/3 transition.
Something like a Django app is much easier to port — both because of the solid groundwork laid by Django itself
This is sort of what I was getting at, though. I remember the days of Django’s “unicode branch” and how much work it took to get Django to a point where Django apps mostly didn’t have to worry about that stuff. As the saying goes, a lot of effort went into making that look effortless, and Django’s early experiences strongly influenced my later feelings about the right way to handle strings.
Assuming by “OP” you mean me (author of the blog post this whole thread is about), I don’t think I’m “underselling”.
Perhaps not the best word choice on my part.
I do think that the experience of making the transition varies a lot, and that may explain some of the heat in this thread and the discussion in general. We’re all frustrated by how things have dragged out.
As the saying goes, a lot of effort went into making that look effortless, and Django’s early experiences strongly influenced my later feelings about the right way to handle strings.
Yes, exactly! A post-mortem should focus on what Python could have done to make the transition work this way for everyone.
Sure. The links I used fail to actually tell the story that I wanted to tell. There was a point in time at which most of Twisted was Python 2 and 2to3 wasn’t capable of making up the difference. At that time, people often asked for Python 3 support. However, Python 3 support was only happening through paid work, and community members were not spending time on the porting effort. Eventually, as you say, many modules were simply removed entirely rather than being ported.
This is the process that I wanted to emphasize as happening repeatedly all over the Python ecosystem. Every library maintainer was asked to put in massive amount of effort, to the point where external funding was considered necessary to get things done. But this effort was only worthwhile because Python 3 supposedly was better than Python 2.
My argument, in ten words or less, is: Okay, yes, but must we have changed “2” to “3”?
I think that misses the point the author is trying to make. He argues that the maintainers of python ARE moving from 2 to 3. What we/you must do in this context is either assume the maintenance burden of a forked python 2 or migrate to python 3. You’re welcome to do neither as well, but of course any bugs related to your ecosystem will go unpatched.
People had already, as you pointed out, survived having to change with to be reserved and then a keyword, and that change was far more intrusive than anything in the 2-to-3 changelist.
Sorry but just to clarify: do you genuinely think that a keyword becoming reserved is a more intrusive change than switching to Unicode strings and explicit conversion to and from Unicode strings and byte sequences?
As you’ve phrased it, no. However, what you’ve described is not what actually happened. Python 2 supports the Unicode sandwich technique just as well as Python 3, and has the unicode type. I would rephrase your question as, “Do I genuinely think that a keyword becoming reserved is a more intrusive change than renaming unicode and str to str and bytes respectively?” And the answer is yes, I do.
So Python 2’s pain is deferred: you think your program is correct, and find out later that it fails with exotic characters.
With Python 3, your code fails immediately, so even if you are only handling ASCII, you have to explicitly deal with the difference between bytes and unicode.
I can’t believe that a “security professional” would countenance implicit, latent bugs like this on a fundamental data type. I’ve done the unicode sandwich for years and write implicitly statically typed python as a matter of practice (which JITs nicely in PyPy without having to wonder “will it blend?”).
Of the several hundred engineers I’ve met and the dozens I’ve worked with daily for years, I can count the number of people who are aware of unicode/str implicit mixing on one hand.
You’d honestly preserve a constant subtle footgun than explicitly expose the incompatible boundaries so they’re dealt with early on and not left as a landmine to appear years later? Wow
Oh, this is lovely writing. People who expect ‘Python 2 is dead’ articles to be uninteresting: don’t skip this article: it’s a thoughtful and insightful essay about about Python’s languages, communities, and history, not just about a version cutover. Warmly recommmended.
A bit late to the party, but I just thought that the people and organizations who are migrating away from Python altogether are not doing that because of the backwards-incompatible changes introduced by Python 3. As has been pointed out time and time again, migrating (or porting, or whatever other term you prefer) from Python 2 to 3 involves less effort than a full reimplementation in a different language.
At the same time, a number of CPython developers have expressed that had they known how much friction this would introduce to the entire ecosystem, such a big break in compatibility would not have happened. In other words, people have learnt their lesson, and it’s extremely unlikely that a similar situation will happen to Python again. Any insinuations about not being able to trust Python with regards to forwards compatibility are more likely to be rooted in frustration than fact.
I sincerely hope that most of those who have decided to abandon Python are doing it for better reasons than out of spite, because that is not exactly a rational factor to base such a decision on. Sure, the friction involved in the upgrade has likely acted as a catalyst, a contributing circumstance, but I don’t see how it could be rationally considered the single most important factor in something like that.
This is probably a topic for another post, but I have a theory that in at least some organizations, a lateral move – adopting a new language/platform and rewriting existing code in it – can be sold to management as enabling something they care about (specific features, claims about performance, whatever) and thus offers a way to actually get to do all the maintenance the organization has deferred for years as part of the rewrite.
I appreciate the time that you took to write this post. I know that the Python community, including myself, can be aggressive and unpleasant in online discussions.
This quote is emblematic of the contempt that you display towards Python users. You imply with your words here that porting RPython to Python 3 is “feasible”, but that ignores the cost of labor. Multiple projects have found that Python 3 porting is too expensive to tackle without dedicated funding and support, including not just Django and PyPy, but also Twisted and entire distros like Ubuntu and Gentoo; if those projects cannot find funding then they do not do the porting.
I don’t understand your denigration of interpreters. Producing and maintaining an interpreter is non-trivial, a language community should strive to have a heterogeneous collection of interpreters and compilers besides the reference implementation, and the big list of Python 3 libraries pledging to drop Python 2.7 is missing not just PyPy, but also Jython and IronPython. While some implementations, like MicroPython have switched exclusively to Python 3, others have not.
To paraphrase somebody else’s insight from an IRC discussion, perhaps, as with Perl and Raku, it is the case that Python’s community became too ideologically burdened, and the fork is the result of an ideology operating without constraints. After all, my observations are only weighty because of the inertia of code running on computers, and perhaps that is the only argument in favor of Python 2: That it is the way that things are being done, regardless of the loudness of the claims of Python 3 proponents. I certainly am not here to praise Python 2 either, but to bury it; in fact, were the language still in one piece, I would only have to dig one tomb and write one eulogy. I need to credit glyph for being the one to first paraphrase Marc Antony, over a decade ago at PyCon; I was so slow to understand what everybody else was trying to point out.
I am not asking for either of these positions. First, let me dispel this idea as you’ve worded it. I understand that a language is not fixed in stone, and that it will constantly change and evolve with usage. Indeed, I believe that languages are expressive media, and that the main purpose of language is to send messages and communicate, but since our understanding of the world changes over time, our languages must change as well. (Also, explicitly, I am not wearing any corporate hats. I have many commit bits, but the only relevant plan I currently have is to remove Python entirely from all active codebases, incrementally and immediately.)
My argument, in ten words or less, is: Okay, yes, but must we have changed “2” to “3”? The massive amount of backporting done so far has revealed that no, most of the new features in Python 3 could have been put into Python 2. People had already, as you pointed out, survived having to change
withto be reserved and then a keyword, and that change was far more intrusive than anything in the 2-to-3 changelist. Meanwhile, the modules that ought to have been killed off directly, most notablyasyncore, were left in Python 3 despite being effectively unmaintained and long since rotting.Surely it would be less work to migrate your Python code to Python 3 than to rewrite it in an entirely different language.
He’s having a tantrum.
And I can say that, having overseen the 2 to 3 conversion for a large adtech codebase and maintained 2/3 compatible libraries. I did that because it was both challenging and required really creative, often perverse, solutions. And this was with my mental model very carefully keeping track of string/binary boundaries. It wasn’t easy and people who balanced both interpreters and didn’t collapse under the devilry of duplicate code (and subsequent bug duplication) have my admiration.
If I could move a stodgy adtech company to 3 for active projects, then he can as well.
I’m not a super coder, just someone who adds asserts and routinely retests her assumptions. If you’re not on 3, it’s either a big ball of mud that should’ve been incrementally rewritten/rearchitected (thus exposing bad design) or you expected an ecosystem to stay in stasis forever. If you’re the latter, compile from source and lockdown your environments more than ever cause you’re on your own.
Already I’ve witnessed removal of python 2 from distributions! It’s getting killed regardless of how anyone feels. Digging ones heels in is just silly.
You should work on your mind-reading. My complaint is mostly that, for over a decade, PyPy developers have been consistently marginalized by the PSF. This forced deprecation of Python 2 is merely one more brick in the wall. While I am fortunate to be independently wealthy, and I also barely contribute to PyPy, the typical PyPy developer is a hard-working academic who is trying to dramatically speed up the Python ecosystem, and the PSF’s continued snub of their work is disappointing, especially given their continued endorsement of corporate-sponsored JITs hacked on top of CPython, like Unladen Swallow, Pyston, and Pyjion.
I’ve overseen 2-to-3 conversions as well. I’ve seen companies that can do it fine, companies that are ready with 2to3, and companies that are totally hopeless. I’ve seen companies that are so behind on security patches that they cannot upgrade away from Python 2, and companies that greenfield with Python 3. It is hard but possible. The main sticking point, which folks seem to constantly paper over, is that Python 2 and Python 3 are fundamentally different langauges, with different grammar and semantics. Indeed, PSF representatives have explicitly asked me to not talk about Python 2 and Python 3 as if they are different, and when I’ve asked followup questions, they’ve indicated that it is because treating the two languages as different undermines the porting effort. This is how the system works.
It is funny that you mention “compile from source;” I use Nix, and so of course there will be CPython and PyPy for Python 2.7 available for a very long time via nixpkgs. Again, I am independently configured, and I am not a part of any core development team. But I am not “on my own;” I am one of a community which maintains the ports tree.
No they’re not. The existence of 2/3 compatible packages utterly shits on your point.
The only “breaking” change that you seem to be whining about as soooo different is the splitting of a primitive type. That’s it. Splitting a fucking type into two. That’s not a fundamental language difference and you know it. You’ve made backwards incompatible changes in your code bases and most likely have split classes and refactored. Tell me, was your program utterly grammatically different? Probably not.
You didn’t whine about long/int merging because it’s a trivial alias to add.
But splitting a type is always a lot more contentious and difficult (probably verging on the biggest type of breaking change possible).
Removing “print” so that it could be substituted with a function hardly counts as a massive grammer change. And the addition of coroutines is strictly an “add on”.
I’m not going to even bother with your “mother loved you best” vis a vis PyPy. Jython has been effectively dead for years (unless they finally got invokedynamic support to actually get usable speeds - I last checked two years ago, so sue me).
I don’t know about the others as they’ve been more like toys than have capability of supporting the ecosystem as-is. You can’t just demand equal buy in without supporting the ecosystem as-is - PyPy literally had to make c-extensions work after claiming for years it was impossible to do (which was absurd - I remember reading one of the JVM Ruby interpreters ended up supporting c-extensions through parsing the c-extension source code into compatible libraries - if ruby could do it, so can python).
Please lower the temperature on your posts. You don’t need to curse and make insulting characterizations of his post to make your points.
Your points withstanding (I should be less overtly crude), the GP’s routine use of “contempt” as an accusative is nothing short of insulting. He paints himself as the victim of the PSF and anyone who agrees with them as doing their bidding. It is extremely unkind and quite conversation terminating.
I get that you feel you’re being fair, but it comes across as uneven enforcement amidst the unfair characterizations and portrayal of an anti-Python 2 conspiracy, when in reality it’s an interpreter hide-bound by Guido’s rules to keep it as simple as possible (thus hamstringing backwards compatibility like seen in the JVM), a hold over from the early anemic days of Python and something that’ll never change.
The existence of polyglot libraries does not “shit[] on my point;” it merely shows how far people can be convinced to go in pursuit of a satisfying end-user experience. The existence of libraries which compile under both a C compiler and also a C++, Objective-C, Zig, or etc. compiler does not invalidate the point that the languages are different, regardless of whether the program texts are identical in those cases.
Here are two short interactions with two CPython versions. I contend that they implement distinct grammars and semantics from each other.
And the other one.
I am not sure why you characterize my points as “whining” and then agree with them by substantiating the needful evidence. It seems that you are angry with me because I have points, not because I don’t have points.
By using a broad brush to label various Python projects as “toys” or “dead”, you only emphasize the culture of contempt which I highlighted in my thread-starting comment. Moreover, you’ve said the quiet part out loud: The reason for this contemptuousness is a tribal belief in the power of C extensions over plain old Python code. What’s remarkable to me is that this belief isn’t actually in the service of speed or compatibility; the PyPy project produced not just PyPy, but also CFFI, in order to put those claims to rest. Instead, the belief seems to be simply one of majority rule: Because CPython is the popular standard, CPython must remain the popular standard. To quote from The Power and The Glory:
Uh, no. With the exception of
nonlocalas a keyword (which btw is no different whenwithas a keyword was introduced), all the examples cited are properties on data objects, which is categorically not “grammar”.Grammar pertains to parsing the language into an syntactically valid AST from which a bytecode suitable for the VM can run on. The only incompatible grammatical changes I can think of off the top of my head are:
with(python 2.4)printtoprint()(Python 3)nonlocal(Python 3) (similar to global except restricted to scopes excluding global)That’s it.
Write a proxy for the builtins and you can support older method returns without a problem. That’s why
sixwas did it’s job so well.Does it support the ecosystem? If not, then not very comparable with CPython in now, isn’t it?
As to dead, Jython still doesn’t do invokedynamic and that’s pretty critical to decent performance. Python is well known to be pretty slow. Jython is even slower. That heavily constrains use case to “we have to run a Python script on the JVM” and that’s about it. And it’s also stuck on 2. So it’s left behind in time to a shrinking ecosystem of pure python 2 modules. Talk about useful 🙄
Actually, I didn’t. You made that up all by yourself honey.
I only cited the support of C-Extensions as part of recognizing the audience demand for being able to use PyPy in a broader case than just pure Python and rare/underused one shot ports of C libraries and C-compatible library exports (did several for previous employers) that typically used C-extensions/Cython.
You’re so wrapped up in inventing enemies that heap contempt on you, but it’s just fellow engineers raising their eyebrows at someone being overly dramatic. Lol contempt. 😂😂😂
I need to go update it for 3.8 final and things coming in 3.9, but I maintain a list of ways to “break” Python, geared at library authors who want to trigger syntax or import errors as quickly as possible when someone tries to use one of their libraries on an unsupported Python version.
It also serves as a sort of tour of changes to Python syntax over the past decade.
I like this! It shows the grammatical additions (SyntaxError raising ones) fairly well!
I don’t have any real opinion on python2/3 myself. However, if the language you use forces you to make changes which you disagree with, then it is not unreasonable to at least consider the other options out there. Maybe it’s simple enough for them to port from python 2 to 3, but will they then have to port from 3 to 4 in the future? Maybe porting from 2 to 3 isn’t a massive deal, but they weren’t super happy with python for other reasons, and it was the straw that broke the camel’s back.
It is more or less twenty years since discussion started about possible breaking changes that eventually became Python3 and 14 years since “official” start. We are not even at the first step for Python4. Based on how often most people change jobs and projects in this industry, it is very unlikely they will need to port to 4 or even worry about what that will entail for a completely imaginary version of the language. They are likely to go through far more impactful transitions before then.
These days I mostly write Python 3 and on occasion still Python 2. I find the notion that they are different languages absurd.
Yes, and it would also be less work to use Python 3 than to write an entirely new language. However, we have reasons, and so we produced Monte, a modern flavor of E. We were already leaving Python, and we were already rewriting our Python-based virtual machine in Monte.
I’d like to think I know a thing or two about large projects porting, and about funding situations.
(for the uninitiated: I got my commit bit on Django in 2007, was its release manager from then up to about 2014, served multiple terms on the framework’s technical board, and am currently in my fifth year on the board of the DSF, the nonprofit foundation set up to support Django and its community, including by fundraising for, and then funding, its development)
Also, hi. I am a Python user, and I’d seriously dispute your assertion that I have “contempt” for Python users.
I don’t understand why you interpret it as “denigration”. You’ve repeatedly indicated your dislike for the Python core team’s and the PSF’s messaging that Python 2 is no longer supported, and you’ve argued that Python 2 is supported for as long as alternative Python 2 interpreters are supported. My point is that while a supported interpreter is a necessary thing for calling Python 2 “supported”, it is not sufficient. When people say they want to keep using Python 2, they mean they also want to keep using all their favorite packages and libraries, and many of those simply do not run on Python 2 anymore. And as far as I know, of non-CPython implementations still supporting Python-2-the-language, only PyPy has ever been robust enough to support some of the really important packages like NumPy/SciPy.
Anyway, this is why I bring up the question of whether you’re willing to commit to also maintaining a complete Python 2 ecosystem including the big third-party packages. The Linux distros who’ll be maintaining Python 2 interpreters for years to come have committed to that, but as far as I can tell nobody else has. And without the ecosystem I don’t think you have a sufficient “Python 2” and as I’ve told you directly and also mentioned in the post, I think you create unnecessary and unwanted burdens on those packages’ maintainers if you insist on saying “Python 2 is still supported”.
OK, but now make a Python 2 that cleanly implements the string change and the reorganization of the builtins and the standard library and doesn’t create a massive backwards-compatibility gap to older Python 2 versions. I don’t think that’s possible, and I do think that at least the string change is an absolutely necessary thing for Python’s future.
The Python Packaging Authority (PyPA) is looking to get rid of Python 2 support, at which point I suppose you’ll shift even more of the burden to the remaining Python 2 users somehow, despite the singular package index still being squarely under PSF control.
The rest of your post fails to address my point. If you can’t stop talking past me, then please don’t bother. You can’t even look at a Python 2 user without the insinuation that they must be responsible for the entirety of their ecosystem.
Frankly, I think that you show your hand when you say “really important packages like NumPy/SciPy.” That’s the direction that you want Python to go in.
Well, I’m not trying to be mean about this, but: this is what the “entitlement” section of the post was about. The PyPA folks don’t owe you a Python-2-compatible packaging toolchain. The Django team doesn’t owe you a Python-2-compatible version of Django. And so on.
If those teams have decided they only want to support Python 3, they have the right to do that! And if you want to keep using those tools and libraries and packages with Python 2, you can. You just have to choose between:
Way way back in the early days of Django when the “magic-removal” effort rewrote the ORM (between Django 0.91 and 0.95), creating a big backwards-incompatible jump for projects to get over, I and some other people took the second option: even after the core Django team wasn’t supporting Django 0.91 anymore, we maintained a Django 0.91 branch for a while in order to gain more time to migrate. It’s open source and you can do that.
I could just as easily flip this around and say you can’t look at someone who develops on Python 3 without insinuating they owe you their time and effort to also maintain a Python 2 version of whatever they build.
When you say “Python 2 is still supported”, you seem content to mean it in the specific narrow sense of “there is at least one group still maintaining a Python 2 interpreter”. What I’ve told you many times now is that when people hear “Python 2 is still supported”, what they will expect that to mean is a wider sense of popular third-party packages still maintaining Python 2 compatibility. I have concerns about that, because I maintain third-party packages and I’d rather not have a stream of people showing up in my bug trackers and getting angry because they heard “Python 2 is still supported” and thought that would mean I’ll maintain packages for them to use on Python 2.
NumPy and SciPy are important packages. I don’t personally use them – as I’ve mentioned now multiple times in these discussions with you, I’m a Django guy, and my main use case for Python is as a backend web development lanaguage – but I know that there’s a large segment of the Python community who rely heavily on the numeric/scientific stack to do their work. I don’t see how anyone could look at the Python community and not come to that conclusion. So regardless of whether I use them or not, NumPy and SciPy are important packages. Just as Jupyter (née IPython) notebooks are important, even though I don’t personally use them. Just as the ML/AI packages are important even though I don’t use them. Just as Flask and SQLAlchemy are important packages, even though I don’t use them. Python’s continued success as a language comes from the large community of people using it for different things. The fact that there are large numbers of people using Python for not-my-use-case with not-the-libraries-I-use is a really good thing!
Anyway. You’re angry at what you see as the marginalization of alternative Python implementations, and especially the one you personally are most involved with (PyPy). That’s very clear (in fact, you’ve even said that pretty explicitly). But you’re so angry about it that you’re lashing out at random people on the internet, and either ignoring or misinterpreting what they say.
And on one level, I get it. The alternative implementation story has been rough! Django used to support IronPython and Jython, for example, and we treated it as a release blocker if we had compatibility issues with them. But at a certain point they started falling further and further behind CPython, and eventually it was clear that it would be a long time before they caught up, if ever. That’s sad – it used to be a great way to sneak Django into a Java or Windows shop – but I don’t believe Django has the resources to keep supporting such widely-diverged platforms long-term. If and when Jython and/or IronPython catch up to CPython (and both of them seem to have Python 3 compatibility as a goal), I’d be happy to push for Django to maintain compatibility with them again.
But on another level, you’re not really helping your case when you treat people as your enemy for no longer supporting Python 2 in their packages. If I didn’t already have a long history of knowing other PyPy people, for example, I’d be coming away with a pretty negative view of the project from my interactions with you.
This document is obsolete. Twisted has been ported to Python 3 (jettisonning a few unmaintained modules in the process). The next release will require Python 3, likely 3.5+.
Twisted did take about 15 million years to port, though, so, it does speak to how difficult porting can be. They did start a long time ago, and also have pretty much finished a while ago as well, so, anyone who hasn’t started should’ve kinda seen this (the end-of-life for Python 2) coming.
Yeah, it was a huge project, and particularly difficult because Twisted is exactly the sort of code that is most difficult to port: it sits at the divide between
bytesandstr. Something like a Django app is much easier to port — both because of the solid groundwork laid by Django itself, and also because the bulk of a web app’s logic is only concerned with text.I do think that the OP undersells how unevenly the pain of the Python 3 transition has been distributed. It has been particularly rough for library maintainers, who often want to retain compatibility for a long time to enable their users to migrate. The end result has been a slow and painful transition with the corresponding toll on the Python community.
I would really like to see a blameless post-mortem of the Python 3 transition. Surely we can learn some lessons from this process that apply to development going forward.
Assuming by “OP” you mean me (author of the blog post this whole thread is about), I don’t think I’m “underselling”. I was just trying to write about a bunch of different thoughts and perspectives that I think either haven’t been covered, or haven’t been covered enough, in the copious literature on the Python 2/3 transition.
This is sort of what I was getting at, though. I remember the days of Django’s “unicode branch” and how much work it took to get Django to a point where Django apps mostly didn’t have to worry about that stuff. As the saying goes, a lot of effort went into making that look effortless, and Django’s early experiences strongly influenced my later feelings about the right way to handle strings.
Perhaps not the best word choice on my part.
I do think that the experience of making the transition varies a lot, and that may explain some of the heat in this thread and the discussion in general. We’re all frustrated by how things have dragged out.
Yes, exactly! A post-mortem should focus on what Python could have done to make the transition work this way for everyone.
Sure. The links I used fail to actually tell the story that I wanted to tell. There was a point in time at which most of Twisted was Python 2 and 2to3 wasn’t capable of making up the difference. At that time, people often asked for Python 3 support. However, Python 3 support was only happening through paid work, and community members were not spending time on the porting effort. Eventually, as you say, many modules were simply removed entirely rather than being ported.
This is the process that I wanted to emphasize as happening repeatedly all over the Python ecosystem. Every library maintainer was asked to put in massive amount of effort, to the point where external funding was considered necessary to get things done. But this effort was only worthwhile because Python 3 supposedly was better than Python 2.
I think that misses the point the author is trying to make. He argues that the maintainers of python ARE moving from 2 to 3. What we/you must do in this context is either assume the maintenance burden of a forked python 2 or migrate to python 3. You’re welcome to do neither as well, but of course any bugs related to your ecosystem will go unpatched.
Sorry but just to clarify: do you genuinely think that a keyword becoming reserved is a more intrusive change than switching to Unicode strings and explicit conversion to and from Unicode strings and byte sequences?
As you’ve phrased it, no. However, what you’ve described is not what actually happened. Python 2 supports the Unicode sandwich technique just as well as Python 3, and has the
unicodetype. I would rephrase your question as, “Do I genuinely think that a keyword becoming reserved is a more intrusive change than renamingunicodeandstrtostrandbytesrespectively?” And the answer is yes, I do.From your link:
I can’t believe that a “security professional” would countenance implicit, latent bugs like this on a fundamental data type. I’ve done the unicode sandwich for years and write implicitly statically typed python as a matter of practice (which JITs nicely in PyPy without having to wonder “will it blend?”).
Of the several hundred engineers I’ve met and the dozens I’ve worked with daily for years, I can count the number of people who are aware of unicode/str implicit mixing on one hand.
You’d honestly preserve a constant subtle footgun than explicitly expose the incompatible boundaries so they’re dealt with early on and not left as a landmine to appear years later? Wow