I don’t know enough about the ins-and-outs of Python’s development, but I can personally vouch for the frustration of developing in Python 3. Not because the language itself is a problem (in fact, I’ve had very few problems with Python 3 as a language), but because the third party libraries which are so prevalent in Python 2 and nowhere near as available in Python 3.
Perhaps forking Python 2 is the proper solution, but there is always an issue with forking something standardized, as XKCD puts so well.
As a user, my only concern is having something usable for professional work. For now, that language is still Python 2 for me. Anecdotally, I am not alone. Whatever solution is chosen, a solution to this issue must be found.
I share your experience. If I could just magically move to writing Python 3 code, I’d be a happy camper. But the migration is a major bummer if you’re a library author since you really need to support both Python 2 and 3 simultaneously.
A few years ago, this was really difficult, but if you have the ability to drop support for Python <=2.5 and Python 3.{0,1,2}, then your life gets a bit easier. Armin Ronacher describes a somewhat sane strategy for writing code that is compatible with both Python 2 and 3.
I’ve done it, so it’s possible. But boy, was it a pain! It utterly sapped the fun out of the task and has been a source of some bugs. There are so many subtle differences between the languages that there is definitely some cognitive overhead to writing the OP’s “Python X” subset.
Aside from that, as you mentioned, dependencies can be a problem too. Many of the major libraries have been ported (or are almost ported), but it’s still incredibly easy to come across a library that doesn’t support 3 yet.
It’s probably likely that folks here already know this, but something I’ve been doing for the past couple years in every Python script I write is just starting with this line:
from __future__ import absolute_import, division, print_function, unicode_literals
The big and famous libraries got ported fairly quickly, but you may have one library that someone took from Sourceforge and then left the company, leaving you to maintain the code. And you can’t migrate until you replace that one library. And as far as the boss is concerned, there is no financial sense in moving to 3.0 (and I agree with him somewhat). It’s always, “Yes, it’s a good thing to do. We’ll put it on the backlog, and come back to it when we have time.”
But you never get time, as there is always one more feature, one more bug to fix.
This, I think, is the biggest reason. There’s enormous boatloads of commercial Python in the world (I’d speculate that it could easily be the majority of Python code written) which is all in Python 2 and which will never get ported (because that’s expensive and yields no business benefit whatsoever) but which still must be supported.
This post was much better than the original. Thanks for sharing! The unfortunate thing about this “counter-argument” is that none of the things would ever happen. The core Python team seems to see nothing wrong with Python 3 as it is.
I didn’t reply on HN, but Python3 already has function type signatures [0] They aren’t enforced, that would be up to a tool, not the base runtime itself.
# this is python3
In [5]: def parb(x: float, y: float) -> float:
...: return 2*x**2 + 2*y**2 - 4
...:
In [10]: parb.__annotations__
Out[10]: {'x': float, 'return': float, 'y': float}
There are also some excellent 3rd party libraries that add nearly any level of typing one would want to the language.
Python is an extremely [1] malleable [2] language that no only supports macros but tail calls [3]. The tail call stuff works, I have implemented beautiful state machines using it.
Yes, Python does provide interesting places to extend it, among them the import hooks. Import hooks allow you to do things like mix other languages in when required. This doesn’t work in all implementations, which is a very valuable aspect of Python.
Macro support in Python is somewhat silly. You can’t modify the parsing, only the interpretation of already defined syntax. That’s still potentially useful I suppose, but it’s pretty limited in its use cases.
Tail calls in Python are not supported. What you’ve linked to is a trick, called trampolining, which can be done in any language that supports calling functions and loops. While the decorator approach makes it slightly nicer, it’s still no where near as powerful as automatically optimized tail calls that you’d get in say, Scheme.
As for type annotations, great. One can even do inference without them with an external tool.
Afaik, MacroPy executes via an import hook so one could do some pretty circuit bendy things, like hylang.
Shedskin is a type inferencing transpiler from Python to C++ that works very well. Check out the examples. One would still need the annotations to type check the code. A side effect of type annotations is that it makes tools like quick check easier to use.
Not sure I get what you are saying about a trampoline based tail call implementation in Python? That the decorator needs to be used? Make a macro in MacroPy that detects recursive calls and applies the decorator automatically!
I’ve used Python for a good number of years successfully. In my opinion there’s no reason to switch to Python 3 as it provides no clear benefits over Python 2. When I use Python, it’s purely out of convenience (I already know it, know a decent number of libraries, etc).
Shedskin’s purpose is simply for performance, which is something that CPython simply doesn’t have without native code, or things like psyco which, was/is a Python specializing compiler and is no longer maintained. There are things like cython which try to bridge a gap between writing C and Python, and they do a failry good job, but in the case of cython, you have to compromise and write in a Python “like” language.
Python 3’s type annotations are pretty meaningless–they’re documentation only unless you add another tool. Adding another tool means you’re likely going to need a build which compromises some of the core aspects of Python that are neat–the REPL (even in its featureless state) and buildless execution cycles (minus the bytecode translation).
So, run the type analysis at runtime. OK, but if I want macros, I have to expand them first, and they must expand with type annotations (or you’re likely going to be getting top level “object” as the type in lots of places). And, if I want to add hylang integration as well, well, now startup has to bootstrap 3 things… And, maybe I’m writing a server with gevent, so that needs monkey patching… And maybe I rely on a Cython module some stats I need to perform…
You can bend Python however you want, sure, but it sure starts to get complicated. It’d be much nicer if this bending came out of the box. Don’t you think?
Re: Trampolines:
The fact that you’re using a trampoline means you’re taking a performance hit for each function call. If you want performance, I’d stay clear of it, and just rewrite the code using a while loop explicitly. It’s a clever trick, but nothing more.
EDIT
Also, I’m not sure how you could make a macro in MacroPy to detect tail calls. I’d have to tell apply the macro somewhere…
In fact, the example MacroPy gives to do this looks like this:
@tco
def fact(n, acc=0):
if n == 0:
return acc
else:
return fact(n-1, n * acc)
I have also been using Python professionally for 14-ish years. I’d like to use some Python3 features but don’t require them. I am still on Python2 (CPython and PyPy).
I kinda like building the language I want out of different libraries. Providing the mechanism (function annotations) while not enforcing a policy is an excellent decision. The syntax is fixed, we largely can’t change that (except for all the hooks available, :) ), but the policies can be programmatic, delegate the decisions to people writing the code. Some people like AoP, ORMs and ??? I program in a mostly functional style using itertools and namedtuples, think Clojure but in Python.
I don’t think bootstrapping your language adds too much complexity and I’d much rather bootstrap it into something I liked than having to modify myself for the language. If Python wasn’t so malleable I wouldn’t be using it. The core devs are heavily into OO, I don’t really do OO, but yet Python still supports functional programming to a level that I feel comfortable. I’d love immutability, pattern matching (this can be added) and tail calls out of the box. But then all of the other people contributing to the ecosystem wouldn’t be here. Python is our CLR. If was designing something from scratch it would be layers of languages with something like an SKI at the bottom with each layer available to the others using a nanopass language delegation framework.
PyPy should remove those trampolines, :)
It isn’t a clever trick, I have used tail calls in Python to have vastly simpler code in production systems. If I used a while loop idiom everywhere I would use a tail call, the code would be a mess. Benchmark the tco recipe, I think you will find it plenty fast for the places it would be used.
# cpython 2.7.6
sys.setrecursionlimit(50000)
# trampoline
In [35]: %timeit factorial(10000)
10 loops, best of 3: 49.7 ms per loop
# stack factorial
In [36]: %timeit stack_fac(10000)
10 loops, best of 3: 38.5 ms per loop
# VM heavy
In [37]: %timeit reduce(operator.mul, range(1,10001))
10 loops, best of 3: 33.7 ms per loop
I have of course picked values that show the trampoline in a good light. For much smaller inputs the non-trampoline code is 3x to 4x faster and largely a non-issue. I use Python for the flexibility.
Edit:
Shedskin also creates native standalone executables, so it isn’t just for performance.
I live in the Python3 world. All new libraries seem to be written for version 3 and most of the major libraries target version 3 as well. It is the efforts in PyPy and Jython that are being killed by this migration.
PyPy is making pypy v3 also so I don’t know how you can say it’s the reason python 3 is “being killed”… http://pypy.org/download.html PyPy3 2.1 beta 1 it just takes a while to move from one version to the other with packages
I know that a couple of libraries we use at work, gevent for example, were only somewhat recently “ported” to python3 for released versions. As such we have a quite extensive codebase on python2, and the large portion of it is web centric – so it would require a good amount of effort to “port” it to python3.
Also until somewhat recently python3 was slower than python2. That is a hard sell for a high volume web stack: spend lots of time upgrading something that works, AND requires more servers? That isn’t going to happen.
My hope is that another 6 months down the road things will look better for python3. I have also been using Go quite a bit more lately. ;)
I’m starting to feel the strain of 2->3. I really want to code off of Python 3 but the library support is just not there. This makes me gravitate towards languages like Go for my next project. While Go doesn’t have quite the library support that Python 2 has, at least I know for certain that my codebase will be targeted by library developers.
I am curious to know why some people are voting this submission as off-topic. The article makes interesting points about issues with Python 3 migration. I don’t personally agree with the conclusion of the article but it’s still a good read if you’re interested in programming language evolution and/or Python.
I don’t know enough about the ins-and-outs of Python’s development, but I can personally vouch for the frustration of developing in Python 3. Not because the language itself is a problem (in fact, I’ve had very few problems with Python 3 as a language), but because the third party libraries which are so prevalent in Python 2 and nowhere near as available in Python 3.
Perhaps forking Python 2 is the proper solution, but there is always an issue with forking something standardized, as XKCD puts so well.
As a user, my only concern is having something usable for professional work. For now, that language is still Python 2 for me. Anecdotally, I am not alone. Whatever solution is chosen, a solution to this issue must be found.
I share your experience. If I could just magically move to writing Python 3 code, I’d be a happy camper. But the migration is a major bummer if you’re a library author since you really need to support both Python 2 and 3 simultaneously.
A few years ago, this was really difficult, but if you have the ability to drop support for Python <=2.5 and Python 3.{0,1,2}, then your life gets a bit easier. Armin Ronacher describes a somewhat sane strategy for writing code that is compatible with both Python 2 and 3.
I’ve done it, so it’s possible. But boy, was it a pain! It utterly sapped the fun out of the task and has been a source of some bugs. There are so many subtle differences between the languages that there is definitely some cognitive overhead to writing the OP’s “Python X” subset.
Aside from that, as you mentioned, dependencies can be a problem too. Many of the major libraries have been ported (or are almost ported), but it’s still incredibly easy to come across a library that doesn’t support 3 yet.
It’s probably likely that folks here already know this, but something I’ve been doing for the past couple years in every Python script I write is just starting with this line:
(Explanation.)
It obviously won’t solve every
2->3problem, but it’s a nice defensive measure that will reduce future burden.Agree with everything you say.
The big and famous libraries got ported fairly quickly, but you may have one library that someone took from Sourceforge and then left the company, leaving you to maintain the code. And you can’t migrate until you replace that one library. And as far as the boss is concerned, there is no financial sense in moving to 3.0 (and I agree with him somewhat). It’s always, “Yes, it’s a good thing to do. We’ll put it on the backlog, and come back to it when we have time.”
But you never get time, as there is always one more feature, one more bug to fix.
This, I think, is the biggest reason. There’s enormous boatloads of commercial Python in the world (I’d speculate that it could easily be the majority of Python code written) which is all in Python 2 and which will never get ported (because that’s expensive and yields no business benefit whatsoever) but which still must be supported.
Someone just posted this “counter-argument” on HN https://medium.com/p/2a7af4788b10
This post was much better than the original. Thanks for sharing! The unfortunate thing about this “counter-argument” is that none of the things would ever happen. The core Python team seems to see nothing wrong with Python 3 as it is.
I didn’t reply on HN, but Python3 already has function type signatures [0] They aren’t enforced, that would be up to a tool, not the base runtime itself.
There are also some excellent 3rd party libraries that add nearly any level of typing one would want to the language.
Python is an extremely [1] malleable [2] language that no only supports macros but tail calls [3]. The tail call stuff works, I have implemented beautiful state machines using it.
The crime with Python3 is that it separates people into one camp or the other. It is like a stop the world collector that lasts years.
Yes, Python does provide interesting places to extend it, among them the import hooks. Import hooks allow you to do things like mix other languages in when required. This doesn’t work in all implementations, which is a very valuable aspect of Python.
Macro support in Python is somewhat silly. You can’t modify the parsing, only the interpretation of already defined syntax. That’s still potentially useful I suppose, but it’s pretty limited in its use cases.
Tail calls in Python are not supported. What you’ve linked to is a trick, called trampolining, which can be done in any language that supports calling functions and loops. While the decorator approach makes it slightly nicer, it’s still no where near as powerful as automatically optimized tail calls that you’d get in say, Scheme.
As for type annotations, great. One can even do inference without them with an external tool.
EDIT: MacroPy looks pretty neat, however.
Afaik, MacroPy executes via an import hook so one could do some pretty circuit bendy things, like hylang.
Shedskin is a type inferencing transpiler from Python to C++ that works very well. Check out the examples. One would still need the annotations to type check the code. A side effect of type annotations is that it makes tools like quick check easier to use.
Not sure I get what you are saying about a trampoline based tail call implementation in Python? That the decorator needs to be used? Make a macro in MacroPy that detects recursive calls and applies the decorator automatically!
Python can be turned into the language you need.
I’ve used Python for a good number of years successfully. In my opinion there’s no reason to switch to Python 3 as it provides no clear benefits over Python 2. When I use Python, it’s purely out of convenience (I already know it, know a decent number of libraries, etc).
Shedskin’s purpose is simply for performance, which is something that CPython simply doesn’t have without native code, or things like psyco which, was/is a Python specializing compiler and is no longer maintained. There are things like cython which try to bridge a gap between writing C and Python, and they do a failry good job, but in the case of cython, you have to compromise and write in a Python “like” language.
Python 3’s type annotations are pretty meaningless–they’re documentation only unless you add another tool. Adding another tool means you’re likely going to need a build which compromises some of the core aspects of Python that are neat–the REPL (even in its featureless state) and buildless execution cycles (minus the bytecode translation).
So, run the type analysis at runtime. OK, but if I want macros, I have to expand them first, and they must expand with type annotations (or you’re likely going to be getting top level “object” as the type in lots of places). And, if I want to add hylang integration as well, well, now startup has to bootstrap 3 things… And, maybe I’m writing a server with gevent, so that needs monkey patching… And maybe I rely on a Cython module some stats I need to perform…
You can bend Python however you want, sure, but it sure starts to get complicated. It’d be much nicer if this bending came out of the box. Don’t you think?
Re: Trampolines: The fact that you’re using a trampoline means you’re taking a performance hit for each function call. If you want performance, I’d stay clear of it, and just rewrite the code using a while loop explicitly. It’s a clever trick, but nothing more.
EDIT Also, I’m not sure how you could make a macro in MacroPy to detect tail calls. I’d have to tell apply the macro somewhere…
In fact, the example MacroPy gives to do this looks like this:
I have also been using Python professionally for 14-ish years. I’d like to use some Python3 features but don’t require them. I am still on Python2 (CPython and PyPy).
I kinda like building the language I want out of different libraries. Providing the mechanism (function annotations) while not enforcing a policy is an excellent decision. The syntax is fixed, we largely can’t change that (except for all the hooks available, :) ), but the policies can be programmatic, delegate the decisions to people writing the code. Some people like AoP, ORMs and ??? I program in a mostly functional style using itertools and namedtuples, think Clojure but in Python.
I don’t think bootstrapping your language adds too much complexity and I’d much rather bootstrap it into something I liked than having to modify myself for the language. If Python wasn’t so malleable I wouldn’t be using it. The core devs are heavily into OO, I don’t really do OO, but yet Python still supports functional programming to a level that I feel comfortable. I’d love immutability, pattern matching (this can be added) and tail calls out of the box. But then all of the other people contributing to the ecosystem wouldn’t be here. Python is our CLR. If was designing something from scratch it would be layers of languages with something like an SKI at the bottom with each layer available to the others using a nanopass language delegation framework.
PyPy should remove those trampolines, :)
It isn’t a clever trick, I have used tail calls in Python to have vastly simpler code in production systems. If I used a while loop idiom everywhere I would use a tail call, the code would be a mess. Benchmark the tco recipe, I think you will find it plenty fast for the places it would be used.
I have of course picked values that show the trampoline in a good light. For much smaller inputs the non-trampoline code is 3x to 4x faster and largely a non-issue. I use Python for the flexibility.
Edit:
Shedskin also creates native standalone executables, so it isn’t just for performance.
I live in the Python3 world. All new libraries seem to be written for version 3 and most of the major libraries target version 3 as well. It is the efforts in PyPy and Jython that are being killed by this migration.
PyPy is making pypy v3 also so I don’t know how you can say it’s the reason python 3 is “being killed”… http://pypy.org/download.html PyPy3 2.1 beta 1 it just takes a while to move from one version to the other with packages
FWIW, I read goalieca’s statement as “PyPy and Jython are being hurt by the migration” rather than “The migration is being hurt by PyPy and Jython.”
(My first reading was the same as yours though. Had to re-read it a couple times.)
I know that a couple of libraries we use at work, gevent for example, were only somewhat recently “ported” to python3 for released versions. As such we have a quite extensive codebase on python2, and the large portion of it is web centric – so it would require a good amount of effort to “port” it to python3.
Also until somewhat recently python3 was slower than python2. That is a hard sell for a high volume web stack: spend lots of time upgrading something that works, AND requires more servers? That isn’t going to happen.
My hope is that another 6 months down the road things will look better for python3. I have also been using Go quite a bit more lately. ;)
I’m starting to feel the strain of 2->3. I really want to code off of Python 3 but the library support is just not there. This makes me gravitate towards languages like Go for my next project. While Go doesn’t have quite the library support that Python 2 has, at least I know for certain that my codebase will be targeted by library developers.
From the header: “photo from wikipedia”
Come on! It’s not a photo from Wikipedia, it’s an engraving by Gustave Doré. Get it right.
I am curious to know why some people are voting this submission as off-topic. The article makes interesting points about issues with Python 3 migration. I don’t personally agree with the conclusion of the article but it’s still a good read if you’re interested in programming language evolution and/or Python.