I thought this was an interesting compare-and-contrast of Python package management with Node and C#. Being a relative newbie to Python packaging, I’m never 100% sure if something is confusing because I haven’t invested the time to learn it yet, or if it’s confusing because it’s actually more complex than it needs to be. The author echos some frustrations I’ve personally felt but couldn’t quite articulate myself.
The biggest surprise to me was that the author argued against virtualenvs! I’ve always assumed that venvs were a necessary evil, and that the way to address their rough edges was to use a tool that manages them for you (like pipenv or Poetry). This blog article is the first place I’ve heard of PDM or PEP 582 — my inclination is to go with the crowd and stick with Poetry (or any more-popular tool that displaces it), but I wish luck to everyone involved in replacing the venv model with something simpler.
They currently are a necessary evil, because something like PEP 582 requires a bunch of buy-in that virtual environments don’t. They were a solution to a problem that didn’t require any action from anyone but the user.
I don’t disagree! Mind you, virtualenv was a revelation back in the day. It’s just a pity the core team have a history of what is at best benign neglect when it comes to packaging, which lead to the setuptools debacle, and how it was left to rot for years after PJE became a motivational speaker. A lot of the problems with the Python ecosystem can be traced to that abandonment.
This thread appears to be a response from someone who has significant experience in the space.
A key quote:
This only scratches the surface of the technical complexity here. The reason there are so many tools for managing Python dependencies is because Python is not a monoculture and different folks need different things.
For the record, I agree with the linked thread and believe that people who post this sort of “fix Python packaging by just doing this” article are unhelpfully naïve at best.
You gotta deal with the Python Core team and the steering council. They have consistently abdicated the details of packaging to the community. They aren’t, at this time, very interesting in taking over packaging and telling the community how to manage their dependencies.
That’s useful to contrast with Rust, where it’s a usual talking point how early on Mozilla very deliberately hired bundler’s creator to implement a package manager for Rust.
I agree with most of that thread, but will I pile one more hot take on top of this part of it:
So you want to fix Python packaging: you fucking can’t. get lost.
You also probably don’t fucking need to.
In the past 23ish years of using python, it has only occasionally been a headache for me. And most of those occasions pre-dated virtual environments. Compared to perl, bash, java, php, ruby, or C++, using other people’s software in my python programs has been, broadly, a cakewalk. Distributing my C++ extensions has occasionally been a headache, but no more so than distributing C++ libraries for people to use from C++ programs.
Once I learned my lessons in the early 2000s about not touching the system python, the situation has only occasionally been a hassle.
And these days, a few minutes’ fiddling with pip-tools or poetry is almost always the worst of it. At least on Linux or Linux-adjacent things.
As thorny as some corners of the ecosystem can be, I’ll take it over node. And CPAN. And ruby gems. And pkg-config. And Conan. And maven. And likely a pile of other things I’ve had to touch for varying amounts of time since the late 90s.
Honestly I feel really frustrated about a lot of the discussion around packaging, because there’s this huge focus on “single tools”. I personally wouldn’t care if there are 30 tools! If there is one tool that “does everything” then I’m happy. I’ll use that, other people can do what they want.
If this list is to be believed, it sounds like Poetry is a couple steps away from having all of the features people want. Poetry is its own project… so … I guess we should all just use Poetry and try super hard to get the remaining features in there?
I feel like the insistence on killing off all the older tools is what’s turning so many conversations to be so devisive. A features-oriented discussion feels like a very good alternative, because maybe someone shows up with great patches for pip and makes it work perfectly. But I feel like it’s pretty clear the reason more tools sprout up is because experienced practitioners are unsatisfied.
“What should beginners use” is one question, but I do not feel like the “what should experienced practitioners use” question is at all answered, at least for a lot of practical use cases short of just throwing everything in docker.
The whole article was great, but I’m going to just start quoting
Furthermore, I consider that the PyPA must be destroyed.
in random unrelated other threads now because it is absolutely correct and it tickles my classicist funnybone.
The thing that should come first is making it easy for developers to accomplish common tasks. Everything else should be second to that. You aren’t going to fix all workflows with v1. That’s fine. Focus on solving a vertical slice and expand.
Just look at ESBuild. Before that people would defend Webpack by saying that all the complexity existed for a reason. It did but also most user needs could be solved with a zero configuration tool, and the remaining people can do something else.
in random unrelated other threads now because it is absolutely correct and it tickles my classicist funnybone.
Considering that various peoples’ anti-PyPA crusades already basically pushed people who were actively trying to help improve packaging to give up because they got sick of all the hate directed at them, maybe you should find something else to “tickle your funnybone”.
There are some interesting language-neutral thoughts here, too. For example, what should a package manager do, from the perspective of a new language? What kinds of responsibilities will it have?
As far as Python specifically is concerned, I think that nixpkgs’ Python framework is clearly the best solution available. It does require a large amount of effort from nixpkgs maintainers, but usage is clean, easy, and reproducible. In terms of the table partway through the article, nixpkgs offers everything besides editable installations and PyPI upload assistance.
I continue to wonder how much of this is due to CPython’s C extension module API. If a Python application were necessarily a tree of Python modules, then maybe installation and management would be easier.
I thought this was an interesting compare-and-contrast of Python package management with Node and C#. Being a relative newbie to Python packaging, I’m never 100% sure if something is confusing because I haven’t invested the time to learn it yet, or if it’s confusing because it’s actually more complex than it needs to be. The author echos some frustrations I’ve personally felt but couldn’t quite articulate myself.
The biggest surprise to me was that the author argued against virtualenvs! I’ve always assumed that venvs were a necessary evil, and that the way to address their rough edges was to use a tool that manages them for you (like pipenv or Poetry). This blog article is the first place I’ve heard of PDM or PEP 582 — my inclination is to go with the crowd and stick with Poetry (or any more-popular tool that displaces it), but I wish luck to everyone involved in replacing the venv model with something simpler.
They currently are a necessary evil, because something like PEP 582 requires a bunch of buy-in that virtual environments don’t. They were a solution to a problem that didn’t require any action from anyone but the user.
That’s a very good description of what went wrong with Python packaging.
I don’t disagree! Mind you, virtualenv was a revelation back in the day. It’s just a pity the core team have a history of what is at best benign neglect when it comes to packaging, which lead to the setuptools debacle, and how it was left to rot for years after PJE became a motivational speaker. A lot of the problems with the Python ecosystem can be traced to that abandonment.
Yeah, I totally agree.
This thread appears to be a response from someone who has significant experience in the space.
A key quote:
For the record, I agree with the linked thread and believe that people who post this sort of “fix Python packaging by just doing this” article are unhelpfully naïve at best.
Another interesting quote is
That’s useful to contrast with Rust, where it’s a usual talking point how early on Mozilla very deliberately hired bundler’s creator to implement a package manager for Rust.
I agree with most of that thread, but will I pile one more hot take on top of this part of it:
You also probably don’t fucking need to.
In the past 23ish years of using python, it has only occasionally been a headache for me. And most of those occasions pre-dated virtual environments. Compared to perl, bash, java, php, ruby, or C++, using other people’s software in my python programs has been, broadly, a cakewalk. Distributing my C++ extensions has occasionally been a headache, but no more so than distributing C++ libraries for people to use from C++ programs.
Once I learned my lessons in the early 2000s about not touching the system python, the situation has only occasionally been a hassle.
And these days, a few minutes’ fiddling with pip-tools or poetry is almost always the worst of it. At least on Linux or Linux-adjacent things.
As thorny as some corners of the ecosystem can be, I’ll take it over node. And CPAN. And ruby gems. And pkg-config. And Conan. And maven. And likely a pile of other things I’ve had to touch for varying amounts of time since the late 90s.
Some related chatter on this topic
Honestly I feel really frustrated about a lot of the discussion around packaging, because there’s this huge focus on “single tools”. I personally wouldn’t care if there are 30 tools! If there is one tool that “does everything” then I’m happy. I’ll use that, other people can do what they want.
If this list is to be believed, it sounds like Poetry is a couple steps away from having all of the features people want. Poetry is its own project… so … I guess we should all just use Poetry and try super hard to get the remaining features in there?
I feel like the insistence on killing off all the older tools is what’s turning so many conversations to be so devisive. A features-oriented discussion feels like a very good alternative, because maybe someone shows up with great patches for
pipand makes it work perfectly. But I feel like it’s pretty clear the reason more tools sprout up is because experienced practitioners are unsatisfied.“What should beginners use” is one question, but I do not feel like the “what should experienced practitioners use” question is at all answered, at least for a lot of practical use cases short of just throwing everything in docker.
The whole article was great, but I’m going to just start quoting
in random unrelated other threads now because it is absolutely correct and it tickles my classicist funnybone.
The thing that should come first is making it easy for developers to accomplish common tasks. Everything else should be second to that. You aren’t going to fix all workflows with v1. That’s fine. Focus on solving a vertical slice and expand.
Just look at ESBuild. Before that people would defend Webpack by saying that all the complexity existed for a reason. It did but also most user needs could be solved with a zero configuration tool, and the remaining people can do something else.
Considering that various peoples’ anti-PyPA crusades already basically pushed people who were actively trying to help improve packaging to give up because they got sick of all the hate directed at them, maybe you should find something else to “tickle your funnybone”.
Please wait until I actually do the absurd thing before chiding me for it.
There are some interesting language-neutral thoughts here, too. For example, what should a package manager do, from the perspective of a new language? What kinds of responsibilities will it have?
As far as Python specifically is concerned, I think that nixpkgs’ Python framework is clearly the best solution available. It does require a large amount of effort from nixpkgs maintainers, but usage is clean, easy, and reproducible. In terms of the table partway through the article, nixpkgs offers everything besides editable installations and PyPI upload assistance.
I continue to wonder how much of this is due to CPython’s C extension module API. If a Python application were necessarily a tree of Python modules, then maybe installation and management would be easier.