Kudos to the OP. I got to about the first url he mentions when trying to port terminator off of setup.py, and said, fucki it, this is useless. Terminator still uses setup.py, and it still works.
Python packaging is a twisty little maze with no forward direction. Everybody says “Stop using setup.py” but when you ask “OK, what should I use” you get the answer the OP gets. “It’s complicated”.
Unfortunately, the communications and transition strategy didn’t get a ton of attention or resources. To wit, it’s still often not clearly communicated that what’s deprecated is only the direct invocation of python setup.py to build distribution artifacts or install packages. Build systems such as setuptools still support its use, but need the static information in pyproject.toml to be able to run it in an appropriate isolated build environment.
Why do you need arbitrary script execution during package installation? I’ve built countless tools at work that I had to package with Python and I’ve never run into a situation yet where using setup.cfg (now pyproject.toml) was insufficient.
I was not able to find any build-system-agnostic way to do codegen, possibly a custom backend but the delegation to an existing backend seemed like a chore and ultimately not very useful.
I ended up giving up on that, I did switch what I could to pyproject.toml, but I left a stripped down setup.py with the relevant bits for codegen-ing the package and declared myself happy.
This feels pedantic but I believe actually isn’t: you don’t need to execute that during installation, because Python packaging separates the build and install steps. Even if a pre-built binary for your platform isn’t available, or if for some reason you choose to use a source package instead, the “install” process historically has been the equivalent of python setup.py build && python setup.py install. And this is important because it means that even if you’re using a source package you can run the build phase of it once and the install phase multiple times.
More modern Python package tooling tends to always build an intermediate .whl package and then install from the .whl, even if the PyPI artifact being installed from is a source package, so in the modern era the install process is closer to python -m build --wheel && python -m pip install dist/*.whl.
It sounds like setup.py has been turned into a build script in that case which feels appropriate. I found your other comment about setup.py still being supported by setuptools interesting. I had no clue as someone who interacts with these systems pretty often!
They probably don’t need arbitrary script execution. But for years and years setup.py was the only way to do it, and so in longer lasting projects setup.py continued to grow in size and complexity. That makes it harder to port to “the new way”, which has been a shifting target anyway.
If a project’s setup.py works, no sense in changing it just for change’s sake.
If a project’s setup.py works, no sense in changing it just for change’s sake.
Except it’s not “for change’s sake”. Imagine how insane it would be to require arbitrary script execution just to get project metadata in a new language today. It’s a massive security hole that should have never existed in the first place.
The original packaging library, distutils, used the executable setup.py file. The followup library, setuptools, continued to use the executable setup.py file but also introduced, over many years, the ability to do more and more configuration statically in a file named setup.cfg, eventually allowing many simple projects to just not have a setup.py at all.
But it was always setuptools-specific, and never standardized. The pyproject.toml file has been gradually standardized as a static declarative package-metadata file over the past few years, and many different tools and types of tools now understand it.
It wasn’t an issue because distutils then setuptools were the standard. Now Python is moving toward a way to declare which build system developers want to use for their own package, they need to standardize the whole process so at the very least metadata can be extracted before the package manager installs the build system.
Wrong question. The problem is “how do I migrate off what everyone has been doing for years”? The person doing it may not even be the one who originally did it this way in the project at hand.
I wrote my own notes on this because I was having trouble figuring out the subset of information I needed for my own projects from other sources - there’s so much material out there, but getting to the core that I needed to know was surprisingly difficult.
Just to post a counterpoint from all of this: I have been using pyproject.toml for new stuff, and there’s a lot of stuff I don’t have to care about anymore. There is real whiplash and frustration to be had on existing projects but there is actual forward movement (though I have my opinions on deprecating things and breaking workflows “because it’s cleaner”…).
Porting existing projects is extremely frustrating because you have to deal with a bunch of stuff, sometimes as a response to a previous workflow’s issues. So you end up needing to port the old tools idiosyncracies, along with handling the new ones. It’s a mess and frustrating, and everything OP brought up is totally valid. Just wanted to reinforce that things are moving forward for at least one subset of the population (which is more than what could be said a couple years ago!)
I have recently create two very simple Python packages with pyproject.toml and struggled a lot both times because is not very clear on what is the path forward (why are there 4 build systems? At least make one canonical! Or at least provide guidance on which to use) and I don’t have a clue on how it works at all anymore (you don’t even specify which files or so get packaged)
The experience in this article overlaps in many ways with a journey I took a month ago. PEP 517 (which is for diversifying build backends) is an area where “there should be one—and preferably only one—obvious way to do it”, one of the Zen of Python’s principals, is being ignored.
I look forward to reading comments explaining why Gregory is wrong because the Python packaging system is the best of all possible worlds, and all you have to do is know exactly which five tools still work and how to use them.
Okay? I’m not defending packaging being good. I’m saying the OP is essentially complaining that they haven’t kept up to date in their own working knowledge, which is true. distutils was deprecated three years ago.
Leibniz wasn’t arguing the world was actually good. He was arguing that it couldn’t be any better given that God was constrained by certain axioms like logical consistency, the principle of sufficient reason, absolute divine simplicity, etc.
Defending Python packaging isn’t always about saying it’s good. It’s more about “well it can’t be any better because X.” So below, Paul Moore says it can’t be any better because of lack of resources. Elsewhere you see people say it can’t be better because Python is trying to do a more complicated thing than other languages. In your comment, the argument is that to improve required a lot of churn, so it’s on Gregory for not keeping up with the churn.
I wouldn’t have linked to it if the comment was by someone else, but it was too perfect to link to a comment by you as refutation of your point. :-P
Except it’s not a refutation as I’ve never made the argument the status quo is good, nor that it can’t be improved. To improve is to require change. I do think most of their complaints are due to them not keeping up even a cursory amount. This is a language that has new major versions every year and we can see they’re putting more effort in the last few releases into cleaning up their house (like the removing dead batteries PEP as an example).
I think it’s at least a little absurd that python has had to iterate somuch for solong to get where it is today. This is a solved problem for many other languages. While you’re not saying the situation is good, it does feel like a little bit of stockholm syndrome when you suggest it’s on the user for not keeping up with the packaging tool of the moment.
That’s the thing, it’s not even tooling of the moment. These are standards that have been in the works for over 7 years now (the PEP for pyproject.toml and build infrastructure is from 2016!). We’re talking about the same tools that have been standard since Python 2 here, pip and setuptools.
I think it is somewhat on the user for using something deprecated for several years then being suddenly surprised when it’s finally removed with that much notice (setuptools added a warning in Oct ’21)
pip and setuptools have certainly been around for a long time but so have lots of other things (pipenv and poetry?). I remember around ~2017 (?) the pip dependency resolver would still leave you with broken dependencies in some cases so those other tools were definitely serving a purpose.
pyproject.toml is from 2016, yes, but it definitely wasn’t usable from the get-go. distutils was only deprecated in 2020.
To be clear, I totally agree that pip and setuptools have been the Right Way for a long time. But I don’t think that has been the consensus in the community, and to someone whose day job is not python it can be hard to discover that.
I think three years is plenty of time to go from deprecation to removal.
But I don’t think that has been the consensus in the community
I’m not sure it’s the consensus even now and I still explore other options every few months to see if things have improved. It feels like Poetry is doing its own thing, Pipenv was only revived for name recognition and hasn’t improved, pip-tools have difficult to use workflows as things still boil down to requirements.txt in most cases. I learned about Hatch from this blogpost and haven’t explored it yet. I’m kind of excited for Ruff’s next idea of tackling this problem.
I really just want a unified, “this fits most use-cases” project manager that supports the library and application workflows, and makes good opinions according to practices of the day (like src/ layout packages, pyproject.toml, etc.)
I do think, though, that the biggest issue we have here is a lack of resources.
I completely disagree with that. The biggest issue is lack of leadership. Someone needs to say that all of the million little flowers blossoming in the PyPA garden are actually weeds and begin the process of pulling them up. Start with the easy cases but have a road map to get to the hard cases. No more layering one tool on top of another. Start from the bottom and work up. Have a vision for what the whole thing looks like and move towards it in deliberate steps. It does not actually take a large team. If anything, the problem with PyPA is that too many people are working in opposing directions. You want there to be between 2 and 5 people working in the same direction. That’s more than enough!
Really what needs to happen is GvR needs to care about packaging, come out of retirement long enough to declare someone the Package Sub-Emperor for the Indefinite Crisis Period and then go back to retirement. Alas.
Really what needs to happen is GvR needs to care about packaging, come out of retirement long enough to declare someone the Package Sub-Emperor for the Indefinite Crisis Period and then go back to retirement. Alas.
The traditional name for this was “BDFL-delegate” for a particular area. And for packaging there are two such delegates: Brett Cannon, and Paul Moore (who is the author of the post you were linked to).
I’m not sure what your intent was with that comment, but the reason is that they’re both active in proposing things in the packaging world, which raises the question of what happens if a packaging PEP is proposed by the person whose job is to make acceptance decisions on packaging PEPs. So there are two people with delegated authority to approve packaging PEPs – Brett and Paul – and whenever one of them writes a PEP the other is the one who makes the decision on whether to approve it.
I don’t think Python packaging is in a good place, but I think only people already using relatively complex or long-standing Python projects bump into it as a notable problem. If you start a new Python project as an apprentice, you use Poetry. If you start a new Python project as a journeyman, you might use Poetry for an application, or Hatch/PDM for an application or library because they respect pyproject.toml. If you’re a master craftsman or greybeard, you can do anything you want except the thing you were doing - this is where the quality of documentation seems to drop off of a cliff, and google-fu will paradoxically give you unintuitive results.
I have started basic scripts in python as a project. And threw the towel when we tried to install the dependencies on s coworker machine and it never managed to install or find them.
It is broken at every level. At this point we write bash. It is atrocious to read, near unmaintainable, cannot have tests and packages are non existent.
Every time I help someone on-board with Python for the first time I spend an uncomfortable amount of time talking to them about virtual environments, and telling them “if something doesn’t work, the first question to ask yourself is whether or not you remembered to ‘activate your virtual environment’”.
I really wish that workflow was more closely baked into the standard path for using Python. Node.js has similar mechanisms but they’re part of the default “npm install” path so they seem to bite people a lot less.
Node.js has similar mechanisms but they’re part of the default “npm install” path so they seem to bite people a lot less.
They’re not - they’re baked into the default module resolution algorithm. The fact that the core Node runtime behaves like this, and has pretty much always behaved like this, is why there aren’t 5 competing venv mechanisms and why there isn’t a spectrum of how integrated the venv stuff is with package managers (npm/yarn/pnpm/etc.). It all Just Works™ if you write a new package manager.
Python, on the other hand, has a legacy problem. Though I’m not sure it’s an insurmountable one.
We tried. It was yelling at us. Couldn’t figure out how to debug a venv, how to consistently use the right one or make it interact with other commands.
At this point, after a decade trying to make venv works, i consider any answer to a python problem that think that venv works or solve something to be irrelevant.
The first step toward any solution for python packaging is to ditch venv. Design solutions that do not need them and actively kill them. They are a constant source of problems and users, universal ly, hate them.
Virtual environment are a failed UX experiment. Time to accept reality.
I’m convinced virtual environments are the right underlying implementation: being able to spin up separate projects with potentially conflicting dependencies feels right to me.
The problem is how to provide a really clear user interface over the top.
In particular, that UI needs to consider the unhappy paths. Venv on the happy path works just fine - but evidently the moment someone steps off that happy path (forgets to run one very specific commands at the right moment, renames a crucial folder, installs an upgraded Python version) things can get very difficult to figure out.
I use venv every day. Vscode and pycharm find them. I debug into them both by stepping from my code into the venv packages and by setting breakpoints in the venv packages. I use pyenv with virtualenv wrapper to get seamless changes of venvs as I change directories in a shell. Pipenv and poetry both work sanely with venvs. I’ve been using venv like this since 2018.
I bring this up as a counterpoint to your strong assertions. I understand a decade of difficulties with a technology can sour anyone. I have several technologies that I feel that way towards.
A heresy that has worked for me in the past, and that may work for you: for scripts where you need a bit more than the standard library but a lot less than full PyPi, telling your coworkers which (e.g.) Ubuntu packages to install in a README.txt works.
This strategy does limit you to Python packages which are fairly widely used (and thus packaged), and does mean you get updates when the distribution maintainers push out package updates (or, more realistically, when you move to a new version of your distribution.) Those are arguably the wrong tradeoffs for a SaaS application that the whole company is focused on, but that’s not the only case where one might want to use Python…
Most of my new projects are more libraries than end-user applications, so I find range-based dependencies in pyproject.toml are fine.
I haven’t actually selected a lockfile tool I like best yet - I’m still mostly using pip-tools for that but I’m ready to upgrade if there’s a clear single winner I can switch to: https://til.simonwillison.net/python/pip-tools
What would you recommend for a Python webapp on NixOS? Asking for a friend, who spent a couple of weeks trying to wrangle those two things together with only partial success. I don’t know what was worse: the fragmentation, or the lack of tooling.
I would use Poetry/Hatch, which work perfectly and as expected, if you set up your development environment with a Nix flake. I think you need to configure poetry with in-project venvs (poetry config virtualenvs.in-project true). e.g. https://gist.github.com/offsetcyan/fd9352a4d00d3c6c940187c8bf341452
But on NixOS in particular, I understand the frustrations in figuring this kind of thing out.
This whole blog post reads like it’s from someone who hasn’t touched Python’s packaging ecosystem for several years and is complaining about every change over the course of several years in one go. I empathize dealing with breaking changes is not fun, but they haven’t even tried to stay current in their working knowledge.
This whole blog post reads like it’s from someone who hasn’t touched Python’s packaging ecosystem for several years and is complaining about every change over the course of several years in one go.
Of course it does: the packaging ecosystem was pretty bad but extremely stable for 10, 15 years. Then things started shifting quickly, and relatively recently packaging tools started emitting scary warnings about things getting deprecated and not being supported anymore (specifically invoking setup.py directly).
So maintainers scramble for an explanation, having not cared for years because they’d gotten a setup which worked for them (as janky as it was) and would just replicate that from project to project.
And they face something which is still bad but also not stable at all, with little to no resources, and fragmented enough you’d think a bomb went off.
they haven’t even tried to stay current in their working knowledge.
Why would they? Dealing with packaging is not interesting, productive, or rewarding, they’ve got other shit on their plate. If you get to a working packaging state you stop touching it. There’s a billion other things to keep up with which actually has a chance of being useful.
Why would they? Dealing with packaging is not interesting, productive, or rewarding, they’ve got other shit on their plate. If you get to a working packaging state you stop touching it. There’s a billion other things to keep up with which actually has a chance of being useful.
Because it’s part of the work. It’s part of being a technologist. Something always changes, and there’s been a lot of discussion in this arena over the past four years in the Python community (and in general, longer; the status quo of Python packaging / dependency management has never been good).
I don’t know about this. I was (am?) a happy user of setup.py/pip/virtualenv for many years and never had big problems with it. Sure, it wasn’t the best, and occasionally I would run into problems that were frustrating, but they were never difficult to fix and things were simple.
With the current state of the world, people need to do a bunch of archaeology to even figure out which question they need to be asking. No longer is it “how do I fix this problem with setup.py”, now it’s “What tool do I need to be asking a question about?”, which in my opinion is actually a worse situation to be in from the end user’s perspective.
I spend hours every week keeping up to date with what’s changing in the general technical landscape, and I have no idea whatsoever what is going on with Python’s packaging ecosystem. Compare that to things like:
Async Rust
The recent Java changes
C++23
I have a great idea of what is going on in those projects just by reading sites like this one. There is a deeper problem at play here than “the author didn’t do their homework”. It might contribute to the problem, but it’s not the sole/root cause IMO.
It is not though. Not in the sense that it’s of any interest if you’re not affected or impacted.
and there’s been a lot of discussion in this arena over the past four years in the Python community (and in general, longer
So what? Do you keep abreast of developments in APL?
the status quo of Python packaging / dependency management has never been good).
No, but I would very much argue people don’t value stability enough. You can’t build on sand, and people are rightfully annoyed when you dynamite their homes and move them to a tent city even if you promise eventual space habitats.
Why is it ridiculous to expect that a package maintainer pays attention to their tools more frequently than once every five years? Based off the OP’s comments in the article while working through the transition, I would’ve assumed they’ve never touched Python before.
It sounds to me like @masklinn is salty because a previously working system has been deprecated before its replacement is ready, which is an entirely reasonable thing to be salty about. And your response is to say that the complaints are invalid, and that the solution to missing documentation is to pay attention to the inner details of a work in progress. It wouldn’t be necessary to follow all the many python packaging projects if they worked and were documented properly.
I can assure you Gregory has touched Python before - they are responsible for some extremely influential and cutting-edge projects. That’s what makes their feedback here particularly notable.
In OP’s defense, if you’re maintaining a project for a decade after a while you have the working build scripts and so aren’t confronted with the new stuff for a very very very long time.
So you end up with two problems:
You need to learn about the new tools
You need to port a complicated, idiosyncratic build to the new tools
I do feel like in a universe where he’s starting a new package, this blog is half as long. But well.. he’s not.
I think that every new tool needs a cookbook for common patterns from older tools. “What about this part of setup.py?” “It’s this now” or “this isn’t supported, you need to customize your build engine” or “This is a non-issue in the new model”. Because most people are working on existing projects, not new ones.
One thing that often confuses me if that why do alternatives like flit, poetry, hatch, pdm, etc. exist? What does one do that the other or setuptools does not?
My understanding is that Poetry predates some of these standards slightly, and has certain benefits (automatically managing your venv, lockfiles). Flit exists to be extremely small and simple and require the least from you to get it working—but provides the least benefit if you start needing C extensions, for instance. Hatch seems to have come later and be very standards driven while providing more than (say) flit or build, but not as much as Poetry.
I don’t believe that it really serves the Python community overall, having a proliferation of options here. Nobody really cares how big or small cargo or cabal are. You use the standard thing and get on with life.
One (setuptools) is a mechanism for building and installing packages. The others are project managers that also include features like dependency resolution, lockfiles, et al. for repeatable builds.
Kudos to the OP. I got to about the first url he mentions when trying to port terminator off of setup.py, and said, fucki it, this is useless. Terminator still uses setup.py, and it still works.
Python packaging is a twisty little maze with no forward direction. Everybody says “Stop using setup.py” but when you ask “OK, what should I use” you get the answer the OP gets. “It’s complicated”.
That’s when I gave up.
Unfortunately, the communications and transition strategy didn’t get a ton of attention or resources. To wit, it’s still often not clearly communicated that what’s deprecated is only the direct invocation of
python setup.pyto build distribution artifacts or install packages. Build systems such as setuptools still support its use, but need the static information inpyproject.tomlto be able to run it in an appropriate isolated build environment.Why do you need arbitrary script execution during package installation? I’ve built countless tools at work that I had to package with Python and I’ve never run into a situation yet where using
setup.cfg(nowpyproject.toml) was insufficient.A common use case is platform-specific configuration for compiling C extensions. See, for example, the ~1kloc setup.py for Pillow.
Codegen from data files as well.
I was not able to find any build-system-agnostic way to do codegen, possibly a custom backend but the delegation to an existing backend seemed like a chore and ultimately not very useful.
I ended up giving up on that, I did switch what I could to pyproject.toml, but I left a stripped down
setup.pywith the relevant bits for codegen-ing the package and declared myself happy.This feels pedantic but I believe actually isn’t: you don’t need to execute that during installation, because Python packaging separates the build and install steps. Even if a pre-built binary for your platform isn’t available, or if for some reason you choose to use a source package instead, the “install” process historically has been the equivalent of
python setup.py build && python setup.py install. And this is important because it means that even if you’re using a source package you can run the build phase of it once and the install phase multiple times.More modern Python package tooling tends to always build an intermediate
.whlpackage and then install from the.whl, even if the PyPI artifact being installed from is a source package, so in the modern era the install process is closer topython -m build --wheel && python -m pip install dist/*.whl.It sounds like setup.py has been turned into a build script in that case which feels appropriate. I found your other comment about setup.py still being supported by setuptools interesting. I had no clue as someone who interacts with these systems pretty often!
They probably don’t need arbitrary script execution. But for years and years setup.py was the only way to do it, and so in longer lasting projects setup.py continued to grow in size and complexity. That makes it harder to port to “the new way”, which has been a shifting target anyway.
If a project’s setup.py works, no sense in changing it just for change’s sake.
Except it’s not “for change’s sake”. Imagine how insane it would be to require arbitrary script execution just to get project metadata in a new language today. It’s a massive security hole that should have never existed in the first place.
This sounds like a big part of the problem…
It’s been a few years, but the ecosystem has finally settled on the standard now.
The original packaging library,
distutils, used the executablesetup.pyfile. The followup library,setuptools, continued to use the executablesetup.pyfile but also introduced, over many years, the ability to do more and more configuration statically in a file namedsetup.cfg, eventually allowing many simple projects to just not have asetup.pyat all.But it was always
setuptools-specific, and never standardized. Thepyproject.tomlfile has been gradually standardized as a static declarative package-metadata file over the past few years, and many different tools and types of tools now understand it.It wasn’t an issue because distutils then setuptools were the standard. Now Python is moving toward a way to declare which build system developers want to use for their own package, they need to standardize the whole process so at the very least metadata can be extracted before the package manager installs the build system.
Standardization by convention is a terrible way to establish policy.
It wasn’t just a convention. distutils was shipped in the standard library and setuptools was explicitly recommended by Python’s documentation.
Wrong question. The problem is “how do I migrate off what everyone has been doing for years”? The person doing it may not even be the one who originally did it this way in the project at hand.
[Comment removed by author]
I wrote this guide to using pyproject.toml a few months ago and I’ve been referring back to it ever since: https://til.simonwillison.net/python/pyproject
I wrote my own notes on this because I was having trouble figuring out the subset of information I needed for my own projects from other sources - there’s so much material out there, but getting to the core that I needed to know was surprisingly difficult.
Just to post a counterpoint from all of this: I have been using pyproject.toml for new stuff, and there’s a lot of stuff I don’t have to care about anymore. There is real whiplash and frustration to be had on existing projects but there is actual forward movement (though I have my opinions on deprecating things and breaking workflows “because it’s cleaner”…).
Porting existing projects is extremely frustrating because you have to deal with a bunch of stuff, sometimes as a response to a previous workflow’s issues. So you end up needing to port the old tools idiosyncracies, along with handling the new ones. It’s a mess and frustrating, and everything OP brought up is totally valid. Just wanted to reinforce that things are moving forward for at least one subset of the population (which is more than what could be said a couple years ago!)
I have recently create two very simple Python packages with pyproject.toml and struggled a lot both times because is not very clear on what is the path forward (why are there 4 build systems? At least make one canonical! Or at least provide guidance on which to use) and I don’t have a clue on how it works at all anymore (you don’t even specify which files or so get packaged)
The experience in this article overlaps in many ways with a journey I took a month ago. PEP 517 (which is for diversifying build backends) is an area where “there should be one—and preferably only one—obvious way to do it”, one of the Zen of Python’s principals, is being ignored.
I look forward to reading comments explaining why Gregory is wrong because the Python packaging system is the best of all possible worlds, and all you have to do is know exactly which five tools still work and how to use them.
I don’t think I’ve read comments indicating Python packaging is good ever? This strawman doesn’t hold up to a slight breeze.
I’ve heard it from Python fanatics myself.
https://lobste.rs/s/yu8tzj/my_user_experience_porting_off_setup_py#c_esw5zu
Okay? I’m not defending packaging being good. I’m saying the OP is essentially complaining that they haven’t kept up to date in their own working knowledge, which is true.
distutilswas deprecated three years ago.Leibniz wasn’t arguing the world was actually good. He was arguing that it couldn’t be any better given that God was constrained by certain axioms like logical consistency, the principle of sufficient reason, absolute divine simplicity, etc.
Defending Python packaging isn’t always about saying it’s good. It’s more about “well it can’t be any better because X.” So below, Paul Moore says it can’t be any better because of lack of resources. Elsewhere you see people say it can’t be better because Python is trying to do a more complicated thing than other languages. In your comment, the argument is that to improve required a lot of churn, so it’s on Gregory for not keeping up with the churn.
I wouldn’t have linked to it if the comment was by someone else, but it was too perfect to link to a comment by you as refutation of your point. :-P
Except it’s not a refutation as I’ve never made the argument the status quo is good, nor that it can’t be improved. To improve is to require change. I do think most of their complaints are due to them not keeping up even a cursory amount. This is a language that has new major versions every year and we can see they’re putting more effort in the last few releases into cleaning up their house (like the removing dead batteries PEP as an example).
I think it’s at least a little absurd that python has had to iterate so much for so long to get where it is today. This is a solved problem for many other languages. While you’re not saying the situation is good, it does feel like a little bit of stockholm syndrome when you suggest it’s on the user for not keeping up with the packaging tool of the moment.
That’s the thing, it’s not even tooling of the moment. These are standards that have been in the works for over 7 years now (the PEP for
pyproject.tomland build infrastructure is from 2016!). We’re talking about the same tools that have been standard since Python 2 here,pipandsetuptools.I think it is somewhat on the user for using something deprecated for several years then being suddenly surprised when it’s finally removed with that much notice (
setuptoolsadded a warning in Oct ’21)pipandsetuptoolshave certainly been around for a long time but so have lots of other things (pipenv and poetry?). I remember around ~2017 (?) the pip dependency resolver would still leave you with broken dependencies in some cases so those other tools were definitely serving a purpose.pyproject.tomlis from 2016, yes, but it definitely wasn’t usable from the get-go.distutilswas only deprecated in 2020.To be clear, I totally agree that
pipandsetuptoolshave been the Right Way for a long time. But I don’t think that has been the consensus in the community, and to someone whose day job is not python it can be hard to discover that.I think three years is plenty of time to go from deprecation to removal.
I’m not sure it’s the consensus even now and I still explore other options every few months to see if things have improved. It feels like Poetry is doing its own thing, Pipenv was only revived for name recognition and hasn’t improved,
pip-toolshave difficult to use workflows as things still boil down torequirements.txtin most cases. I learned aboutHatchfrom this blogpost and haven’t explored it yet. I’m kind of excited for Ruff’s next idea of tackling this problem.I really just want a unified, “this fits most use-cases” project manager that supports the library and application workflows, and makes good opinions according to practices of the day (like
src/layout packages,pyproject.toml, etc.)In that case, you may find this response from a CPython core dev & pip maintainer to the OP surprising.
Interesting.
I completely disagree with that. The biggest issue is lack of leadership. Someone needs to say that all of the million little flowers blossoming in the PyPA garden are actually weeds and begin the process of pulling them up. Start with the easy cases but have a road map to get to the hard cases. No more layering one tool on top of another. Start from the bottom and work up. Have a vision for what the whole thing looks like and move towards it in deliberate steps. It does not actually take a large team. If anything, the problem with PyPA is that too many people are working in opposing directions. You want there to be between 2 and 5 people working in the same direction. That’s more than enough!
Really what needs to happen is GvR needs to care about packaging, come out of retirement long enough to declare someone the Package Sub-Emperor for the Indefinite Crisis Period and then go back to retirement. Alas.
The traditional name for this was “BDFL-delegate” for a particular area. And for packaging there are two such delegates: Brett Cannon, and Paul Moore (who is the author of the post you were linked to).
Two dictators? Bless.
I’m not sure what your intent was with that comment, but the reason is that they’re both active in proposing things in the packaging world, which raises the question of what happens if a packaging PEP is proposed by the person whose job is to make acceptance decisions on packaging PEPs. So there are two people with delegated authority to approve packaging PEPs – Brett and Paul – and whenever one of them writes a PEP the other is the one who makes the decision on whether to approve it.
I don’t think Python packaging is in a good place, but I think only people already using relatively complex or long-standing Python projects bump into it as a notable problem. If you start a new Python project as an apprentice, you use Poetry. If you start a new Python project as a journeyman, you might use Poetry for an application, or Hatch/PDM for an application or library because they respect pyproject.toml. If you’re a master craftsman or greybeard, you can do anything you want except the thing you were doing - this is where the quality of documentation seems to drop off of a cliff, and google-fu will paradoxically give you unintuitive results.
I have started basic scripts in python as a project. And threw the towel when we tried to install the dependencies on s coworker machine and it never managed to install or find them.
It is broken at every level. At this point we write bash. It is atrocious to read, near unmaintainable, cannot have tests and packages are non existent.
But at least it runs.
Were you using virtual environments?
Every time I help someone on-board with Python for the first time I spend an uncomfortable amount of time talking to them about virtual environments, and telling them “if something doesn’t work, the first question to ask yourself is whether or not you remembered to ‘activate your virtual environment’”.
I really wish that workflow was more closely baked into the standard path for using Python. Node.js has similar mechanisms but they’re part of the default “npm install” path so they seem to bite people a lot less.
They’re not - they’re baked into the default module resolution algorithm. The fact that the core Node runtime behaves like this, and has pretty much always behaved like this, is why there aren’t 5 competing venv mechanisms and why there isn’t a spectrum of how integrated the venv stuff is with package managers (npm/yarn/pnpm/etc.). It all Just Works™ if you write a new package manager.
Python, on the other hand, has a legacy problem. Though I’m not sure it’s an insurmountable one.
We tried. It was yelling at us. Couldn’t figure out how to debug a venv, how to consistently use the right one or make it interact with other commands.
At this point, after a decade trying to make venv works, i consider any answer to a python problem that think that venv works or solve something to be irrelevant.
The first step toward any solution for python packaging is to ditch venv. Design solutions that do not need them and actively kill them. They are a constant source of problems and users, universal ly, hate them.
Virtual environment are a failed UX experiment. Time to accept reality.
I’m convinced virtual environments are the right underlying implementation: being able to spin up separate projects with potentially conflicting dependencies feels right to me.
The problem is how to provide a really clear user interface over the top.
In particular, that UI needs to consider the unhappy paths. Venv on the happy path works just fine - but evidently the moment someone steps off that happy path (forgets to run one very specific commands at the right moment, renames a crucial folder, installs an upgraded Python version) things can get very difficult to figure out.
This is a really hard design problem!
I use venv every day. Vscode and pycharm find them. I debug into them both by stepping from my code into the venv packages and by setting breakpoints in the venv packages. I use pyenv with virtualenv wrapper to get seamless changes of venvs as I change directories in a shell. Pipenv and poetry both work sanely with venvs. I’ve been using venv like this since 2018.
I bring this up as a counterpoint to your strong assertions. I understand a decade of difficulties with a technology can sour anyone. I have several technologies that I feel that way towards.
A heresy that has worked for me in the past, and that may work for you: for scripts where you need a bit more than the standard library but a lot less than full PyPi, telling your coworkers which (e.g.) Ubuntu packages to install in a README.txt works.
This strategy does limit you to Python packages which are fairly widely used (and thus packaged), and does mean you get updates when the distribution maintainers push out package updates (or, more realistically, when you move to a new version of your distribution.) Those are arguably the wrong tradeoffs for a SaaS application that the whole company is focused on, but that’s not the only case where one might want to use Python…
I’m finding that you don’t need to use Poetry or Hatch or PDM to use pyproject.toml.
https://til.simonwillison.net/python/pyproject
I wouldn’t recommend someone start a new project without a package & dependency management tool like PDM (which supports lockfiles) or Hatch.
Because of the need for lock files?
Most of my new projects are more libraries than end-user applications, so I find range-based dependencies in pyproject.toml are fine.
I haven’t actually selected a lockfile tool I like best yet - I’m still mostly using pip-tools for that but I’m ready to upgrade if there’s a clear single winner I can switch to: https://til.simonwillison.net/python/pip-tools
What would you recommend for a Python webapp on NixOS? Asking for a friend, who spent a couple of weeks trying to wrangle those two things together with only partial success. I don’t know what was worse: the fragmentation, or the lack of tooling.
I would use Poetry/Hatch, which work perfectly and as expected, if you set up your development environment with a Nix flake. I think you need to configure poetry with in-project venvs (
poetry config virtualenvs.in-project true). e.g. https://gist.github.com/offsetcyan/fd9352a4d00d3c6c940187c8bf341452But on NixOS in particular, I understand the frustrations in figuring this kind of thing out.
This whole blog post reads like it’s from someone who hasn’t touched Python’s packaging ecosystem for several years and is complaining about every change over the course of several years in one go. I empathize dealing with breaking changes is not fun, but they haven’t even tried to stay current in their working knowledge.
Of course it does: the packaging ecosystem was pretty bad but extremely stable for 10, 15 years. Then things started shifting quickly, and relatively recently packaging tools started emitting scary warnings about things getting deprecated and not being supported anymore (specifically invoking setup.py directly).
So maintainers scramble for an explanation, having not cared for years because they’d gotten a setup which worked for them (as janky as it was) and would just replicate that from project to project.
And they face something which is still bad but also not stable at all, with little to no resources, and fragmented enough you’d think a bomb went off.
Why would they? Dealing with packaging is not interesting, productive, or rewarding, they’ve got other shit on their plate. If you get to a working packaging state you stop touching it. There’s a billion other things to keep up with which actually has a chance of being useful.
Because it’s part of the work. It’s part of being a technologist. Something always changes, and there’s been a lot of discussion in this arena over the past four years in the Python community (and in general, longer; the status quo of Python packaging / dependency management has never been good).
I don’t know about this. I was (am?) a happy user of setup.py/pip/virtualenv for many years and never had big problems with it. Sure, it wasn’t the best, and occasionally I would run into problems that were frustrating, but they were never difficult to fix and things were simple.
With the current state of the world, people need to do a bunch of archaeology to even figure out which question they need to be asking. No longer is it “how do I fix this problem with setup.py”, now it’s “What tool do I need to be asking a question about?”, which in my opinion is actually a worse situation to be in from the end user’s perspective.
I spend hours every week keeping up to date with what’s changing in the general technical landscape, and I have no idea whatsoever what is going on with Python’s packaging ecosystem. Compare that to things like:
I have a great idea of what is going on in those projects just by reading sites like this one. There is a deeper problem at play here than “the author didn’t do their homework”. It might contribute to the problem, but it’s not the sole/root cause IMO.
It is not though. Not in the sense that it’s of any interest if you’re not affected or impacted.
So what? Do you keep abreast of developments in APL?
No, but I would very much argue people don’t value stability enough. You can’t build on sand, and people are rightfully annoyed when you dynamite their homes and move them to a tent city even if you promise eventual space habitats.
You are being ridiculously confrontational here. Are you the OP?
The parent is being ridiculously confrontational because what you’re saying is ridiculous.
Why is it ridiculous to expect that a package maintainer pays attention to their tools more frequently than once every five years? Based off the OP’s comments in the article while working through the transition, I would’ve assumed they’ve never touched Python before.
It sounds to me like @masklinn is salty because a previously working system has been deprecated before its replacement is ready, which is an entirely reasonable thing to be salty about. And your response is to say that the complaints are invalid, and that the solution to missing documentation is to pay attention to the inner details of a work in progress. It wouldn’t be necessary to follow all the many python packaging projects if they worked and were documented properly.
I can assure you Gregory has touched Python before - they are responsible for some extremely influential and cutting-edge projects. That’s what makes their feedback here particularly notable.
In OP’s defense, if you’re maintaining a project for a decade after a while you have the working build scripts and so aren’t confronted with the new stuff for a very very very long time.
So you end up with two problems:
I do feel like in a universe where he’s starting a new package, this blog is half as long. But well.. he’s not.
I think that every new tool needs a cookbook for common patterns from older tools. “What about this part of setup.py?” “It’s this now” or “this isn’t supported, you need to customize your build engine” or “This is a non-issue in the new model”. Because most people are working on existing projects, not new ones.
One thing that often confuses me if that why do alternatives like flit, poetry, hatch, pdm, etc. exist? What does one do that the other or setuptools does not?
My understanding is that Poetry predates some of these standards slightly, and has certain benefits (automatically managing your venv, lockfiles). Flit exists to be extremely small and simple and require the least from you to get it working—but provides the least benefit if you start needing C extensions, for instance. Hatch seems to have come later and be very standards driven while providing more than (say) flit or build, but not as much as Poetry.
I don’t believe that it really serves the Python community overall, having a proliferation of options here. Nobody really cares how big or small cargo or cabal are. You use the standard thing and get on with life.
One (setuptools) is a mechanism for building and installing packages. The others are project managers that also include features like dependency resolution, lockfiles, et al. for repeatable builds.