This is literally an attempt to prevent people from breaking their installs, by shutting down one of the most common ways that happens (installing packages into the “system” Python).
I’m honestly not even sure at this point what would make people happy – if something doesn’t get fixed, people complain it’s broken, and if it does get fixed people complain that it’s a change from previous behavior.
The Python packaging ecosystem is so busted, fragmented, and incompatible that people are breaking it all the time by doing innocuous things.
They release a “fix” for this which breaks my own Python packaging environment in the process, taking me a ton of time and effort to set up a virtual environment and port our service to work inside of it with Docker etc.
The Python packaging ecosystem is so busted, fragmented, and incompatible that people are breaking it all the time by doing innocuous things.
The Python packaging ecosystem has standard tools that work quite reliably. The main problem is people who refuse to use them, build their own rickety Jenga tower of stuff as an alternative, and then blame “Python packaging” when it doesn’t work.
They release a “fix” for this which breaks my own Python packaging environment in the process, taking me a ton of time and effort to set up a virtual environment and port our service to work inside of it with Docker etc.
For years now, pip has tried to detect this kind of situation and issue a warning reminding you that installing into the global “system” Python is a bad idea. And every best-practices guide for how to package and containerize Python that I’m aware of has recommended avoiding installing into the “system” Python and instead creating a venv.
But finally, distros who are tired of getting bug reports from people who did this and broke their system are just enforcing a blunt “nope, you can’t do that”. As I see it, you’ve had ample warning and ample time to learn about and adopt the clear recommended best practice (which is not some sort of constantly-changing thing – “use a venv” has been the advice for years and years now). I don’t see how it’s Python’s fault that you refused up until now.
That dude who founded Sentry just released a new one like 2 weeks ago as well and my Twitter is full of people all hyped about it.
I’m not even trying to say that this fix from Debian totally disabling pip is a bad thing. I’m just complaining about how hilariously messed up the whole Python packaging ecosystem is and what a pain it is to work with it.
There’s literally a website dedicated to listing the dozens of different options for just package management
Which doesn’t contradict what I said. There are in fact standard reliable tools: setuptools as the package build backend, pip as the package installer/uninstaller, and venv as the isolation mechanism.
And like I said, a huge percentage of perceived issues with “Python packaging” are people deciding to go off and build their own thing instead. At which point the pain they deal with is entirely their fault and nobody else’s, because the standard tools really do work and are not hard at all to use.
Really it’s like if people insisted Rust packaging was some horrid hellscape and the moment you asked them what problems they have with Cargo they’re like “Cargo? I don’t use that…”`
I was just using the standard tools (pip) and they broke unexpectedly and took a good amount of effort to fix.
People have built these myriad alternatives because they have gripes with the default tools, like pip installing stuff with root permissions at the system level, or virtualenv causing 8 different identical 2GB copies of Tensorflow to get installed on my system in various random subdirs, etc.
It’s very strange that you choose to defend Python packaging of all things. It’s notoriously bad; people make memes about it. And in the ~8 years I’ve been programming, it’s gotten worse if anything - not even counting the 2->3 transition that people are still paying down the costs of like 15 years later.
I think a part of a problem/perception is that there is quite a lot of blogposts each advocating different approaches. When I was searching some time ago, almost nothing pointed to authoritative sources (at least not the top results where from python.org or pypa.org). When I looked for an inspiration how my colleagues approach building/testing/publishing a python package/app, I was again met with thousands ways of entering the world of hurt — because misinformation spreads easily and is hard to fix.
It’s a Docker container. It is the whole universe as far as packing is concerned. The fact that I have to create a virtual environment inside my Docker container so I can install a Snowflake Python library is ridiculous to me.
It’s a Docker container. It’s an entire isolated operating system, and it turns out a lot of Linux distros use and rely on tools written in Python, which makes messing with the modules available to the default system Python a very bad idea. So you use a venv, which is the standard recommended thing and has been for years.
Heh, this is ironic, no? Docker containers are there to provide packaging, isolation, and distribution of apps. The fact that many use them as a replacement for a VM just shows people don’t follow the best practices. Just the same as with pip…
If one uses docker containers in the right way, having a venv inside of it really feels wrong, because the container should be an unit of isolation.
Like I said, the issue is that Python is both a language people want to write applications in, and a language distros want to write system tooling in. Nobody would be surprised if you told them that replacing key .so files in a system shared location might break things. Why are people surprised if you tell them that replacing key .py files in a system shared location might break things?
So the Docker container is not, in itself, a sufficient unit of isolation for Python code, and never was. Which is why creating a venv is such a strongly recommended practice (and now enforced by some distros).
The fact that distro managed packages for language libraries is a thing at all seems ridiculous to me. It’s a losing game. Some libraries end up specially anointed because they have a maintainer for some distro, some dont, there’s no relation between the annoited ones and the good ones*, and the distro ones are always out of date. It puts languages in a weird place because they want to support their own packaging ecosystem, and also interoperable with a ton of distros, which is a drawn and quartered specification. So like, either distros manage packages and anything else you have to install from source (hope your distro has the right build system and compiler, resolve your deps manually), or languages have environments they can self manage. NPM spearheaded this wacky idea of “why not both” in which if you sudo it, it will futz around in directories that really should be distro managed to fake up a “global install” concept, and python followed. This has since been regarded as a bad idea, and this is the band-aid rip. It never was what it pretended to be, you got lucky if it ever worked for you, I haven’t used snowflake, maybe they figured something out I didn’t see anywhere else.
for a fun time, find the python fuse library that works on freebsd. There is one. It’s not the one in ports.
Edit: I don’t know how to make the asterisk not a bullet :/
NB: PEP 668 prescribes, and pip provides, a --break-system-packages flag for users who want to use pip to manage an EXTERNALLY-MANAGED package library.
I guess an alternative (and better?) approach is to create a separate virtual environment for each system package, or even better, have all packages ship their own virtual environment. From my understanding, many Linux distributions maintain a so-called “system Python”, e.g. /usr/bin/python3, and all system packages share that particular Python distribution, which is clearly suboptimal. For example, the package update-manager depends on python3-yaml==5.3.1, but what if another system package depends on python3-yaml==6.0.0? You get a version conflict, and PEP 668 doesn’t help that.
Essentially, PEP 668 says that “system Python” should not be touched by the user, but I argue that such a globally mutable “system Python” shouldn’t even exist.
I think that the idea is that any scripts that rely on the system python should not use external packages at all. And if they need to use an external package, it should be installed and managed through the systems package manager and not pip.
Yeah, I proposed this idea because I find myself using pipx much oftener than apt when installing Python-based packages like pipenv, icdiff, black, etc.
Spent 4 hours last week fixing our CI to be able to install some modules again.
Just one more page in the saga of Python’s strange, convoluted, and always-broken package management ecosystem.
This is literally an attempt to prevent people from breaking their installs, by shutting down one of the most common ways that happens (installing packages into the “system” Python).
I’m honestly not even sure at this point what would make people happy – if something doesn’t get fixed, people complain it’s broken, and if it does get fixed people complain that it’s a change from previous behavior.
The Python packaging ecosystem is so busted, fragmented, and incompatible that people are breaking it all the time by doing innocuous things.
They release a “fix” for this which breaks my own Python packaging environment in the process, taking me a ton of time and effort to set up a virtual environment and port our service to work inside of it with Docker etc.
Quite a sad state of affairs
pytnon3 -m venvand sourcing the resultant activation scripts as part of the CI setup has been the right answer since dubstep was cool…What’s sad about ensuring nobody has to struggle with what we did ever again?
Couldn’t you have told me this when dubstep got cool ? Finding out about this now, after just going full on pip install..
Well, if it’s any consolation, dubstep seeps poised for a comeback, so in a way, I did ;)
The Python packaging ecosystem has standard tools that work quite reliably. The main problem is people who refuse to use them, build their own rickety Jenga tower of stuff as an alternative, and then blame “Python packaging” when it doesn’t work.
For years now,
piphas tried to detect this kind of situation and issue a warning reminding you that installing into the global “system” Python is a bad idea. And every best-practices guide for how to package and containerize Python that I’m aware of has recommended avoiding installing into the “system” Python and instead creating a venv.But finally, distros who are tired of getting bug reports from people who did this and broke their system are just enforcing a blunt “nope, you can’t do that”. As I see it, you’ve had ample warning and ample time to learn about and adopt the clear recommended best practice (which is not some sort of constantly-changing thing – “use a venv” has been the advice for years and years now). I don’t see how it’s Python’s fault that you refused up until now.
There’s literally a website dedicated to listing the dozens of different options for just package management: https://chadsmith.dev/python-packaging/
That dude who founded Sentry just released a new one like 2 weeks ago as well and my Twitter is full of people all hyped about it.
I’m not even trying to say that this fix from Debian totally disabling pip is a bad thing. I’m just complaining about how hilariously messed up the whole Python packaging ecosystem is and what a pain it is to work with it.
Which doesn’t contradict what I said. There are in fact standard reliable tools:
setuptoolsas the package build backend,pipas the package installer/uninstaller, andvenvas the isolation mechanism.And like I said, a huge percentage of perceived issues with “Python packaging” are people deciding to go off and build their own thing instead. At which point the pain they deal with is entirely their fault and nobody else’s, because the standard tools really do work and are not hard at all to use.
Really it’s like if people insisted Rust packaging was some horrid hellscape and the moment you asked them what problems they have with Cargo they’re like “Cargo? I don’t use that…”`
I was just using the standard tools (pip) and they broke unexpectedly and took a good amount of effort to fix.
People have built these myriad alternatives because they have gripes with the default tools, like pip installing stuff with root permissions at the system level, or virtualenv causing 8 different identical 2GB copies of Tensorflow to get installed on my system in various random subdirs, etc.
It’s very strange that you choose to defend Python packaging of all things. It’s notoriously bad; people make memes about it. And in the ~8 years I’ve been programming, it’s gotten worse if anything - not even counting the 2->3 transition that people are still paying down the costs of like 15 years later.
You may have been using standard tools, but you weren’t using them correctly for whatever reason.
Now you are forced to either use them correctly (which is a good thing) or add a single flag to keep using them incorrectly at your own risk.
Not really rant worthy.
I think a part of a problem/perception is that there is quite a lot of blogposts each advocating different approaches. When I was searching some time ago, almost nothing pointed to authoritative sources (at least not the top results where from python.org or pypa.org). When I looked for an inspiration how my colleagues approach building/testing/publishing a python package/app, I was again met with thousands ways of entering the world of hurt — because misinformation spreads easily and is hard to fix.
4 hours? And you don’t have to deal with the consequences of folks doing ‘sudo pip install’ ever again? And you’re … complaining?
It’s a Docker container. It is the whole universe as far as packing is concerned. The fact that I have to create a virtual environment inside my Docker container so I can install a Snowflake Python library is ridiculous to me.
It’s a Docker container. It’s an entire isolated operating system, and it turns out a lot of Linux distros use and rely on tools written in Python, which makes messing with the modules available to the default system Python a very bad idea. So you use a venv, which is the standard recommended thing and has been for years.
Heh, this is ironic, no? Docker containers are there to provide packaging, isolation, and distribution of apps. The fact that many use them as a replacement for a VM just shows people don’t follow the best practices. Just the same as with pip…
If one uses docker containers in the right way, having a venv inside of it really feels wrong, because the container should be an unit of isolation.
Like I said, the issue is that Python is both a language people want to write applications in, and a language distros want to write system tooling in. Nobody would be surprised if you told them that replacing key
.sofiles in a system shared location might break things. Why are people surprised if you tell them that replacing key.pyfiles in a system shared location might break things?So the Docker container is not, in itself, a sufficient unit of isolation for Python code, and never was. Which is why creating a venv is such a strongly recommended practice (and now enforced by some distros).
The fact that distro managed packages for language libraries is a thing at all seems ridiculous to me. It’s a losing game. Some libraries end up specially anointed because they have a maintainer for some distro, some dont, there’s no relation between the annoited ones and the good ones*, and the distro ones are always out of date. It puts languages in a weird place because they want to support their own packaging ecosystem, and also interoperable with a ton of distros, which is a drawn and quartered specification. So like, either distros manage packages and anything else you have to install from source (hope your distro has the right build system and compiler, resolve your deps manually), or languages have environments they can self manage. NPM spearheaded this wacky idea of “why not both” in which if you sudo it, it will futz around in directories that really should be distro managed to fake up a “global install” concept, and python followed. This has since been regarded as a bad idea, and this is the band-aid rip. It never was what it pretended to be, you got lucky if it ever worked for you, I haven’t used snowflake, maybe they figured something out I didn’t see anywhere else.
Edit: I don’t know how to make the asterisk not a bullet :/
NB: PEP 668 prescribes, and pip provides, a
--break-system-packagesflag for users who want to use pip to manage anEXTERNALLY-MANAGEDpackage library.https://github.com/pypa/pip/pull/11780
I guess an alternative (and better?) approach is to create a separate virtual environment for each system package, or even better, have all packages ship their own virtual environment. From my understanding, many Linux distributions maintain a so-called “system Python”, e.g.
/usr/bin/python3, and all system packages share that particular Python distribution, which is clearly suboptimal. For example, the packageupdate-managerdepends onpython3-yaml==5.3.1, but what if another system package depends onpython3-yaml==6.0.0? You get a version conflict, and PEP 668 doesn’t help that.Essentially, PEP 668 says that “system Python” should not be touched by the user, but I argue that such a globally mutable “system Python” shouldn’t even exist.
I think that the idea is that any scripts that rely on the system python should not use external packages at all. And if they need to use an external package, it should be installed and managed through the systems package manager and not pip.
pipx - install and run Python applications in isolated environments
Yeah, I proposed this idea because I find myself using
pipxmuch oftener thanaptwhen installing Python-based packages like pipenv, icdiff, black, etc.