The tl;dr is that PEP 668 provides a mechanism – a specially-named sentinel file at a particular location – to declare a Python installation/environment as “externally-managed” (its packages are installed and updated via some non-pip package manager, such as apt). Recent versions of pip respect this and will refuse to install packages into such an “externally-managed” Python environment.
Debian are marking their default system Python as “externally managed”, so any workflows on any Debian-based distros which assumed the ability to do a pip install with the system Python will be broken and will have to either switch to using apt to install the desired packages, or create and use a Python virtual environment to pip install into.
Posting this today because Ubuntu 23.04 has been out for a few days and people are apparently tripping over this as they upgrade to it.
(I am and long have been on Team Use A Venv, Always, Yes, Even In A Container, Really” and recommend you join us)
so any workflows on any Debian-based distros which assumed the ability to do a pip install with the system Python will be broken
From my experience, this was already broken in practice. The moment you did that on a Debian box the python situation was so messed up you could rather reinstall than try to fix it.
It’s going to be a problem mostly for Debian-based containers, where you treat the system Python as your environment (for good reason - you’ll never upgrade afterwards anyway).
As specified in the PEP, the process for a compliant installer is that if it detects it’s not running in a venv and it finds the EXTERNALLY-MANAGED sentinel file at the correct location, it’s supposed to instantly bail out and display an error message (and the sentinel file can provide the error message to display).
It’s a weird state that system python’s have to be there (for the system) but are effectively always too broken to use (same on macOS and even with Homebrew).
Thanks for posting this. This is much better feedback than a “the filesystem is read-only” type of error. I went ahead and implemented it for nixpkgs as well. https://github.com/NixOS/nixpkgs/pull/229166
I always install in my home directory (I’m not a monster) but having packages in my home directory installed at different times, with different python versions has caused me all kinds of low level annoying problems.
Yes. But if you pass ‘–break-system-packages’ you get the same behavior as before, you just need to set a scary-sounding option :-)
You can set an alias if you want that.
It’s unclear from the context; are they pushing this change out to stable releases, or is this something that’s going to happen in the next stable release? Changing it in existing releases is gonna break a whole lot of folks’ Dockerfiles and other automation.
The tl;dr is that PEP 668 provides a mechanism – a specially-named sentinel file at a particular location – to declare a Python installation/environment as “externally-managed” (its packages are installed and updated via some non-
pippackage manager, such asapt). Recent versions ofpiprespect this and will refuse to install packages into such an “externally-managed” Python environment.Debian are marking their default system Python as “externally managed”, so any workflows on any Debian-based distros which assumed the ability to do a
pip installwith the system Python will be broken and will have to either switch to usingaptto install the desired packages, or create and use a Python virtual environment topip installinto.Posting this today because Ubuntu 23.04 has been out for a few days and people are apparently tripping over this as they upgrade to it.
(I am and long have been on Team Use A Venv, Always, Yes, Even In A Container, Really” and recommend you join us)
From my experience, this was already broken in practice. The moment you did that on a Debian box the python situation was so messed up you could rather reinstall than try to fix it.
That’s a big part of why this feature was developed.
It’s going to be a problem mostly for Debian-based containers, where you treat the system Python as your environment (for good reason - you’ll never upgrade afterwards anyway).
That’s definitely going to break some stuff at work… but I’m happy to see it.
sudo pip installcauses no end of trouble already.I’m curious if this will also affect
pip installinto the user home. I’d love it if I could prevent that from working too.As specified in the PEP, the process for a compliant installer is that if it detects it’s not running in a venv and it finds the
EXTERNALLY-MANAGEDsentinel file at the correct location, it’s supposed to instantly bail out and display an error message (and the sentinel file can provide the error message to display).However, a user with sufficient system permissions can still work around this by passing an explicit
--break-system-packagesflag topip install. So it’s not a full lockdown.I think I will keep quiet about the
--break-system-packagesescape-valve unless there’s an emergency. :-)I wonder if I could add an
EXTERNALLY-MANAGEDfile to my home directory somewhere?The name of that flag is perfect.
It’s a weird state that system python’s have to be there (for the system) but are effectively always too broken to use (same on macOS and even with Homebrew).
Thanks for posting this. This is much better feedback than a “the filesystem is read-only” type of error. I went ahead and implemented it for nixpkgs as well. https://github.com/NixOS/nixpkgs/pull/229166
To everyone asking, yes it prevents
pip install --useras well. The most recommended solution is pipx (sudo apt install pipx).I am not gonna be able to install pipx using the recommended instructions though.
This is gonna be a mess.
apt install pipxWhich is why I recommended installing it from system packages.
Does this prevent installing packages on my home directory as well? That would be annoying.
I always install in my home directory (I’m not a monster) but having packages in my home directory installed at different times, with different python versions has caused me all kinds of low level annoying problems.
Use pyenv to build pythons that you control in your home directory.
Yes. But if you pass ‘–break-system-packages’ you get the same behavior as before, you just need to set a scary-sounding option :-) You can set an alias if you want that.
This is going to break some stuff, but I think it is a good move.
It’s unclear from the context; are they pushing this change out to stable releases, or is this something that’s going to happen in the next stable release? Changing it in existing releases is gonna break a whole lot of folks’ Dockerfiles and other automation.
Next stable. Ubuntu 23.04 (Lunar) which just came out, and Debian 12 (bookworm).
…next Ubuntu version update at work that I have to manage will be a bit spicy, won’t it. XD
Maybe I can finally make people stop making dysfunctional build workflows though.