This is an interesting take on solving the egg and chicken problem of getting a python development environment installed. I have some concerns, like reinventing some substantial part of pip, and doing so on not-python. Is that a good idea? But Python was always thought-out so it could talk easy with faster languages, so, maybe it’s not a big deal.
As with a lot of stuff in this space, it will come down to what gets adoption. But from a cursory glance, I think I wouldn’t be made if this did.
An interesting avenue here is that the Rust ecosystem tends to end up with some high quality crates that can be easily shared across different tools that are adjacent (e.g. how ruff uses the RustPython parser, or shared libraries coming out of the git-branchless/jujutsu/sapling work). The posy README already mentions the pubgrub crate for dependency resolution for example.
In general though, I’m pretty excited about this. The current setup of package/tooling/etc. managers for Python works but it always feels like we’re taking a slight step back when coming from a Rust world. In particular, the idea of being able to do posy run ... and having it use the correct python version without needing an extra tool installed such as pyenv and ensuring all your dependencies are installed and there’s no interference with the global env (something which to my understanding PEP-582 doesn’t address) could be a significant developer experience boost.
Yeah, I love pyenv, best tool available for managing multiple python versions. But I don’t even have the words in English to describe how I feel about how it actually works, so I have to resort to my native language and call it a “gambiarra do caralho”.
Very interesting, thanks for the submission! This does seem to have some similarties to pyoxidizer / pyembed or at least it looks to me that way (from the outside), that there would be a big potential to re-use code (given that both use Rust). Yet, Pyoxidizer is not mentioned anywhere, so I assume the two projects are separate.
Even though it would probably make sense to somehow separate the interpreter- from the env-management part, I’d think it would make sense for those two projects to re-use each others code in some way or other, but maybe I’m missing something?
I think that’s a bit of an unfair take. They are talking about making it as easy as possible for newbies to bootstrap a python environment in Windows/Linux/macOS. If your answer to that is Nix, the bootstrapping would become a nightmare.
So how do I setup for this project?
Either you install this whole operating system where everything works very different to what you’re used to, or you install this Nix cli tool. Ah you’re using Windows? Sorry, unless you start by installing this WSL thing.
Now, imagine you want to add a new dependency, well, you may need to either put it in pyproject and then use this glue called poetry2nix, that sometimes works and sometimes doesn’t, if it doesn’t work maybe you can add it directly using python3XPackages.package and if it’s not there, then good luck, you either open a patch to nixpkgs poetry2nix adding your package or learn how to package a python library and also contribute to nixpkgs. The other option is to create a virtualenv the old-fashioned way and then use nix as kind of a pyenv.
–
I don’t think any of that sounds better than what is being proposed in the link.
I don’t mean to dismiss the author’s work, but to point out the continued insular choices of the Python core teams. Instead of installing Nix, the author asks us to install Cargo and then go through a standard Rust-project workflow; they are comparable in complexity and extent.
You may choose to continue using pyproject.toml, Poetry, virtualenv, pyenv, etc. but the direct path is to use nix-shell to configure an entire development environment in an atomic step. The list of packages can be contained to a single line in a single file; here is an example from a homelab application which I updated recently.
Contributing to nixpkgs is not trivial, but it is not difficult either. Here is a recent PR I authored for adding a Python package. It’s shorter than a setup.py, in my experience! Also, you don’t have to contribute new packages to nixpkgs; instead, you can add them to your local Nix expressions on a per-project basis.
Please also keep in mind that all of this discussion is within the context of Python packaging difficulties. Languages without extension modules don’t require all of this effort; all we need instead is to install a runtime directly from an upstream package-builder, whether that’s a distro, vendor, or third-party packager. We should imagine that a language is either designed to have lots of extensions and be an integrator of features, or designed for monolithic applications which reimplement every feature and are suitable for whole-program transformation. Python picked both, and so gets neither.
Hypothetically, just as a thought experiment and nothing else, I think maybe a case could be made that running Nix inside WSL and cross-compiling from that to Windows might be sorta acceptable. I don’t think that’s a realistic thing to propose: it’s a ton of work for starters, and the payoff would be pretty dubious since you’d have this really long painful edit/test cycle.
WSL is like Electron: it makes it easy for a developer to provide something to the user, but the thing provided is much worse than a corresponding native-solution. I’d struggle to integrate a WSL app with my native Windows powershell scripts.
The longer answer is: python has supported Windows natively for over a decade (how well might be up for debate, but it was supported), it’s not reasonable for them to suddenly say “use Linux inside Windows or get fucked”, and it’s not reasonable to expect them to do so, either.
I don’t have any statistics, but I would bet that the vast majority of Windows users (corporate IT managed machines) can’t enable WSL. Python is actually very easy to install on Windows with the Microsoft Store. Requiring users to enable WSL and understand how to use linux would be a large obstacle.
Or just with the installer from python.org, which will install to AppData by default (I think? At least if you choose the install just for me option), so no admin permissions needed.
I don’t mean to dismiss the author’s work, but to point out the continued insular choices of the Python core teams.
No, you are doing what you always do: pushing your preferred tools as the only acceptable tools, such that all development on all other tools must cease and all people everywhere must adopt only and exclusively your preferred tools. And along the way you throw in the usual (un)healthy dose of bashing anyone who dares to develop other tools, since obviously it’s bad and wrong for them to do so when the One True Thing has already been invented and thus they must be doing that for bad reasons.
Sometimes you do this with PyPy versus CPython. Sometimes with functional progamming/category theory versus other paradigms. Sometimes with Nix versus literally everything. But it’s always the same basic dismissive/attacking approach.
I didn’t read your comment, and boy, there’s so much more I disagree with.
but the direct path is to use nix-shell
To whom? There’s a world of people for whom nix is a non starter. Everyone using Visual Studio. Or working on computers they don’t fully control (enterprise developers). Or people that like Bluetooth to work, so they can’t use Linux (this is half in jest, half serious). “There’s no silver bullet” applies to your favorite thing too.
all we need instead is to install a runtime directly from an upstream package-builder
What language is like that? Ruby has C extensions, JavaScript has them, Java has jni. Even go, which is famous for reinventing the wheel a lot, has cgo. In every single language that isn’t C you will, at some point, have problems trying to install a package that needs to compile something in another language.
The reason it happens so much more in python is actually kind of a feature, not a bug: python was designed to be easily extendable, specifically in C, although that feature was perhaps not as well designed as we would like, in hindsight.
We should imagine that a language is either designed to have lots of extensions and be an integrator of features, or designed for monolithic applications which reimplement every feature and are suitable for whole-program transformation.
Maybe in a perfect world but … I don’t think any languages really fit this binary, well, binarilly (?). At most some are more at one end than the other, but I’m honestly struggling to find utility in the whole classification really.
Everyone using Visual Studio. Or working on computers they don’t fully control (enterprise developers).
I don’t understand, why would Nix be a blocker in those contexts? If you don’t fully control the computer, wouldn’t you have trouble installing all the Rust thingies anyhow?
I don’t think that addresses my question. I genuinely don’t get why Nix would be a blocker to people using Visual Studio (VS Code(?)), are plugins sandboxed, or unable to interact with binaries/run commands in some other way?
You’re applying things I said about one thing to to other things that I didn’t for it to be applied to.
One of my disagreements is with the idea that nix is some sort of ideal goal that ever developer is converting to. This idea breaks down as soon as you realize that people writing C# on Visual Studio (not VSCode) will never adopt something like nix, unless it’s fully integrated with Windows, like every single other tool they use.
The other disagreement is with the idea that the way the project linked can currently be used is the final interface: it clearly isn’t, they clearly say it will be a single binary in the future.
Only the second one has anything to do with Python tools. The first one is just a criticism of the idea of nix as the best thing ever that everyone should use and can do no wrong.
The Vision
The goal is for posy to act as a kind of high-level frontend to python: you install posy, then run posy [args] some_python.py and it takes care of everything up until entering the python interpreter. That includes:
installing Python (posy is a pure-rust single-file binary; it doesn’t assume you have anything else installed) (…)
I for one would like to strongly encourage anyone who would like to make an attempt at “reinventing Nix”, since a thing that is like Nix but avoids some of its pain points could potentially be delightful.
This is an interesting take on solving the egg and chicken problem of getting a python development environment installed. I have some concerns, like reinventing some substantial part of pip, and doing so on not-python. Is that a good idea? But Python was always thought-out so it could talk easy with faster languages, so, maybe it’s not a big deal.
As with a lot of stuff in this space, it will come down to what gets adoption. But from a cursory glance, I think I wouldn’t be made if this did.
An interesting avenue here is that the Rust ecosystem tends to end up with some high quality crates that can be easily shared across different tools that are adjacent (e.g. how ruff uses the RustPython parser, or shared libraries coming out of the git-branchless/jujutsu/sapling work). The posy README already mentions the pubgrub crate for dependency resolution for example.
In general though, I’m pretty excited about this. The current setup of package/tooling/etc. managers for Python works but it always feels like we’re taking a slight step back when coming from a Rust world. In particular, the idea of being able to do
posy run ...and having it use the correct python version without needing an extra tool installed such as pyenv and ensuring all your dependencies are installed and there’s no interference with the global env (something which to my understanding PEP-582 doesn’t address) could be a significant developer experience boost.Yeah, I love pyenv, best tool available for managing multiple python versions. But I don’t even have the words in English to describe how I feel about how it actually works, so I have to resort to my native language and call it a “gambiarra do caralho”.
Very interesting, thanks for the submission! This does seem to have some similarties to pyoxidizer / pyembed or at least it looks to me that way (from the outside), that there would be a big potential to re-use code (given that both use Rust). Yet, Pyoxidizer is not mentioned anywhere, so I assume the two projects are separate.
Even though it would probably make sense to somehow separate the interpreter- from the env-management part, I’d think it would make sense for those two projects to re-use each others code in some way or other, but maybe I’m missing something?
So, reinventing Nix?
I think that’s a bit of an unfair take. They are talking about making it as easy as possible for newbies to bootstrap a python environment in Windows/Linux/macOS. If your answer to that is Nix, the bootstrapping would become a nightmare.
So how do I setup for this project?Either you install this whole operating system where everything works very different to what you’re used to, or you install this Nix cli tool. Ah you’re using Windows? Sorry, unless you start by installing this WSL thing.
Now, imagine you want to add a new dependency, well, you may need to either put it in pyproject and then use this glue called poetry2nix, that sometimes works and sometimes doesn’t, if it doesn’t work maybe you can add it directly using
python3XPackages.packageand if it’s not there, then good luck, you either open a patch to nixpkgs poetry2nix adding your package or learn how to package a python library and also contribute to nixpkgs. The other option is to create a virtualenv the old-fashioned way and then use nix as kind of a pyenv.–
I don’t think any of that sounds better than what is being proposed in the link.
I don’t mean to dismiss the author’s work, but to point out the continued insular choices of the Python core teams. Instead of installing Nix, the author asks us to install Cargo and then go through a standard Rust-project workflow; they are comparable in complexity and extent.
You may choose to continue using
pyproject.toml, Poetry, virtualenv, pyenv, etc. but the direct path is to use nix-shell to configure an entire development environment in an atomic step. The list of packages can be contained to a single line in a single file; here is an example from a homelab application which I updated recently.Contributing to nixpkgs is not trivial, but it is not difficult either. Here is a recent PR I authored for adding a Python package. It’s shorter than a
setup.py, in my experience! Also, you don’t have to contribute new packages to nixpkgs; instead, you can add them to your local Nix expressions on a per-project basis.Please also keep in mind that all of this discussion is within the context of Python packaging difficulties. Languages without extension modules don’t require all of this effort; all we need instead is to install a runtime directly from an upstream package-builder, whether that’s a distro, vendor, or third-party packager. We should imagine that a language is either designed to have lots of extensions and be an integrator of features, or designed for monolithic applications which reimplement every feature and are suitable for whole-program transformation. Python picked both, and so gets neither.
I don’t understand how Nix would even be an alternative option when the goal is to support MacOS, Linux, and Windows.
Hypothetically, just as a thought experiment and nothing else, I think maybe a case could be made that running Nix inside WSL and cross-compiling from that to Windows might be sorta acceptable. I don’t think that’s a realistic thing to propose: it’s a ton of work for starters, and the payoff would be pretty dubious since you’d have this really long painful edit/test cycle.
Doesn’t WSL solve that problem out of the gate?
WSL is like Electron: it makes it easy for a developer to provide something to the user, but the thing provided is much worse than a corresponding native-solution. I’d struggle to integrate a WSL app with my native Windows powershell scripts.
The short answer is no.
The longer answer is: python has supported Windows natively for over a decade (how well might be up for debate, but it was supported), it’s not reasonable for them to suddenly say “use Linux inside Windows or get fucked”, and it’s not reasonable to expect them to do so, either.
I don’t have any statistics, but I would bet that the vast majority of Windows users (corporate IT managed machines) can’t enable WSL. Python is actually very easy to install on Windows with the Microsoft Store. Requiring users to enable WSL and understand how to use linux would be a large obstacle.
Or just with the installer from python.org, which will install to AppData by default (I think? At least if you choose the install just for me option), so no admin permissions needed.
All the cool kids use
wingetNo, you are doing what you always do: pushing your preferred tools as the only acceptable tools, such that all development on all other tools must cease and all people everywhere must adopt only and exclusively your preferred tools. And along the way you throw in the usual (un)healthy dose of bashing anyone who dares to develop other tools, since obviously it’s bad and wrong for them to do so when the One True Thing has already been invented and thus they must be doing that for bad reasons.
Sometimes you do this with PyPy versus CPython. Sometimes with functional progamming/category theory versus other paradigms. Sometimes with Nix versus literally everything. But it’s always the same basic dismissive/attacking approach.
Maybe don’t do that?
I really appreciate this comment. It helps me understand you.
I didn’t read your comment, and boy, there’s so much more I disagree with.
To whom? There’s a world of people for whom nix is a non starter. Everyone using Visual Studio. Or working on computers they don’t fully control (enterprise developers). Or people that like Bluetooth to work, so they can’t use Linux (this is half in jest, half serious). “There’s no silver bullet” applies to your favorite thing too.
What language is like that? Ruby has C extensions, JavaScript has them, Java has jni. Even go, which is famous for reinventing the wheel a lot, has cgo. In every single language that isn’t C you will, at some point, have problems trying to install a package that needs to compile something in another language.
The reason it happens so much more in python is actually kind of a feature, not a bug: python was designed to be easily extendable, specifically in C, although that feature was perhaps not as well designed as we would like, in hindsight.
Maybe in a perfect world but … I don’t think any languages really fit this binary, well, binarilly (?). At most some are more at one end than the other, but I’m honestly struggling to find utility in the whole classification really.
I don’t understand, why would Nix be a blocker in those contexts? If you don’t fully control the computer, wouldn’t you have trouble installing all the Rust thingies anyhow?
https://lobste.rs/s/j9gr9z/pybi_posy#c_gnarky
I don’t think that addresses my question. I genuinely don’t get why Nix would be a blocker to people using Visual Studio (VS Code(?)), are plugins sandboxed, or unable to interact with binaries/run commands in some other way?
You’re applying things I said about one thing to to other things that I didn’t for it to be applied to.
One of my disagreements is with the idea that nix is some sort of ideal goal that ever developer is converting to. This idea breaks down as soon as you realize that people writing C# on Visual Studio (not VSCode) will never adopt something like nix, unless it’s fully integrated with Windows, like every single other tool they use.
The other disagreement is with the idea that the way the project linked can currently be used is the final interface: it clearly isn’t, they clearly say it will be a single binary in the future.
Only the second one has anything to do with Python tools. The first one is just a criticism of the idea of nix as the best thing ever that everyone should use and can do no wrong.
This is clearly a very early stage prototype, I didn’t see any claims that this is the final interface they people should use today.
In fact, straight from the readme:
Has nix managed to one-up rust on its evangelism task force?
Not that I know a lot about nix but the only similarity I see between nix and this proposal is that they are both made with code?
Ok, to be fair, they are made with code and related to managing packages. And they mention immutability somewhere in their description.
So, 3 similarities. Maybe I am wrong. Still sounds like a far-fetched comparison.
I for one would like to strongly encourage anyone who would like to make an attempt at “reinventing Nix”, since a thing that is like Nix but avoids some of its pain points could potentially be delightful.