I’m a little confused by what it does. It gives a Rust wrapper library for embedding CPython and also gives a single static binary for distributing CPython (wrapped in it’s Rust library). So the primary benefits are easier interop with Rust programs and easier distribution because it’s a single binary to get “CPython”.
This is approximately my understanding. The related PyOxidizer project seems to be like PyInstaller except that its built on Rust infrastructure and has some other tradeoffs (in-memory loading of packages vs. unpacking to temp directory?). I think I could use another blog post, etc., to better explain when/why I’d reach for this toolset.
In my understanding PyOxidizer is an “umbrella” project featuring many of what you’ve described, plus probably much more.
However, strictly speaking of pyoxy, what the article describes it seems to serve the following use-case:
having a self-contained Python run-time (CPython 3.9 at the moment it seems); self-contained in the sense that you don’t need anything else except that executable (and your code) to run it; (at the moment not being statically linked, you also need glibc on Linux, but this is not a major issue for a first release;)
that self-contained executable can be relocated anywhere on your file-system, thus deploying a Python-based application in an uncontrolled environment becomes easier;
being a single file, it’s also lighter on the file-system, especially at starup, as there aren’t hundreds of small files to be found and read off disk;
Basically if before I wasn’t too comfortable writing operations scripts in Python, as opposed to bash, because I never knew which turn the Python on my distribution would take, now I can have the pyoxy stored alongside my scripts, and be sure they’ll keep on running as long as I need them, without needlessly breaking due to Python upgrades.
Besides having the Python runtime (with pyoxy run-python), it also allows the user to run pyoxy run-yaml, where the yaml stems from the fact that the user can provide a custom YAML file to instruct the interpreter how it should be set-up. For example I usually use #!/usr/bin/env -S python2.7 -u -O -O -B -E -S -s -R in my scripts, and this always leaves a mess in ps and htop; with the new feature I would use something like #!/usr/bin/env pyoxy run-yaml (see the article), and the process tree would be much cleaner, and also I would have more control over the interpreter setup.
Fully statically linked Linux binaries are possible. PyOxidizer supports it. But you can’t load compiled extension modules (.so files) from static binaries. Since I was initially going for a drop-in replacement for “python, I didn’t want to sacrifice ecosystem compatibility.
You could perhaps provide two kinds of builds, at least on Linux, one dynamic as the current one, and one static for those that need maximum portability but are OK with the limitation of running only Python native code.
As a pyenv fan, a single binary multi platform python distribution seems awesome. Is it statically linked?
Speaking of pyenv, would be cool if the python “payload” could be feed separately to the pyoxy binary. Then, if I wanted multiple python versions (historically, a pain in the ass to get on Linux and Mac), I could have one pyoxy binary and multiple python “bundles”.
I guess I could also just get pyoxys with different python versions embedded, but that feels less “neat” for some reason?
It seems this use-case (having pyoxy as separate of the user’s bundled Python code) you are describing is on the roadmap.
However even as it is right now I think it is able to run a zipapp (i.e. a Zip file that contains a __main__.py file in the root of the archive): pyoxy run-python ./app.zip
Regarding the static linking, the Linux binary is not statically linked, however it needs nothing more than Linux’s glibc, thus you can easily copy-paste it between any “recent” Linux distribution without issues:
I’m a little confused by what it does. It gives a Rust wrapper library for embedding CPython and also gives a single static binary for distributing CPython (wrapped in it’s Rust library). So the primary benefits are easier interop with Rust programs and easier distribution because it’s a single binary to get “CPython”.
Do I have that right?
This is approximately my understanding. The related PyOxidizer project seems to be like PyInstaller except that its built on Rust infrastructure and has some other tradeoffs (in-memory loading of packages vs. unpacking to temp directory?). I think I could use another blog post, etc., to better explain when/why I’d reach for this toolset.
In my understanding PyOxidizer is an “umbrella” project featuring many of what you’ve described, plus probably much more.
However, strictly speaking of
pyoxy, what the article describes it seems to serve the following use-case:glibcon Linux, but this is not a major issue for a first release;)Basically if before I wasn’t too comfortable writing operations scripts in Python, as opposed to
bash, because I never knew which turn the Python on my distribution would take, now I can have thepyoxystored alongside my scripts, and be sure they’ll keep on running as long as I need them, without needlessly breaking due to Python upgrades.Besides having the Python runtime (with
pyoxy run-python), it also allows the user to runpyoxy run-yaml, where theyamlstems from the fact that the user can provide a custom YAML file to instruct the interpreter how it should be set-up. For example I usually use#!/usr/bin/env -S python2.7 -u -O -O -B -E -S -s -Rin my scripts, and this always leaves a mess inpsandhtop; with the new feature I would use something like#!/usr/bin/env pyoxy run-yaml(see the article), and the process tree would be much cleaner, and also I would have more control over the interpreter setup.Fully statically linked Linux binaries are possible. PyOxidizer supports it. But you can’t load compiled extension modules (.so files) from static binaries. Since I was initially going for a drop-in replacement for “python, I didn’t want to sacrifice ecosystem compatibility.
The glibc based builds I believe conform to Python’s manylinux2014 platform level and are about as portable as you can get before going fully static. https://pyoxidizer.readthedocs.io/en/latest/pyoxy_installing.html#linux
You could perhaps provide two kinds of builds, at least on Linux, one dynamic as the current one, and one static for those that need maximum portability but are OK with the limitation of running only Python native code.
All in all, great work!
As a pyenv fan, a single binary multi platform python distribution seems awesome. Is it statically linked?
Speaking of pyenv, would be cool if the python “payload” could be feed separately to the pyoxy binary. Then, if I wanted multiple python versions (historically, a pain in the ass to get on Linux and Mac), I could have one pyoxy binary and multiple python “bundles”.
I guess I could also just get pyoxys with different python versions embedded, but that feels less “neat” for some reason?
Anyway, cool stuff.
It seems this use-case (having
pyoxyas separate of the user’s bundled Python code) you are describing is on the roadmap.However even as it is right now I think it is able to run a
zipapp(i.e. a Zip file that contains a__main__.pyfile in the root of the archive):pyoxy run-python ./app.zipRegarding the static linking, the Linux binary is not statically linked, however it needs nothing more than Linux’s
glibc, thus you can easily copy-paste it between any “recent” Linux distribution without issues:I think it is possible to have it statically linked (at least on Linux), but given the above I think it’s enough for a first release. :)