1. 35

  2. 3

    In today’s release, PyOxidizer is good at producing executables embedding Python. It doesn’t yet venture too far into the distribution part of the problem (I want it to be trivial to produce MSI installers, DMG images, deb/rpm packages, etc). But on Linux, this is already a huge step forward because PyOxidizer makes it easy (hopefully!) to produce binaries that should just work on other machines. (Anyone who has attempted to distribute Linux applications will tell you how painful this problem can be.)

    This is IMO a really good call.

    Packaging is itself another problem, but it’s a problem that’s been solved over and over again in different environments by different tools.

    For Unix-oid OSen you’ve got fpm and maybe other things exist in other ecosystems, I dunno.

    But making one self contained binary is a HUGE first step. Thank you for making it so easy! Now maybe some folks will stick with Python rather than immediately feeling the need to rewrite their codebase in Golang because it has self contained binaries! (Golang seems like a great tool, not a Golang slight, but Python is way behind the 8 ball here and this tool helps a lot!)

    1. 2

      Good effort, good luck with the project. Will try and keep an eye on :)

      1. 2

        PyOxidizer is a Rust application and requires Rust 1.33+ to be installed in order to build binaries.

        Hmm, it would have been nicer if PyOxidizer had been meta. i.e. it itself had a version that was self contained single file executable, so that those of us who are not interested in installing yet another language tool chain on our computers could grumble less.

        1. 7

          I acknowledged this in the post:

          It is a bit unfortunate that I force users to install Rust before using PyOxidizer, but in my defense the target audience is technically savvy developers, bootstrapping Rust is easy, and PyOxidizer is young, so I think it is acceptble for now.

          I will almost certainly provide pre-built executables once the project is more mature. Thank you for the feedback.

            1. 1

              That’s great! Always on the lookout for good ways to distribute Python code to the end user. I generally deal with CLI programs, but I’ve created PySide based programs and programs using other toolkits. The other tools I’ve used (PyInstaller, cx_Freeze type things) tend to not do well with some frameworks. Hope this will deal with those too!

              1. 1

                I applaud you for your strategy and tactics! Wonderfully done. I was thinking of similar for a different language. I will really have to deconstruct what you have done here.

                What was your inspiration? Are there similar systems? What is your long term goal?

                What would it take to support PyPy?

                1. 11

                  Inspiration was a few things.

                  First, I’m a core contributor to Mercurial, which is a massive, system’s level (mostly) Python application. From a packaging and distribution standpoint, my personal belief is that Python hinders Mercurial. We can’t aggressively adopt modern Python versions, there’s a lot of variance in Python execution environments that create a long tail of bugs, etc. On the performance front, Python’s startup overhead is pretty poor. This prevents things like hg status from being perceived as instantaneous. It also slows down the test suite by literally minutes on some machines. And packaging Mercurial for multiple platforms takes a lot of effort because this isn’t a solved problem for the Python ecosystem. While there are a lot of good things about Mercurial being implemented in Python (and I don’t want to be perceived as advocating for porting Mercurial away from Python - because I don’t), it feels like Mercurial is constantly testing the limits of Python on a few fronts. This isn’t good for Mercurial. It isn’t a good sign for the longevity of Python if they can’t “support” large, mature projects like Mercurial.

                  So a big source of inspiration was… frustration, specifically around how it felt that Python was limiting Mercurial’s potential.

                  Another source of inspiration was my general attitude of not accepting the status quo. I’m always thinking about why things have to be the way they are. A large part of PyOxidizer was me questioning “why can’t I have a single file executable that runs Python?” I helped maintain Firefox’s build system for several years and knew enough about the low-level build system bits to understand what the general problems with binary distribution were. I knew enough about CPython’s internals (from developing Python C extensions) that I had confidence to dive really deep to be able to implement PyOxidizer. I felt like I knew enough about some relatively esoteric systems (notably build systems and CPython internals) to realize that others who had ventured into the Python application packaging space were attempting to solve this problem within constraints given to them by how CPython is commonly compiled. I realized I possessed the knowledge to change the underlying system and to coerce it to do what I wanted (namely produce self-contained executables containing Python). In other words, I changed the rules and created a new opportunity for PyOxidizer to do something that nobody had ever done in the public domain (Google has produced self-contained Python executables for years using similar but sufficiently different techniques).

                  If you want to learn more about the technical journey, I suggest reading https://gregoryszorc.com/blog/2018/12/18/distributing-standalone-python-applications/.

                  As for similar systems, other than WASM, I’m not aware of other “interpreted/scripting languages” (like Python) that have solutions that do what PyOxidizer does. I’m sure they exist. But Python is the only language in this language space that I’ve used significantly in the past ~8 years and I’m not sure what other languages have implemented. Obviously you can do these single executable tricks in compiled languages like Go and Rust.

                  My long term goal is to make Python application distribution painless. A secondary goal is to make Python applications developed with PyOxidizer “better” than normal Python applications. This can be done through features like an integrated command server and providing Rust and Python access to each other’s capabilities. I want Python application maintainers to focus on building great applications, not worry about packaging/distribution.

                  PyPy could theoretically be supported if someone produces a Python distribution conforming to the format documented by https://github.com/indygreg/python-build-standalone. In theory, PyOxidizer is agnostic about the flavor of the Python distribution it uses as long as that Python distribution provides object files that can be relinked and implements aspects of Python’s C API for interpreter control. There would likely need to be a few pieces in PyOxidizer, such as how the extension modules C array is defined. There’s probably a way to express this in python-build-standalone’s distribution descriptor JSON document such that it can be abstracted across distributions. I would very much like to support PyPy and I envision I will cross this bridge eventually. I think there are more important features to work on first, such as compiling C extensions and actually making distribution easy.

                  1. 1

                    Thank you for such a detailed response.

                    I love that you have standardized an interface contract for Python runtimes.

                    This looks like it could give organizations confidence in their deployment runtime, while no longer being tied to specific distros and can start using more niche and esoteric libraries that might be difficult to install. This is a form of containerization for Python applications.

                    What I am really interested in, is because you own both sides of the system, is streamlining the bidirectional call boundary. Having control over the shell that runs on the host, the Rust wrapper and the VM there is an opportunity to short circuit some of the expense of calling into C or how data is laid out in memory. In a quick ripgrep through the code, I couldn’t find any reference to cffi. Do you plan on supporting cffi or is it already handled? I am really curious to learn about what your integration plans look like. Great work.

              2. 1

                That’s an awful lot of heavy lifting you’re asking from a tool maintainer.

                And, I mean, ‘brew/apt/yum install rust’ isn’t generally a particularly big ask of you, the end user :)

                1. 4

                  But, but, this is solving pip install x … don’t you think at least the irony should be acknowledged?

              3. 1

                Sorry if I missed this but care to elaborate if there are any advantages over pex besides multi-platform? This sounds exciting, but for the last few years I’ve been very happy with pex :)