1. 12
  1. 5

    Recently moved a project away from this to poetry. We have 2 separate applications which both depend on a shared library. Before, each had a requirements.in and a requirements-dev.in for testing and dev tooling that shouldn’t go to prod (plus the .txts of course). Let’s say, updating those was fun and in our container images, we had to reconstruct the directory structure since we used PYTHONPATH to wire it up.

    I guess the common package should have been a proper package from the start, but replacing everthing with poetry felt nice.

    Next step, nixifying everything for caching 🙂

    1. 4

      I guess the common package should have been a proper package from the start

      I’m not trying to be mean here, but my experience is that most of the time when people have stories about how Python packaging was awful/difficult/complex/problem-causing for them, at the root is always something like this, where a decision was made to go against the natural/standard flow. Building a Python package and then installing it into multiple deployed things really isn’t that difficult (you can learn how to build a basic Python package in an hour or so from the official documentation) — what was difficult, seemingly, was dealing with the consequences of deciding not to do it.

      1. 1

        To be clear, we didn’t have a whole lot of pain with what we did, it just felt like a relevant experience to the article.

        We started simple and replaced it with something more appropriate once we outgrew it

    2. 5

      I’ve switched to Poetry across the board and haven’t regretted it.

      I’m troubled that this article mentions neither the pyproject.toml standard, now six years past acceptance three days from now, nor Poetry. I understand some people are set in their ways but I’ve been spending a good bit of time migrating codebases from the old tools to Poetry and the response has generally been “where has this been all of my life.”

      1. 6

        I’m troubled that this article mentions neither the pyproject.toml standard, now six years past acceptance three days from now, nor Poetry

        The short answers are:

        • Because pyproject.toml, in the way you’re advocating it, is for building a package, and I explicitly am not recommending people build a package.
        • For the reasons explicitly stated in the post about why I didn’t recommend any specific all-in-one tool.

        The longer answer on pyproject.toml is that while the original spec for it is six years old, the original spec was to solve a very specific chicken-and-egg problem. Your own link says that packages can contain a pyproject.toml file, not that they must or that pyproject.toml is the one true universal standard packaging file, and goes on to explain the actual original use case (bootstrapping a non-setuptools build system without running into the “we have to run setup.py to find out how to run setup.py” chicken-and-egg problem which used to exist). The official how-to-build-a-package documentation similarly creates a pyproject.toml, and then proceeds to put only build-system information into it, and only to explicitly specify setuptools, which in one form or another would be the assumed default anyway. Then that documentation goes on to have you fill out a setup.py file.

        I suspect most of the confusion about this stems from the fact that once pyproject.toml existed as a standard file — even though it was for a completely different purpose — everyone jumped on it as a place to put their own tool’s configuration, much as people used to push for putting such things in setup.cfg. Couple that with the fact that third-party tools like Poetry do have to use pyproject.toml for its intended purpose, and so put all their other config in there too for convenience’s sake, and you get the impression that “pyproject.toml is the one true universal config file for all things packaging” is what was formally standardized, when that’s not exactly what happened.

        As for Poetry: a longer answer is that while I expect a good proportion of comments in reply to this article to be “just use Poetry”, my issue is that not so long ago they would have been “just use Pipenv” and not so long from now they might well be “just use (some other thing that doesn’t have a hype machine behind it yet, here in mid-2022)”. Which comes with its own set of concerns — see, for example, the continuing drama over the fact that the PyPA docs decided to cover and recommend Pipenv for managing a local environment, and how much this still outrages people today and spawns endless conspiracy theories about the people who work on packaging. The longevity and stability of pip, setuptools and venv is very explicitly what I said I am after. This is not the same as saying “your favorite alternative tool is bad”. It is just me stating my own priorities. If your priorities are different, then by all means feel free to ignore what I recommend and do something else. Just don’t assume that I have to have the same priorities, or come to the same conclusions about them, as you do.

        1. 3

          my issue is that not so long ago they would have been “just use Pipenv”

          I don’t get the issues you listed to be honest. Pipenv is still a valid choice and is better than what we had before. Poetry would be likely a better choice today. It’s not like they have a chance of disappearing in the next decade, but if you’re worried about that, you can do “pipenv lock” or “poetry export” to generate requirements.txt as the worst case backup plan and keep it with releases. That way you get the modern tooling with all its benefits and can always fall back to the saved requirements if poetry somehow becomes not runnable.

          1. 2

            Distilling my position as much as possible:

            • The standard tools — pip, setuptools, venv — are effective, solid, reliable, and not going anywhere.
            • The trendy tool may change year to year or more often.
            • Keeping up with trendy things is likely to violate my desire for “boring” unless I devote a lot of time and effort to learning and adopting things over and over so I’m aware of their ins and outs and quirks.
            • I have limited time and capacity to do that, so I choose to spend what time and capacity I do have on tools in my direct problem domain (backend web applications) and not on tools in indirect problem domains (packaging).
            • Therefore, I stick to the standard tools, and doing so is my recommendation for achieving “boring” processes.

            Again: this does not mean your favorite alternative tool is bad or will fail or that you shouldn’t use it. It also does not mean “Please, keep trying to talk me into using Poetry”. I’m aware of Poetry. I’ve explained why I don’t use or recommend it or any other all-in-one tool.

            1. 1

              Pipenv is unstable, which is the absolute worst thing for a package manager to be. I want to setup my packages and have it work for the rest of time, not break because the package manager changed under me.

          2. 1

            “where has this been all of my life.”

            Yes. The useless snarky answer is that it’s been in bundler since ~2010. :|

            Next stop, asdf over virtualenv or pyenv. For any and all languages.

            1. 1

              I recently shot down using asdf over pyenv in a devex setup tool for my team for “install an install plugin into an install tool you just installed so you can install an installer” but the abstraction is sufficiently pluggable that swapping in asdf would be pretty easy.

              I can’t tell if you’re recommending for or against asdf. Thoughts?

              1. 4

                I like asdf for about 10 languages so far. It’s great but I can’t prove it. Here are my opinions and tastes.

                • asdf manages python versions across projects. Any common projects that share python versions, share that version. Any shared globals you want (like pytest or ipython) you will just have to have shared and have to deal with that. I’ve never run into issues. The issues I run into is projects having different versions. You might say “that’s why you use virtualenv!”. No. Python is not the only language and you will likely need node.js (even as only a user) or something else. Maybe, who knows. But once you need another language, then use asdf. So why not just use asdf now?

                • After you have python installed, install poetry globally. Then create a project. You can use .tool-versions to have asdf switch to the right python version per project on cd. Each project is already repeatable through poetry. You can even fallback to requirements using poetry export -f requirements -o requirements.txt. This might help with deployment if you don’t think poetry is a prod tool, or a slow switch.

                • Now your machine is project isolated and the git repo stands by itself too. This is functionally the same as virtualenv + requirements except for the following features:

                  • asdf works on many languages
                  • poetry has dev dependencies and scripts
                  • it’s a recognizable dev workflow coming from other languages

                Now all my languages start to look the same:

                Python = asdf + poetry
                Javascript = asdf + yarn
                Ruby = asdf + bundler
                Rust = asdf + cargo
                Elixir = asdf + mix

                The API surface of asdf is the same so it’s one thing to remember. The workflow of poetry | yarn | bundler | cargo | mix is very similar so it’s not the same commands but sort of the same dance. That’s why my recent exposure to python was so jarring. That’s why the poetry author has the why section.

                The only downside (to me) of asdf is shims. A recent upgrade broke all my shims. The fix was easy.

                1. Remove all shim files (rm ~/.asdf/shims/*)
                2. For each language, run asdf reshim that-lang. And this is true if you pip install sometimes. Sometimes asdf doesn’t know that you have installed a “binary”. So you just run asdf reshim python. It’s not that bad.

                Oh, and I guess the experience greatly depends on the plugin quality. But I think having everyone join forces will work. Instead of having many language managers, you have one with a plugin system. I really wonder if there should be a global package manager with the same design (maybe there already is one)?

                1. 2

                  This sounds almost exactly what I’ve done so far, just s/asdf/pyenv since I’m doing data pipelines right now and have no other languages for asdf to manage. I even found a frustrating bug in poetry export that’s now been around for like 2 years… I’ll eventually fix it but I’m awaiting 1.2.0 when poetry-export becomes a plugin.

                  What I really want is a make deps check test build workflow and I’m 99% of the way there.

                2. 1

                  At what point does the tool choice matters? If you use .python-version, you should have a valid “python” shim whether you use asdf or pyenv. Or did you mean devex as in pre-built system images for developers?

                  From my environment, asdf is absolutely magic because I deal every day with ruby, python, Terraform, node, elixir, and probably some other things I’m forgetting now. If I was using just python, it probably wouldn’t matter as much.

                  1. 1

                    The devex I’m shooting for is one-shot:

                    git clone git@example.com:org/more_snek.git
                    cd more_snek 
                    make deps check test build

                    This clones the code, installs Python based on .python-version and Poetry (if not already installed) and all the project dependencies, runs linters ‘n’at, unit and integration tests, and builds any artifacts such as wheels or Docker containers.

                    While I could allow for some freedom for how my users install Python, I chose explicitness over implicitness for now and blessed pyenv for now. Our codebases are data pipelines that are Python-only right now, so we’re deftly avoiding everything else or whatever brew install something installs is good enough; we don’t need specific versions of something at the moment.

                    1. 1

                      My approach to that is a strict system / build separation. I.e. there may be a “make usual_dev_setup” or equivalent, but everything else calls “python” and assumes things resolve to the right version correctly through whatever system is installed. But I’m dealing with lots of projects every day, so this may be different from your experience.

                      1. 1

                        Great minds think alike. make test PYTHON=/whatever/python3 would let my users override the blessed configuration. They have virtually no reason to do so, though. I’ve got this on five codebases right now with planned expansion to six more in the next month.

            2. 4

              Use poetry. Export requirements.txt for legacy reasons. Poetry is bundler is cargo is yarn.