I find Poetry much nicer to work with than anything else I’ve tried for Python, both for authoring libraries and for applications that only consume other libraries.
Virtual environment management
Managing virtual environments is generally a pain, but having Poetry manage it for me when I run poetry install is great. It supports different virtual environments with different Python versions, so you don’t need to throw away your working setup to test out a new Python version.
Lock file
Pinning your transient dependencies means you won’t have your application suddenly break because a dependency of one of our dependencies suddenly released a breaking change, which then requires you to spend an inordinate amount of time to work out why the change you deployed suddenly broke something entirely unrelated, because the new version was pulled in.
Building and releasing of Python packages
It’s baked right into the tool you already use to manage your project, so you don’t need to write custom scripts to bump the package’s version number, build it or release it to PyPI.
I’m sure Poetry isn’t without flaws, but which project is? It’s working very well for me at least.
Poetry wasn’t the first project to support all those things, nor was it the last. There’s intense competition in the all-singing-all-dancing-all-in-one-packaging-tool space in Python these days.
As the author of the linked “boring dependency management” article, though, I explicitly do not use Poetry or any other such all-in-one tool because I don’t mind using the slightly lower-level standard tools to accomplish my goals, and they avoid the headaches that can come from being an early adopter of things like Poetry, Hatch, PDM, etc.
Poetry would take an inordinately long time to resolve the required dependencies to install a package. Perhaps it was a one-off type of thing? Unfortunately not. It got to the point where I actively avoided using poetry to install dependencies and resorted to adding the dependency in pyproject.toml by hand, installing it locally using pip, and exporting the requirements as I did before
Computing a solution to the dependency graph (i.e. generating the lock file) is an NP-hard problem. It gets much slower with every package that you add. In a large project, it will take a long time, especially since poetry is written in Python. The slowness is usually not a big deal, because you only have to compute a new lock file when you want to add/remove/update dependencies.
It sounds like the author is computing a new lock file every time their build script is run. If so, that’s a misuse of poetry. You should generate it once, check it in to the repo, and then you can run poetry install to fetch the packages after that.
tl;dr
I think the author has unrealistic expectations for a Python package manager and also is probably misusing poetry. IMO poetry is the best Python package manager and we should be encouraging its use, not spreading FUD about it.
If I recall correctly, the slowness of python dependency resolution comes from the fact that setup.py has to be executed to get the list of dependencies. Repeating that invocation for all candidate versions really quickly adds up.
Computing a solution to the dependency graph (i.e. generating the lock file) is an NP-hard problem. It gets much slower with every package that you add.
While NP-hard problems are in principle extremely slow, in practice we’re really good at quickly solving even very large problems, as long as they’re not pathological. “Very large” in this case is millions of clauses — something well beyond the size of the typical dependency graph
In a large project, it will take a long time, especially since poetry is written in Python.
On the other hand, if Poetry wasn’t offloading the problem to a dedicated SAT solver, that’s a pretty big design failure.
The slowness is due in part to the fact that calculating the full dependency set requires downloading and inspecting the packages. Even ones that use static metadata declaration, so that the setup.py script doesn’t exist/doesn’t have to be executed, because even the static package metadata is still inside the package itself. There’s work ongoing to expose more package metadata directly through the API of the Python Package Index, which would allow fetching the dependency list for a package without needing to obtain a copy of the full package, but it’s not 100% there yet.
The same issue also affects pip and its dependency solver, and any other tools which try to calculate dependency trees for Python packages.
On the other hand, if Poetry wasn’t offloading the problem to a dedicated SAT solver, that’s a pretty big design failure.
I wouldn’t say so: based on experience with Cargo, I think reasonable design choices are:
write a naive backtracking solver by hand (Rust & Cargo)
write a generic solver specifically for dependency resolution (Dart & pubgrub)
Most of complexity in Cargo’s implementation seems to come from domain modeling (a lot of various kinds of dependencies) and desire to provide good error messages. Performance is also important, but I wouldn’t say it’s an overriding concern.
Maybe for Python with more historically baroque dep graphs and low performance reaching for off-the shelf solver makes more sense. But then, I’d assume that for Python packaging tools being “pure Python” is a hard requirement (you don’t want installing the installing tool to be harder than actually installing stuff!), and I’d assume good SAT solvers are not pure Python?
In my defense, I was most definitely not computing poetry.lock every time. I generated it once, committed it, ran the relevant commands (e.g. ‘add’), and observed obscene amounts of slowness that I didn’t feel was justified. Maybe it was to be expected given the inherent difficulty in dependency resolution and I should have adjusted my expectations accordingly, but when I was able to switch to the pip-tools workflow I no longer had to contend with such annoyances given the already low amount of value Poetry was providing.
If Poetry works for you, then I am genuinely glad for you as I am a strong believer in having tools around that remove unnecessary friction from our lives. Poetry just didn’t fit the bill for me and made me be more wary of introducing such tools (and any dependencies, to be honest) in the future.
Computing a solution to the dependency graph (i.e. generating the lock file) is an NP-hard problem. It gets much slower with every package that you add.
It’s not necessary NP-hard. If you only allowed one-sided constraints (eg, bounding semver ranges from below, and allowing duplicating major versions), greedy algorithm works. This probably doesn’t work for Python, which has to work with existing conventions, instead of designing something easily solvable.
boring is good, because all fancy stuff will break sooner or later.
for dependency management i heartily recommend pip-tools, giving you pip-compile and pip-sync commands that work on requirements files, which are the ‘native’ format used by pip (de facto standard python package installer). this is also a good read on the topic: https://hynek.me/articles/python-app-deps-2018/
for venv management on local dev machines (not elsewhere), use direnv, which is a generic solution. unfortunately often overlooked and hence underappreciated, because many people look for python-specific things. with direnv, an .envrc file containing a single line (layout python or layout pyenv 3.11.1) is enough to get automagic venv creation+activation. and it also works in editors. boring is good, again.
I am the author of the “boring dependency management” article linked near the bottom of the post, and pip-tools is the only thing I recommend that isn’t one of the big three standard Python packaging tools (pip, setuptools, venv). I also only recommend it because it’s a relatively simple thing that could be quickly replaced with a script that calls pip freeze/pip download/pip hash/etc. if I ever needed to. The convenience is that I don’t have to write that script myself when i use pip-tools.
boring is good, because all fancy stuff will break sooner or later.
I have the opposite experience. Our team uses pip-compile and we’re constantly running into problems. Someone invariably generates the lockfile with a different environment, or pip-compile fails to find a resolution that Poetry does. It’s gotten to the point where we’ve created a Docker image that contains a custom script just so we can generate our lockfile consistently. In my mind that is not boring. The boring thing would be to use the tool that is popular and just works.
I’m not saying Poetry is perfect, and the brownout was terrible. But if a workflow needs to be distributed across a team, then it’s better to use something well trodden and that works out of the box versus kludging something together. If a workflow is for personal use only, then of course do whatever floats your boat.
I’m struggling to understand, if you’re already containerized, why your entire dependency management workflow wouldn’t be run in the container. I certainly run pip-compile in the container where I intend to pip install, for example.
If the requirements are shared across a large team, not everyone will have the desire, knowledge or even ability (i.e, there are people using Windows) to do their development in a Docker container.
Docker works on Windows. And given that it provides a consistent environment and allows for the local and production environments to be as identical as possible, I’m a huge fan of just Dockerizing everything from the start. People can still write code in their favorite editor/IDE, just run it in Docker.
requirement files can also contain platform-specific things, so generatingrequirements.txt from requirements.in (via pip-compile) requires running inside the same (or similar) environment. containers to the rescue, and you can also leverage them in your CI environment. see https://peps.python.org/pep-0496/#examples
If I start talking about quality, someone says that they are just a data scientist. If I start talking about other languages or frameworks, the person is monolingual and can’t be convinced. It’s Blubb paradox but for tools. You can do equivalent things with requirements.txt but you don’t know what you are missing. “I can just make a requirements.dev.txt!”. So the pattern of DIY and lack of conventions continues.
I enjoyed poetry’s Why statement on their old README, where they explained why they invented it. I latched onto this section.
And then later, Pipenv (which I was using) failed to solve the tree on a project. I switched to poetry in place, exporting to requirements.txt for CI and ease of switch reasons and it just worked. Not only that, it just lines up with the dev flow in cargo, npm, yarn, bundler, mix, shards, etc.
None of this really proves anything. It’s opinion, taste, bias. But what would be a lot better than boring dependency management would be no conversation at all.
The 3 laws of programming language design
What you get right nobody mentions.
I’m mostly tired of the topic because the python devs / community I’ve encountered just hasn’t felt the happy path so it’s really hard to explain how I’d test out a new version of a library in a branch. Or the conversation ends with “I don’t need that, I’m doing data science”.
I think the CI blackout getting as far as it did cost a lot of trust the community had in the Poetry team. But I think the larger problem – doing a breaking change like this in their 1.2 release, rather than running a major version bump – persists, especially because they do intend to remove the old installer script entirely at some point.
(personally I do not understand why they couldn’t just tell people to pip install poetry as the bootstrap in the first place, or why they feel the need to break stuff when people who need old versions could be told to just curl from a GitHub permalink or whatever since it’s not like the repo history will be cleared, but I also don’t have the time to go digging through all the closed/pruned issue discussions to try to find out)
So I use and like poetry (mostly because it feels ergonomically very similar to cargo, which I quite like), and this part of your comment confuses me a little:
I do not understand why they couldn’t just tell people to pip install poetry
Is that not the standard advice? I don’t recall how I arrived at this practice, but I’d have told you it was from poetry documentation.
When I’m setting up a new machine, set up a new venv and pip install poetry there, then symlink that poetry binary in my path.
(If it’s a machine where I need to switch between python versions much, I use pyenv to make it easy to do that.)
The whole “CI brownout” fiasco was because an older installer script is being deprecated, and even for CI they only say that “manual” (i.e., pip install poetry) installation should be “seriously considered”, in a paragraph which also mentions the “streamlined and simplified” installation from their installer script.
I don’t personally use Poetry and never have, so I don’t know how much of their installation advice actually sticks.
The script that they’re now recommending does exactly “just use pip” to install poetry in its own venv.
r"""
This script will install Poetry and its dependencies in an isolated fashion.
It will perform the following steps:
* Create a new virtual environment using the built-in venv module, or the virtualenv zipapp if venv is unavailable.
This will be created at a platform-specific path (or `$POETRY_HOME` if `$POETRY_HOME` is set:
- `~/Library/Application Support/pypoetry` on macOS
- `$XDG_DATA_HOME/pypoetry` on Linux/Unix (`$XDG_DATA_HOME` is `~/.local/share` if unset)
- `%APPDATA%\pypoetry` on Windows
* Update pip inside the virtual environment to avoid bugs in older versions.
* Install the latest (or a given) version of Poetry inside this virtual environment using pip.
* Install a `poetry` script into a platform-specific path (or `$POETRY_HOME/bin` if `$POETRY_HOME` is set):
- `~/.local/bin` on Unix
- `%APPDATA%\Python\Scripts` on Windows
* Attempt to inform the user if they need to add this bin directory to their `$PATH`, as well as how to do so.
* Upon failure, write an error log to `poetry-installer-error-<hash>.log and restore any previous environment.
This script performs minimal magic, and should be relatively stable. However, it is optimized for interactive developer
use and trivial pipelines. If you are considering using this script in production, you should consider manually-managed
installs, or use of pipx as alternatives to executing arbitrary, unversioned code from the internet. If you prefer this
script to alternatives, consider maintaining a local copy as part of your infrastructure.
For full documentation, visit https://python-poetry.org/docs/#installation.
"""
I’m sure I either got my approach from this script or from some prior description of the practice that got wrapped up into it. (I only switched to poetry last year.)
I hadn’t heard much of the “CI brownout” fiasco. Thanks for explaining that.
As the other commenter said, this does more than “just use pip” – it creates a virtualenv to isolate Poetry from all other Python tooling on the system, installs Poetry into that virtualenv, and then does does/tells you to do path manipulation to ensure that the resulting poetry executable can be found and invoked.
It’s also the opposite of how normal pip usage goes, because you never want to use a single system-wide copy of pip – you always want to invoke whatever pip is local to the Python install/env you’re working in (which is why the general advice is to do, e.g., python -m pip install instead of pip install). But Poetry wants you to have a single system-wide install of Poetry and always use it no matter what Python env/install you’re working in.
But Poetry wants you to have a single system-wide install of Poetry and always use it no matter what Python env/install you’re working in.
Good point. I suppose my habit of using pyenv to keep different versions of python around is papering over that just to the point I’ve not been burned by it. That’s probably pure luck. I could get burned if two projects that both use the same version of python need conflicting versions of poetry. That hasn’t happened yet but it’d damn sure break things in a difficult-to-troubleshoot way if it did.
I think there’s some subtle context they should make explicit. The way I understood it, they do the pip installation in an isolated environment, but they don’t want people to just “pip install poetry”, because where this installation goes and how to get that executable in the PATH is often not well understood. (If it was, third of the reason for poetry existing would be gone)
Dear gods yes. All of this. Also that IIRC they’ve changed their own dependency file format 2 or 3 times in the 1.x era and I’d regard that as a “bump the major version” thing.
Poetry is kinda nice but really, really, really, I want cargo. What I really want is a wrapper around python which accepts basically all the same command line arguments, but looks for a thing like a poetry config and does all the environment setup for me.
Maybe this is a “there are too many standard ways of doing things” thing but I desparately want to point to a binary that will always work when working with Python and deps for basically everyone.
I find Poetry much nicer to work with than anything else I’ve tried for Python, both for authoring libraries and for applications that only consume other libraries.
Virtual environment management
Managing virtual environments is generally a pain, but having Poetry manage it for me when I run
poetry installis great. It supports different virtual environments with different Python versions, so you don’t need to throw away your working setup to test out a new Python version.Lock file
Pinning your transient dependencies means you won’t have your application suddenly break because a dependency of one of our dependencies suddenly released a breaking change, which then requires you to spend an inordinate amount of time to work out why the change you deployed suddenly broke something entirely unrelated, because the new version was pulled in.
Building and releasing of Python packages
It’s baked right into the tool you already use to manage your project, so you don’t need to write custom scripts to bump the package’s version number, build it or release it to PyPI.
I’m sure Poetry isn’t without flaws, but which project is? It’s working very well for me at least.
Poetry wasn’t the first project to support all those things, nor was it the last. There’s intense competition in the all-singing-all-dancing-all-in-one-packaging-tool space in Python these days.
As the author of the linked “boring dependency management” article, though, I explicitly do not use Poetry or any other such all-in-one tool because I don’t mind using the slightly lower-level standard tools to accomplish my goals, and they avoid the headaches that can come from being an early adopter of things like Poetry, Hatch, PDM, etc.
Computing a solution to the dependency graph (i.e. generating the lock file) is an NP-hard problem. It gets much slower with every package that you add. In a large project, it will take a long time, especially since poetry is written in Python. The slowness is usually not a big deal, because you only have to compute a new lock file when you want to add/remove/update dependencies.
It sounds like the author is computing a new lock file every time their build script is run. If so, that’s a misuse of poetry. You should generate it once, check it in to the repo, and then you can run
poetry installto fetch the packages after that.tl;dr I think the author has unrealistic expectations for a Python package manager and also is probably misusing
poetry. IMOpoetryis the best Python package manager and we should be encouraging its use, not spreading FUD about it.If I recall correctly, the slowness of python dependency resolution comes from the fact that
setup.pyhas to be executed to get the list of dependencies. Repeating that invocation for all candidate versions really quickly adds up.While NP-hard problems are in principle extremely slow, in practice we’re really good at quickly solving even very large problems, as long as they’re not pathological. “Very large” in this case is millions of clauses — something well beyond the size of the typical dependency graph
On the other hand, if Poetry wasn’t offloading the problem to a dedicated SAT solver, that’s a pretty big design failure.
The slowness is due in part to the fact that calculating the full dependency set requires downloading and inspecting the packages. Even ones that use static metadata declaration, so that the
setup.pyscript doesn’t exist/doesn’t have to be executed, because even the static package metadata is still inside the package itself. There’s work ongoing to expose more package metadata directly through the API of the Python Package Index, which would allow fetching the dependency list for a package without needing to obtain a copy of the full package, but it’s not 100% there yet.The same issue also affects
pipand its dependency solver, and any other tools which try to calculate dependency trees for Python packages.I wouldn’t say so: based on experience with Cargo, I think reasonable design choices are:
Most of complexity in Cargo’s implementation seems to come from domain modeling (a lot of various kinds of dependencies) and desire to provide good error messages. Performance is also important, but I wouldn’t say it’s an overriding concern.
Maybe for Python with more historically baroque dep graphs and low performance reaching for off-the shelf solver makes more sense. But then, I’d assume that for Python packaging tools being “pure Python” is a hard requirement (you don’t want installing the installing tool to be harder than actually installing stuff!), and I’d assume good SAT solvers are not pure Python?
Turns out almost no package managers use SAT! I was totes wrong.
In my defense, I was most definitely not computing
poetry.lockevery time. I generated it once, committed it, ran the relevant commands (e.g. ‘add’), and observed obscene amounts of slowness that I didn’t feel was justified. Maybe it was to be expected given the inherent difficulty in dependency resolution and I should have adjusted my expectations accordingly, but when I was able to switch to thepip-toolsworkflow I no longer had to contend with such annoyances given the already low amount of value Poetry was providing.If Poetry works for you, then I am genuinely glad for you as I am a strong believer in having tools around that remove unnecessary friction from our lives. Poetry just didn’t fit the bill for me and made me be more wary of introducing such tools (and any dependencies, to be honest) in the future.
It’s not necessary NP-hard. If you only allowed one-sided constraints (eg, bounding semver ranges from below, and allowing duplicating major versions), greedy algorithm works. This probably doesn’t work for Python, which has to work with existing conventions, instead of designing something easily solvable.
boring is good, because all fancy stuff will break sooner or later.
for dependency management i heartily recommend pip-tools, giving you
pip-compileandpip-synccommands that work on requirements files, which are the ‘native’ format used bypip(de facto standard python package installer). this is also a good read on the topic: https://hynek.me/articles/python-app-deps-2018/for venv management on local dev machines (not elsewhere), use direnv, which is a generic solution. unfortunately often overlooked and hence underappreciated, because many people look for python-specific things. with direnv, an
.envrcfile containing a single line (layout pythonorlayout pyenv 3.11.1) is enough to get automagic venv creation+activation. and it also works in editors. boring is good, again.I am the author of the “boring dependency management” article linked near the bottom of the post, and pip-tools is the only thing I recommend that isn’t one of the big three standard Python packaging tools (pip, setuptools, venv). I also only recommend it because it’s a relatively simple thing that could be quickly replaced with a script that calls
pip freeze/pip download/pip hash/etc. if I ever needed to. The convenience is that I don’t have to write that script myself when i use pip-tools.I have the opposite experience. Our team uses
pip-compileand we’re constantly running into problems. Someone invariably generates the lockfile with a different environment, orpip-compilefails to find a resolution that Poetry does. It’s gotten to the point where we’ve created a Docker image that contains a custom script just so we can generate our lockfile consistently. In my mind that is not boring. The boring thing would be to use the tool that is popular and just works.I’m not saying Poetry is perfect, and the brownout was terrible. But if a workflow needs to be distributed across a team, then it’s better to use something well trodden and that works out of the box versus kludging something together. If a workflow is for personal use only, then of course do whatever floats your boat.
I’m struggling to understand, if you’re already containerized, why your entire dependency management workflow wouldn’t be run in the container. I certainly run
pip-compilein the container where I intend topip install, for example.If the requirements are shared across a large team, not everyone will have the desire, knowledge or even ability (i.e, there are people using Windows) to do their development in a Docker container.
Docker works on Windows. And given that it provides a consistent environment and allows for the local and production environments to be as identical as possible, I’m a huge fan of just Dockerizing everything from the start. People can still write code in their favorite editor/IDE, just run it in Docker.
Yeah not disagreeing with the value of containerizing things, but Docker works on Windows if your company pays for a Docker Desktop subscription.
requirement files can also contain platform-specific things, so generating
requirements.txtfromrequirements.in(viapip-compile) requires running inside the same (or similar) environment. containers to the rescue, and you can also leverage them in your CI environment. see https://peps.python.org/pep-0496/#examplesIf I start talking about quality, someone says that they are just a data scientist. If I start talking about other languages or frameworks, the person is monolingual and can’t be convinced. It’s Blubb paradox but for tools. You can do equivalent things with
requirements.txtbut you don’t know what you are missing. “I can just make a requirements.dev.txt!”. So the pattern of DIY and lack of conventions continues.I enjoyed poetry’s Why statement on their old README, where they explained why they invented it. I latched onto this section.
And then later, Pipenv (which I was using) failed to solve the tree on a project. I switched to poetry in place, exporting to requirements.txt for CI and ease of switch reasons and it just worked. Not only that, it just lines up with the dev flow in cargo, npm, yarn, bundler, mix, shards, etc.
None of this really proves anything. It’s opinion, taste, bias. But what would be a lot better than boring dependency management would be no conversation at all.
The 3 laws of programming language design
I’m mostly tired of the topic because the python devs / community I’ve encountered just hasn’t felt the happy path so it’s really hard to explain how I’d test out a new version of a library in a branch. Or the conversation ends with “I don’t need that, I’m doing data science”.
It almost sounds like the poetry team gave up on the CI blackout idea once people started to complain.
I think the CI blackout getting as far as it did cost a lot of trust the community had in the Poetry team. But I think the larger problem – doing a breaking change like this in their 1.2 release, rather than running a major version bump – persists, especially because they do intend to remove the old installer script entirely at some point.
(personally I do not understand why they couldn’t just tell people to
pip install poetryas the bootstrap in the first place, or why they feel the need to break stuff when people who need old versions could be told to justcurlfrom a GitHub permalink or whatever since it’s not like the repo history will be cleared, but I also don’t have the time to go digging through all the closed/pruned issue discussions to try to find out)So I use and like poetry (mostly because it feels ergonomically very similar to cargo, which I quite like), and this part of your comment confuses me a little:
Is that not the standard advice? I don’t recall how I arrived at this practice, but I’d have told you it was from poetry documentation.
When I’m setting up a new machine, set up a new venv and pip install poetry there, then symlink that poetry binary in my path.
(If it’s a machine where I need to switch between python versions much, I use pyenv to make it easy to do that.)
https://python-poetry.org/docs/#installing-with-the-official-installer
The whole “CI brownout” fiasco was because an older installer script is being deprecated, and even for CI they only say that “manual” (i.e.,
pip install poetry) installation should be “seriously considered”, in a paragraph which also mentions the “streamlined and simplified” installation from their installer script.I don’t personally use Poetry and never have, so I don’t know how much of their installation advice actually sticks.
I see.
The script that they’re now recommending does exactly “just use pip” to install poetry in its own venv.
I’m sure I either got my approach from this script or from some prior description of the practice that got wrapped up into it. (I only switched to poetry last year.)
I hadn’t heard much of the “CI brownout” fiasco. Thanks for explaining that.
As the other commenter said, this does more than “just use pip” – it creates a virtualenv to isolate Poetry from all other Python tooling on the system, installs Poetry into that virtualenv, and then does does/tells you to do path manipulation to ensure that the resulting
poetryexecutable can be found and invoked.It’s also the opposite of how normal
pipusage goes, because you never want to use a single system-wide copy ofpip– you always want to invoke whateverpipis local to the Python install/env you’re working in (which is why the general advice is to do, e.g.,python -m pip installinstead ofpip install). But Poetry wants you to have a single system-wide install of Poetry and always use it no matter what Python env/install you’re working in.Good point. I suppose my habit of using pyenv to keep different versions of python around is papering over that just to the point I’ve not been burned by it. That’s probably pure luck. I could get burned if two projects that both use the same version of python need conflicting versions of poetry. That hasn’t happened yet but it’d damn sure break things in a difficult-to-troubleshoot way if it did.
I think there’s some subtle context they should make explicit. The way I understood it, they do the pip installation in an isolated environment, but they don’t want people to just “pip install poetry”, because where this installation goes and how to get that executable in the PATH is often not well understood. (If it was, third of the reason for poetry existing would be gone)
Dear gods yes. All of this. Also that IIRC they’ve changed their own dependency file format 2 or 3 times in the 1.x era and I’d regard that as a “bump the major version” thing.
Poetry is kinda nice but really, really, really, I want
cargo. What I really want is a wrapper aroundpythonwhich accepts basically all the same command line arguments, but looks for a thing like a poetry config and does all the environment setup for me.Maybe this is a “there are too many standard ways of doing things” thing but I desparately want to point to a binary that will always work when working with Python and deps for basically everyone.
Slow