Tldr the author likes tools that are statically compiled executables and (reading between the lines) seems to be installing all their Python tools in a single Python installation rather than one installation per tool (aka a venv or its equivalent).
In fairness to the author if you use a Linux distribution that attempts to shove everything into your system installation, you will have a bad time. Python also doesn’t come built in with a tool that neatly solves this problem, soluble though it is; and there are a bunch of third party tools to solve this very problem.
For someone who isn’t into Python, the whole thing with environments is alien. It adds extra steps to installation, and extra steps when using Python.
I suspect it doesn’t seem like a big deal if it’s your standard procedure, but when I need to use Python once every couple of years, this is my experience: https://xkcd.com/1987/
I mentioned this in another comment, and other people have brought it up as well, but it’s very much not some sort of unique-only-to-Python thing. It’s a consequence of the fact that Python’s import statement is the equivalent of dynamic linking, and so you now get to solve the problem of “OK, where do we put and search for the dynamically-linked libraries”.
“DLL hell” is the same problem. “Error: libfoo.so.3 not found” is the same problem. Every language that supports dynamic linking eventually hits this and has to come up with a solution for it. You mostly wouldn’t notice it in the context of, say, C or C++ because probably a system package manager is smoothing it over for you and doing exactly the sort of things that would feel “alien” if you did them yourself, but the solution has to exist one way or another.
And ironically, the venv solution is effectively what Rust and Go and many other platforms all do – they create and use an isolated location on the filesystem for storing dependencies. It’s just that in the case of languages which are static-linking-only, that solution only needs to be applied at build time, while for languages with dynamic linking it needs to be persisted over to run time.
These differences matter. Even the defaults and conventions matter, because for end users it doesn’t matter that in theory it could have been better, if it isn’t in practice.
Theoretically people could rewrite their whole Python programs with all of their dependencies to be a single .py file behaving like “statically linked”, but AFAIK that would be weird, and devs just don’t do that, and users don’t expect programs distributed this way. But in Go that’s the default.
It is an extra step, but pipx[0] handles the easy case of installing isolated tools for you. If all you want to do is use python based tools, it is what you want.
Python is an anti-social language. It focuses primarily on the needs of developers and not as much on the user of the software.
I think that’s close, but not quite it. It focuses on the needs of some developers to the exclusion of others. It does not enforce any sort of balance.
Say someone on your team has a Jupyter notebook they use to calculate something important to the company. Great! Okay, now try to run it on the developer who sits across from them at the office. Bzzt, nope, not going to work. You’ll need to start pip freezing and hope that there aren’t any important C build tools needed to get it working.
Have a web team with separate frontend and backend devs. Enjoy explaining to the frontend devs what a virtual env is and why they should care. :-) Python’s “it works on my machine” factor is so bad that it propelled Docker into prominence. But your coworker who is a good designer and HTML editor might not know what a PATH is, let alone Docker.
There’s the classic XKCD about his Python installs being a “superfund site”. Here’s the thing though: Randall Munroe is a developer.
This article is being poorly received here—just pip freeze or use Poetry or follow some other conflicting advice!—because for these developers there are patterns in place that work. But for me as someone who tries to bridge between different people at shops that aren’t all Python experts, it’s a nightmare.
Python’s “it works on my machine” factor is so bad that it propelled Docker into prominence. But your coworker who is a good designer and HTML editor might not know what a PATH is, let alone Docker.
Not to drag out the Go vs Python flamewar more than necessary, but I do think this is a good way to get a handle on the culture gap between the two languages. Lots of things in Go are inconvenient or ugly for the individual but good for the group or others.
So, eg it’s a pain to me that I can’t just version tag my projects however I want, but I have to use the ugly /v2 semantic version imports system. But for other users, it’s good that they have clear and simple rules about versions and importing that work across all packages. Lots of times, when you see someone having a problem with Go, it’s because they want to do something non-standard, and then the pro-Go people end up sounding like cultists because we say “oh no, it’s for your own good not to have unused imports” or whatever. But the core is the culture clash: am I working for Go or is Go working for me? If you work for Go, it benefits the ecosystem, but maybe you would rather just do your own thing.
To paraphrase a Jedi master: When as old as Python your favorite language is, look as good it will not.
Or to just directly quote Stroustrup: There are only two kinds of languages: the ones people complain about and the ones nobody uses.
Basically everything you complain about from Python is directly or indirectly a consequence of its age and widespread adoption. Check back in a few decades to see if Go gets to a similar spot.
Also:
This article is being poorly received here—just pip freeze or use Poetry or follow some other conflicting advice!—because for these developers there are patterns in place that work.
I see one pretty bombastic chain going after the author’s tone. You’re the only comment that’s mentioned pip freeze so far.
It may well be that people who aren’t you don’t have the same experiences you do with certain tools and come to different conclusions. You’re free to do what works for you, but you seem to spend an inordinate amount of time on this site, from what I recall, trying to put down other people’s preferences. Maybe do less of that?
Basically everything you complain about from Python is directly or indirectly a consequence of its age
Another phrasing: many of the things that older languages do poorly and newer languages do well are a direct consequence of people having learned from the older languages’ missteps.
Basically everything you complain about from Python is directly or indirectly a consequence of its age and widespread adoption. Check back in a few decades to see if Go gets to a similar spot.
I’m certain that languages like Go will accumulate warts over time, but I can’t help but think that newer languages will fair better structurally. There have been so many huge advances to basic programming practices in the past 30 years and older languages were not built with them in mind. Programming languages built 30 years ago didn’t have to think about package managers, formatter, fuzzing, testing, documentation sites, and probably a lot more.
I’m sure that there will be new things that today’s languages haven’t considered, but at least they have examples of how to accommodate those major features from the last 30 years to draw from.
I’m sure that there will be new things that today’s languages haven’t considered, but at least they have examples of how to accommodate those major features from the last 30 years to draw from.
Again, I think time is not going to be as kind as Go’s advocates want it to be. I’ll pick on one specific issue and one general issue to illustrate why:
A specific issue: Go, for all that people are promoting it as better than Python on “packaging”, initially shipped with… “just list some Git repos as strings at the top of your file” as the “package manager”. And a lot of details of Go’s “proper” added-much-later package manager are still influenced heavily by that early design choice. The infamous /v2 thing, for example, is a consequence of how bad Go’s initial ability to specify and target versions of dependencies was. And the whole interface of Go modules is kind of clunky and shows as having been retrofitted on after the language was already in pretty wide use, rather than something that was thoughtfully designed to go with the language. And with Go’s policies being what they are on compatibility (or at least in theory), I’m not sure it’s possible to improve significantly on this in the future.
And a general issue: Go is, well, sort of infamous for actively refusing to learn from “the last 30 years” (and really more than just 30 years) of programming language design and evolution. It’s one of the most common criticisms of Go! The error handling, the weird quirks and limitations of the type system, the difficulty-bordering-on-outright-lack of interoperability, the massive hostility to generics… all of this and more is important stuff from the past few decades of learning about how to make programming languages that Go is proud of not making use of.
So I’m not convinced at all that Go is going to look significantly better at its 30th birthday than Python did.
Go, for all that people are promoting it as better than Python on “packaging”, initially shipped with… “just list some Git repos as strings at the top of your file” as the “package manager”.
TBH, I had forgotten about that. I can’t imagine that will age well. Just thinking about the Heroku and Gitlab free tier changes that have happened I can only imagine the sort of bit rot that might occur with direct references to GitHub repositories.
I think Python developers don’t really understand how bad it is because they know how to work around it. They have already picked their favorite python environment manager for their own projects, they know exactly what to do with other people’s code when it has a requirements.txt or a setup.py or a venv file or a pipenv file or a pyenv file or a virtualenv file or a poetry file or whatever else. And for system stuff, they make conscious and informed decisions about whether to install a package using their system package manager or using pip, and when using pip, they know whether they want the package locally or globally, and they know how to resolve problems where they have already installed a package globally with pip and their system package manager wants to overwrite the same files.
It’s not like it’s the only community with that problem. C++ people may be a bit too quick to dismiss concerns about how hard memory management is, because we know how to choose between unique_ptr and shared_ptr and raw pointers and references and stack allocation, and we may have found strategies to read the multi-megabyte template or overload resolution errors and find the nugget of useful information in the sea of useless logs. But in general, C++ people probably don’t expect non-C++ developers to deal with C++ code as much as Python developers expect non-Python developers to deal with Python code.
I agree that it’s easy to forget how complicated Python tooling can be, but trying to set up a moderately complex project on someone else’s machine usually serves as a good reminder
I once dealt with a legacy project where I had to recreate their dependencies by guessing from their imports and testing whether it worked. I had to go so far as checking commit timestamps and correlating it to release versions through pypi.
Markdown changed your “dash dash global” to “emdash global”, which made me misread your comment.
Virtual envs can be used like node_modules, but a) they aren’t the default b) they need to be activated (or install a tool to activate them) c) they also copy the Python libraries themselves, but not enough to keep it from breaking when I upgrade homebrew Python.
Python has all the pieces for dependency management, but it’s just pieces and you have to have someone on your team write a blog post about why at your company we’re all going to do X. No one writes a blog post about how they use node_modules. :-)
I didn’t say it’s perfect (or even good), but it’s pretty much the same situation as node just with bad defaults. If people understand node they should be able to understand the python situation. (And at this point I mostly assume that front end devs know node, which might be a poor assumption.)
No, you don’t have to “activate” a node_modules folder, and frankly, it’s very embarrassing and hacky that virtual envs require it. (Again, my otherwise competent coworker did not know what a PATH was. How can I explain sourcing a Bash script vs running it?) I first saw activation with Ruby Gems, and I remember thinking as an arrogant Python user, “oh those monkey patching Ruby people! They’re even monkey patching my shell variables, how gross!” Little did I know that Python would copy it soon after. It really should have been made a default many years ago now, and to TFA’s point, it’s very anti-social that it has not been made the default.
My dude, the question you posed was “how do I explain this to someone?” I proposed explanation by analogy. Analogy does not imply a 1 to 1 correspondence. It does not imply that the things are equal in value. The idea here is that this new thing serves a similar function to another thing that they might know. It serves as a great jumping off point to discuss the differences. (Nested dependencies vs one common dependency, how PATH works, etc.)
Do I love virtual envs? No, I don’t, but that wasn’t your question. I get the sense that you never really wanted an answer to the question. You wanted to complain. I don’t really want to listen to you complain so I’m going to ignore this thread.
It’s a rhetorical question. I haven’t worked with that colleague for three years.
Any analogy is going to have points of commonality and points outside the scope of the analogy, but “venv is like node_modules” does very little work as an analogy. The only things they have in common is having project dependencies somewhere. Everything else is different. I guess it’s better than having no point of reference, but it leaves a big gap.
My understanding is that packages installed with pip do not ‘own’ their dependencies - as in, if you install package A that requires version 1 of package B, package B will be in the global scope and could conflict with user-installed package C that need version 2 of package B. Is that correct?
While reading this post, I thought “what an utter load of horse shit”. Then I read some of the comments here and come to see there are some merits to the post. Python libraries can be difficult to tame. The biggest pitfall is trying to use your system libraries, but like others have pointed out, this is a problem for dynamically linked native code as well.
So, if I get this right, the argument in the post basically boils down to “you must be able to produce a standalone portable executable for a language to be worthy”. As others point out here, this is possible in Python, but neither easy nor the standard way of doing things. Other languages differ in degrees of how easy they make this. And with Nix or Docker, this is trivial for all languages, at the cost of considerable additional complexity.
But in the end, you still want to be able to produce that standalone binary from the sources, which is difficult for all languages as soon as 3rd party dependencies and versioning come into play, especially after you’ve allowed the code to bit rot for a few weeks/months/years (depending on the ecosystem).
I developed a hatred of Python about 20 years ago because absolutely nothing that I used that was written in Python worked correctly out of the box. The Xen tooling and the Jabber interop transports were both written in Python and in fairly basic usage, both of them required me to edit the source code of the installed packages to not crash. One of these fixes made me absolutely detest the language: the developer used an else clause on a while loop and expected it to execute if the loop did not execute. It turns out that this else clause is actually a catch in any other language: it executes if the loop terminates via an abnormal exit.
Over the next 10 years, I asked every Python advocate that I encountered what the semantics of that control structure were. Eventually, after 10 years, I found one who gave me the right answer. This isn’t some weird niche interaction of features of the kind that makes people hate C++, this is the semantics of one of the primitive flow-control constructs built into the language.
My other favourite was that someone had indented a line with tabs by mistake and Python didn’t warn about this (10-15 years later, it grew a warning, but I think it’s still opt in) and so control flow didn’t go the way that it appeared to because Python interpreted a tab as a certain number of spaces (instead of, you know, the thing that the character is defined to mean: one indent level) and his editor gave it a different width.
I’ve since written some Python because Pandas is a very nice library and I’m willing to put up with a bad language for a good library, but the language is awful and the tooling isn’t much better. For example, after a recent pipenv upgrade, I started getting deprecation warnings from Pandas. They show up with a line number pointing to a fairly long expression. They don’t tell me what the deprecated method that I’m calling is, just that one of the things in that expression called something that needed to be done differently. Yay. So now I know that my code will break at some point in the future, but I don’t know what I need to change to fix it. That pretty much sums up Python for me.
Ugh, that while..else is a total wart indeed, and every time I see it used, I have to read up on it and even then it still makes no sense to me. But that’s not the point of the post here, AFAIK.
I think it’s part of the problem. The author is talking about how hard it is to ship Python code that actually works for your users, when they have a different environment to you. The while...else thing is part of the reason that this is the case: it’s very easy to write code that looks correct and, as long as you always enter the while loop at least once and exit normally will work for you. It then won’t work for your consumer. The same is true of the exciting significant whitespace (which could have been trivially fixed by requiring that, per file, every line either starts with spaces xor tags, followed by a non-whitespace character, and doing a quick scan on file load to enforce this). The longer-term problems are exacerbated by the useless deprecation warnings.
Over the next 10 years, I asked every Python advocate that I encountered what the semantics of that control structure were. Eventually, after 10 years, I found one who gave me the right answer.
Once on the Python ideas mailinglist, someone was defending the feature… while misunderstanding the semantics and thinking it was an if-empty clause. :-)
In a pinch, even Java provides a better alternative as you have the ability to build all-in-one-jar files that contain all the dependencies. Not fashionable, but objectively a far less brittle option.
There are multiple approaches to distributing Python applications in a way that bundles all their dependencies.
But none of them are standard. I think it goes to the article’s point: individual devs have solutions to this problem, but there is no blessed solution from the Python Foundation because the emphasis is on solving individual needs, not creating a working ecosystem.
A large part of Python’s success has been that the esosystem is broad enough to contain a wide range of competing alternatives to problems, and the Python Foundation has explictly avoided “blessed” solutions as they see their role as maintaining the language rather than the ecosystem. I’d go as far as to say that not being controlled by a single entity have having competing approaches a defining feature of a programming language “ecosystem” — without that you just have a bundle of software from a single organisation.
In this specific case the emphasis is correctly on solving “individual” needs as there’s no single approach that would work for all users, and so many more users have solutions to their problems than they would if there was one single way to package Python applications.
Unfortunately this only works if your code is 100% pure Python… As soon as you rely on a dependency that requires some native code, zipapp stops helping with the portability…
(Granted, the same happens also with compiled languages, unless you are able to statically link your executable, which isn’t always that simple…)
Static built tools come with their own issues. When they work, they’re great and you don’t have to think about them, but the moment they fall over you’re SOL trying to track down the cause. Are debugging symbols available? can you track down the matching source tree? can you gdb your way through templated code? What if you need to make a tweak to the tools? what if the tooling is a living part of your source tree?
I feel that packaging tooling is a job best done by a standardised tool rather than the language itself. Half the problem is that you have bespoke packaging tools being written for every language, and they don’t all come with the same set of restrictions and caveats. Using nix or apt/yum/dnf or snap/flatpak puts the burden of comparability checking on the packager. as the end user, if you’re installing random unpackaged tools into a running system, you are purposefully breaking the system and you get what you asked for.
Chris Siebenmann has made the point before on his blog that it’s nice with Python that you have all the source in one-ish place. As a developer, I’ve inherited Apache servers with ancient PHP apps on them, and it’s nice that I can just rsync the stuff back to my computer, run git init and start hacking away without knowing if the developer who was there before me even used a VCS at all.
But I think a static executable could have this advantage: there’s no reason you couldn’t just embed all the source code into the binary and have a command flag or a tool that can extract it all again. Go programs already have debug information with line numbers for panics, so probably it’s not a big step to just include the whole source. It’s just that no one ever does that. It would be interesting if that were added as a mode though.
The OP says the happy path is that the developer can do it all at build time, and that it can be automated. Both of these things are true for Python as well, just not “By Default”. So of course, developers are lazy and make it your problem if you want to use Python software.
It would be nice if Python fixed their tools to have a sane default around deployment, but it’s obvious nobody is willing to do that hard work(since it’s not a technical problem, but a political problem).
Non-Lazy developers, like Calibre developers and early versions of Dropbox decided to not be lazy, there are many other instances where Python developers did the work, and automated it. I do it at $WORK, because I have to support Windows and MacOS end-users with a Python codebase. My deliverables are MSI’s and .DMG’s, from Python. It’s not really even that hard to do, but it doesn’t come out of the box by default.
So one could argue, the problem is lazy developers, and not really Python’s ecosystem. Python’s perspective is, delivering code is the developers problem, not Python’s and it’s a reasonable perspective, as there is always debate about the best way(s) to deploy software.
The OP is arguing that because binaries are not enabled by default, one should avoid the entire Python ecosystem, which to me is not really reasonable. Perhaps they should just avoid the Python tools that don’t have a good deployment story, because the developers were lazy. Or contribute upstream to those tools a good deployment story yourself, if one is so inclined.
On the one hand, I have no problem with criticizing Python as a language. At my current job I generally write in Python, I’ve spent a lot of time dealing with Python build system issues, and I’m well-aware of the ways in which your program can fail because the Python environment is somehow not set up correctly, even when you use more modern tools like poetry to help. I know what it’s like to contaminate your system python install because a venv wasn’t set up correctly, and I recognize that having to have a separate venv for every Python project (or do something even more exotic like embedding a Python interpreter in your final distributable) is inelegant and bloated, to say the least.
I’ve also used plenty of software written in Python that I don’t work on myself, just as an ordinary desktop linux user. I know what it looks like when some tool someone wrote in python fails because something about your OS’s standard python environment isn’t what the particular tool you’re using expects. I don’t think I would go so far as to say that Python is an anti-social language; but I’m well aware of what a pain it can be to debug software written with it for reasons that fundamentally have to do with the environment rather than a bug in the code per se.
At the same time, I think the best place to solve these problems is at the OS level, rather than a vague commitment on the part of developers to not use python. Even if we grant for the sake of argument that using Python meaningfully harms other developers or computer users (which, to be clear, I reject), computer users should expect their operating system to facilitate using it anyway. My views on this are largely informed by using Nix (and, for that matter, debugging environment-related issues in languages that have nothing to do with Python, like Node or for that matter good old fashioned C that still needs to link to the right version of a dynamic library for your program not to crash in an unhelpful way). When you get software packaged by your OS, your OS should make sure to set up the environment such that that software can run - Nix does this in a clever way of course, but other operating systems too still need to do something to make sure that what the program expects to be available exists on the system. Simply refusing to use Python doesn’t solve this problem, and if this problem were solved there’d be no reason to avoid Python in particular.
The author says “This might provoke Python programmers, but Python is an anti-social language. It focuses primarily on the needs of developers and not as much on the user of the software.” and I don’t totally disagree, but I think you could just as easily rephrase it as “Python focuses primarily on the needs of developers as part of empowering all users to be developers of their own software”
We use Python extensively at the NRAO. Most of my group’s software used to be written in Java, and we’ve been in the process of converting it to Python since about 2015 or 2016. The team has mixed feelings about this; some really detest Python’s whitespace syntax (I find this to be a fairly superficial opinion) and others rather miss the typing of Java, but I don’t think anyone misses Java per-se, rather than either the curly-brace affordance or the static typing affordance.
We have made a point of using the new optional typing system and there are tests, although I can confess the coverage is not as high as I would like, on the newer projects our discipline has been better and we have settled into a routine with typing and testing that makes Python sufficiently robust for our purposes.
The majority of our code is delivered as web applications. For this, the venv story has been acceptable, but I certainly wouldn’t call it “good.” Some of our applications must be delivered as scripts to be executed on cluster nodes. For this domain, Python has been basically unacceptable. We are using pex to produce standalone executables, but they are essentially self-extracting executables with venvs inside them. As you can imagine, these become large very quickly; we have simple scripts with a few dependencies that turn into pexes that number in the hundreds of megabytes. This is pretty unhelpful, but we live in a world where clusters are frequently managed by external partners and thus we cannot assume much about the executing machine. For these specific programs, we are migrating to Go, because it is easier to produce static executables that work on a wide variety of platforms. We audited many languages for this task but ultimately concluded that, for this purpose, Go was probably the best choice for us. If this domain counts as “tooling” then I suppose it supports the post’s conclusion. I certainly agree with the author insofar as Python developers, for the most part, become accustomed to the pain of certain things, and either internalize habits that do not create pain or else foist problems on the end users. For web apps, the end user doesn’t have to run the code, so it’s fine. But for executables it is a pain.
I do not see us migrating away from Python wholesale in the foreseeable future. This is partly because the scientific libraries available for Python are not matched by Go or much of anything else, and the scientific community, if anything, appears to be investing even more heavily in Python (although there is motion towards Julia).
You should really use something like Go where it’s statically compiled into one binary and you don’t have to be a pip wizard to fix weird problems when they happen (not if, when)
I dunno, all this stuff is weird to me since I know that people have been building sysadmin-y scripts and tools and daemons and other things in Python for literal decades and it’s basically just in the past couple years that people seem to have decided that none of it ever happened or worked. And weirdly there’s an inverse relationship between the quality of Python’s packaging ecosystem – which used to be horrible and now is pretty darned good – and the loudness of the complaints or the strenuousness of the people insisting Python packaging is unfit for any purpose.
Like, I remember vividly what it was like to try to package and distribute Python in 2008. It was awful! Django literally maintained a “no external dependencies” policy for years because of how awful it was. And now it’s easy! It’s as if all the complaints that ought to have been happening back then have somehow been timeskipped forward to now.
Maybe it’s even the problem that packaging became simpler and people start to include more external packages into their code.
When I think about my Python experience at work (with too many badly tracked dependencies) I immediately want to avoid Python for any simple tooling, even though it is my primary language. But on the other hand, if I just use stdlib Python it’s so good for small scripts: Copy to my server, execute, fix bugs on the server, copy back to local PC.
I think part of the issue is that packaging ecosystems keep evolving. Once Python had nothing, but it was okay, because everything had nothing. Then Python got pip and PyPI, which was a big step up! But around the same time Node came out, and while it was a mess with unpinned and nested dependencies, the concept of a node_modules folder was a big step forward and eventually they started pinning all the dependencies and it became pretty easy to use. (Of course, since it was easy to use, people made crap like leftpad and iseven.) Now compared to Go modules or Cargo, Python has fallen pretty badly behind.
It’s interesting to imagine what will happen next to raise the bar again. Probably something inspired by Nix?
I have similar experiences. However, I realised that I really fret if I have to package and share any python code if it’s more than a single file (some of this is rooted in how we do things at work). For sharing a single file, even with several dependencies, I have no fears, as I usually wrap it with Nix, just as described here: https://notes.yukiisbo.red/posts/2021/07/Spice_up_with_Nix_Scripts.html
And weirdly there’s an inverse relationship between the quality of Python’s packaging ecosystem – which used to be horrible and now is pretty darned good – and the loudness of the complaints or the strenuousness of the people insisting Python packaging is unfit for any purpose.
That’s probably an indication of how much easier it got for other languages. It seems like Go and Rust are defining the standard language tooling. Even if python got a lot friendlier it’s not as friendly as those languages.
Or maybe it’s an indication of pythons rise in popularity. When you are a niche language you get dedicated developers who care to learn the whole process. When you are popular everyone has to interact with you even if they didn’t take the time to really learn it.
Go and Rust are also kind of cheating, because they both, for the moment, are static-linking-only. Which means you only have to worry about having local copies of dependencies in a build directory as you compile. As soon as you introduce dynamic linking, you have to make decisions about where to look for your dynamically-linked dependencies, and you’re back into the thing people specifically are complaining most about in Python, including in this thread (the need for venv or a similar construct to isolate potentially-conflicting sets of dependencies from each other).
Rust at least seems open to the idea of eventually supporting dynamic linking, but it requires committing to an ABI and solving a lot of hard problems that Cargo currently doesn’t have to deal with. Go, I don’t know.
Or, just maybe, we should use well-supported languages we’re good at and that we enjoy.
My experience with Python tooling in regards to embedded development is good, especially because in the world of embedded development where you might need to rapidly iterate on how your tooling works you can dive into the source and modify it without much stress. It’s inane to me the author hasn’t heard of poetry or pipenv or such yet, maybe a couple minutes of Google would save him weeks yearly.
This might provoke Python programmers, but Python is an anti-social language. It focuses primarily on the needs of developers and not as much on the user of the software. That’s not very nice. In fact, it is pretty arrogant.
It is okay to feel provoked by this statement. As pointed out previously: you have probably invested a lot of time in Python. You will be inclined to justify and defend that investment. I would urge you to take some time to think about this and try to calm your urge to come up with counter-arguments.
Wow are you an arrogant asshole.
It is worth observing that both sides in the polarized response validates what I’m trying to bring attention to. And it also validates the inclusion of the second paragraph under the “Anti social behavior” section. If this paragraph provokes you it kind of means that you are part of the problem and you aren’t doing Python any favors by manifesting the problem. I’m sorry, if you feel offended, but being offended isn’t helpful. For anyone.
Or maybe it means that you’re an arrogant asshole.
Anyway it seems his language of choice is go; I remember when Hugo switched from Python pygments to Go chroma for syntax highlighting. I work in a couple of niche languages without default highlighters. To add a custom highlighter to pygments, I used the built-in extension points and could make tweaks on the fly. To iterate on a custom highlighter to chroma, I had to recompile the entire Hugo binary every single time.
Ended up being easier to stay on an older version of Hugo.
Go has always done caching per package (and it got better a couple years ago), so it would be weird if you were really compiling the whole binary each time. It was probably an incremental build unless you used Docker or something without a persistent cache.
It is slightly weird to me that the thing the author brings up here is the tendency of dynamically linked software to break (which isn’t really unique to Python) rather than, say, application startup times.
The reason I think this is weird is that the dynamic linking thing is tractable to solve, by a couple methods. One is packaging things up as [deb, rpm, brew formulas, nix derivations]. The application startup time problem, on the other hand, doesn’t seem very tractable to me without a lot of changes to the interpreter.
I’ve had good luck using PyInstaller to package Python scripts into a single binary. I’ve seen lots of open source projects that use it as well. It sometimes takes a bit of figuring out and it’s definitely not part of the standard tooling, though.
pipx is pretty great for “I want to run this script that’s on PyPI and I absolutely do not care about installing its dependencies or setting up a virtual environment for it” but doesn’t help at all for stuff that’s not packaged as a module.
Another quite annoying property of (large) python programs is their dreadful startup time. For example, google-cloud-sdk (which I thankfully don’t have to use much) takes a full second, on a beefy machine, to display --help. A page of text. I wish that thing was written in Go (or rust), which at least is very nice on the program’s users.
That’s not a property of python. If an app takes that much time to display the help text, that’s entirely on the app. (and the decision to load everything up front)
There are multiple stages here. If you’re saying “it should take less than 100ms”, then sure, that’s on the language. But “less than 1 second” then that’s on the app itself.
Yeah, this is a pervasive pain. I get the same with Azure’s CLI and some of the AWS tools. I think it’s particularly bad with things like cloud services CLIs because over time they end up with hundreds to thousands of endpoints that they load code for on process start.
I’m also used to seeing tools written in Ruby exhibit the same problem for the same reason.
Packaging python can be a wild ride. For anyone feeling the pain, this guide can help you better understand. I love python and agree that the distribution is painful.
Tldr the author likes tools that are statically compiled executables and (reading between the lines) seems to be installing all their Python tools in a single Python installation rather than one installation per tool (aka a venv or its equivalent).
In fairness to the author if you use a Linux distribution that attempts to shove everything into your system installation, you will have a bad time. Python also doesn’t come built in with a tool that neatly solves this problem, soluble though it is; and there are a bunch of third party tools to solve this very problem.
For someone who isn’t into Python, the whole thing with environments is alien. It adds extra steps to installation, and extra steps when using Python.
I suspect it doesn’t seem like a big deal if it’s your standard procedure, but when I need to use Python once every couple of years, this is my experience: https://xkcd.com/1987/
I mentioned this in another comment, and other people have brought it up as well, but it’s very much not some sort of unique-only-to-Python thing. It’s a consequence of the fact that Python’s
importstatement is the equivalent of dynamic linking, and so you now get to solve the problem of “OK, where do we put and search for the dynamically-linked libraries”.“DLL hell” is the same problem. “Error:
libfoo.so.3not found” is the same problem. Every language that supports dynamic linking eventually hits this and has to come up with a solution for it. You mostly wouldn’t notice it in the context of, say, C or C++ because probably a system package manager is smoothing it over for you and doing exactly the sort of things that would feel “alien” if you did them yourself, but the solution has to exist one way or another.And ironically, the
venvsolution is effectively what Rust and Go and many other platforms all do – they create and use an isolated location on the filesystem for storing dependencies. It’s just that in the case of languages which are static-linking-only, that solution only needs to be applied at build time, while for languages with dynamic linking it needs to be persisted over to run time.These differences matter. Even the defaults and conventions matter, because for end users it doesn’t matter that in theory it could have been better, if it isn’t in practice.
Theoretically people could rewrite their whole Python programs with all of their dependencies to be a single .py file behaving like “statically linked”, but AFAIK that would be weird, and devs just don’t do that, and users don’t expect programs distributed this way. But in Go that’s the default.
It is an extra step, but pipx[0] handles the easy case of installing isolated tools for you. If all you want to do is use python based tools, it is what you want.
[0] https://github.com/pypa/pipx
Right, it’s not completely unfounded
I think that’s close, but not quite it. It focuses on the needs of some developers to the exclusion of others. It does not enforce any sort of balance.
Say someone on your team has a Jupyter notebook they use to calculate something important to the company. Great! Okay, now try to run it on the developer who sits across from them at the office. Bzzt, nope, not going to work. You’ll need to start pip freezing and hope that there aren’t any important C build tools needed to get it working.
Have a web team with separate frontend and backend devs. Enjoy explaining to the frontend devs what a virtual env is and why they should care. :-) Python’s “it works on my machine” factor is so bad that it propelled Docker into prominence. But your coworker who is a good designer and HTML editor might not know what a PATH is, let alone Docker.
There’s the classic XKCD about his Python installs being a “superfund site”. Here’s the thing though: Randall Munroe is a developer.
This article is being poorly received here—just pip freeze or use Poetry or follow some other conflicting advice!—because for these developers there are patterns in place that work. But for me as someone who tries to bridge between different people at shops that aren’t all Python experts, it’s a nightmare.
Amen.
Not to drag out the Go vs Python flamewar more than necessary, but I do think this is a good way to get a handle on the culture gap between the two languages. Lots of things in Go are inconvenient or ugly for the individual but good for the group or others.
So, eg it’s a pain to me that I can’t just version tag my projects however I want, but I have to use the ugly /v2 semantic version imports system. But for other users, it’s good that they have clear and simple rules about versions and importing that work across all packages. Lots of times, when you see someone having a problem with Go, it’s because they want to do something non-standard, and then the pro-Go people end up sounding like cultists because we say “oh no, it’s for your own good not to have unused imports” or whatever. But the core is the culture clash: am I working for Go or is Go working for me? If you work for Go, it benefits the ecosystem, but maybe you would rather just do your own thing.
To paraphrase a Jedi master: When as old as Python your favorite language is, look as good it will not.
Or to just directly quote Stroustrup: There are only two kinds of languages: the ones people complain about and the ones nobody uses.
Basically everything you complain about from Python is directly or indirectly a consequence of its age and widespread adoption. Check back in a few decades to see if Go gets to a similar spot.
Also:
I see one pretty bombastic chain going after the author’s tone. You’re the only comment that’s mentioned
pip freezeso far.It may well be that people who aren’t you don’t have the same experiences you do with certain tools and come to different conclusions. You’re free to do what works for you, but you seem to spend an inordinate amount of time on this site, from what I recall, trying to put down other people’s preferences. Maybe do less of that?
Another phrasing: many of the things that older languages do poorly and newer languages do well are a direct consequence of people having learned from the older languages’ missteps.
I’m certain that languages like Go will accumulate warts over time, but I can’t help but think that newer languages will fair better structurally. There have been so many huge advances to basic programming practices in the past 30 years and older languages were not built with them in mind. Programming languages built 30 years ago didn’t have to think about package managers, formatter, fuzzing, testing, documentation sites, and probably a lot more.
I’m sure that there will be new things that today’s languages haven’t considered, but at least they have examples of how to accommodate those major features from the last 30 years to draw from.
Again, I think time is not going to be as kind as Go’s advocates want it to be. I’ll pick on one specific issue and one general issue to illustrate why:
A specific issue: Go, for all that people are promoting it as better than Python on “packaging”, initially shipped with… “just list some Git repos as strings at the top of your file” as the “package manager”. And a lot of details of Go’s “proper” added-much-later package manager are still influenced heavily by that early design choice. The infamous
/v2thing, for example, is a consequence of how bad Go’s initial ability to specify and target versions of dependencies was. And the whole interface of Go modules is kind of clunky and shows as having been retrofitted on after the language was already in pretty wide use, rather than something that was thoughtfully designed to go with the language. And with Go’s policies being what they are on compatibility (or at least in theory), I’m not sure it’s possible to improve significantly on this in the future.And a general issue: Go is, well, sort of infamous for actively refusing to learn from “the last 30 years” (and really more than just 30 years) of programming language design and evolution. It’s one of the most common criticisms of Go! The error handling, the weird quirks and limitations of the type system, the difficulty-bordering-on-outright-lack of interoperability, the massive hostility to generics… all of this and more is important stuff from the past few decades of learning about how to make programming languages that Go is proud of not making use of.
So I’m not convinced at all that Go is going to look significantly better at its 30th birthday than Python did.
TBH, I had forgotten about that. I can’t imagine that will age well. Just thinking about the Heroku and Gitlab free tier changes that have happened I can only imagine the sort of bit rot that might occur with direct references to GitHub repositories.
I think Python developers don’t really understand how bad it is because they know how to work around it. They have already picked their favorite python environment manager for their own projects, they know exactly what to do with other people’s code when it has a requirements.txt or a setup.py or a venv file or a pipenv file or a pyenv file or a virtualenv file or a poetry file or whatever else. And for system stuff, they make conscious and informed decisions about whether to install a package using their system package manager or using pip, and when using pip, they know whether they want the package locally or globally, and they know how to resolve problems where they have already installed a package globally with pip and their system package manager wants to overwrite the same files.
It’s not like it’s the only community with that problem. C++ people may be a bit too quick to dismiss concerns about how hard memory management is, because we know how to choose between unique_ptr and shared_ptr and raw pointers and references and stack allocation, and we may have found strategies to read the multi-megabyte template or overload resolution errors and find the nugget of useful information in the sea of useless logs. But in general, C++ people probably don’t expect non-C++ developers to deal with C++ code as much as Python developers expect non-Python developers to deal with Python code.
I agree that it’s easy to forget how complicated Python tooling can be, but trying to set up a moderately complex project on someone else’s machine usually serves as a good reminder
I once dealt with a legacy project where I had to recreate their dependencies by guessing from their imports and testing whether it worked. I had to go so far as checking commit timestamps and correlating it to release versions through pypi.
By default, pip is like using npm install —global. Virtual envs are the equivalent to node_modules.
Markdown changed your “dash dash global” to “emdash global”, which made me misread your comment.
Virtual envs can be used like node_modules, but a) they aren’t the default b) they need to be activated (or install a tool to activate them) c) they also copy the Python libraries themselves, but not enough to keep it from breaking when I upgrade homebrew Python.
Python has all the pieces for dependency management, but it’s just pieces and you have to have someone on your team write a blog post about why at your company we’re all going to do X. No one writes a blog post about how they use node_modules. :-)
I didn’t say it’s perfect (or even good), but it’s pretty much the same situation as node just with bad defaults. If people understand node they should be able to understand the python situation. (And at this point I mostly assume that front end devs know node, which might be a poor assumption.)
No, you don’t have to “activate” a node_modules folder, and frankly, it’s very embarrassing and hacky that virtual envs require it. (Again, my otherwise competent coworker did not know what a PATH was. How can I explain sourcing a Bash script vs running it?) I first saw activation with Ruby Gems, and I remember thinking as an arrogant Python user, “oh those monkey patching Ruby people! They’re even monkey patching my shell variables, how gross!” Little did I know that Python would copy it soon after. It really should have been made a default many years ago now, and to TFA’s point, it’s very anti-social that it has not been made the default.
My dude, the question you posed was “how do I explain this to someone?” I proposed explanation by analogy. Analogy does not imply a 1 to 1 correspondence. It does not imply that the things are equal in value. The idea here is that this new thing serves a similar function to another thing that they might know. It serves as a great jumping off point to discuss the differences. (Nested dependencies vs one common dependency, how PATH works, etc.)
Do I love virtual envs? No, I don’t, but that wasn’t your question. I get the sense that you never really wanted an answer to the question. You wanted to complain. I don’t really want to listen to you complain so I’m going to ignore this thread.
It’s a rhetorical question. I haven’t worked with that colleague for three years.
Any analogy is going to have points of commonality and points outside the scope of the analogy, but “venv is like node_modules” does very little work as an analogy. The only things they have in common is having project dependencies somewhere. Everything else is different. I guess it’s better than having no point of reference, but it leaves a big gap.
My understanding is that packages installed with pip do not ‘own’ their dependencies - as in, if you install package A that requires version 1 of package B, package B will be in the global scope and could conflict with user-installed package C that need version 2 of package B. Is that correct?
Yeah, that is a major difference between node_modules and virtual envs.
Hard agree about the Docker thing—before Docker I wouldn’t even have attempted to set up a local environment for someone who isn’t a Python developer.
While reading this post, I thought “what an utter load of horse shit”. Then I read some of the comments here and come to see there are some merits to the post. Python libraries can be difficult to tame. The biggest pitfall is trying to use your system libraries, but like others have pointed out, this is a problem for dynamically linked native code as well.
So, if I get this right, the argument in the post basically boils down to “you must be able to produce a standalone portable executable for a language to be worthy”. As others point out here, this is possible in Python, but neither easy nor the standard way of doing things. Other languages differ in degrees of how easy they make this. And with Nix or Docker, this is trivial for all languages, at the cost of considerable additional complexity.
But in the end, you still want to be able to produce that standalone binary from the sources, which is difficult for all languages as soon as 3rd party dependencies and versioning come into play, especially after you’ve allowed the code to bit rot for a few weeks/months/years (depending on the ecosystem).
I developed a hatred of Python about 20 years ago because absolutely nothing that I used that was written in Python worked correctly out of the box. The Xen tooling and the Jabber interop transports were both written in Python and in fairly basic usage, both of them required me to edit the source code of the installed packages to not crash. One of these fixes made me absolutely detest the language: the developer used an
elseclause on awhileloop and expected it to execute if the loop did not execute. It turns out that thiselseclause is actually acatchin any other language: it executes if the loop terminates via an abnormal exit.Over the next 10 years, I asked every Python advocate that I encountered what the semantics of that control structure were. Eventually, after 10 years, I found one who gave me the right answer. This isn’t some weird niche interaction of features of the kind that makes people hate C++, this is the semantics of one of the primitive flow-control constructs built into the language.
My other favourite was that someone had indented a line with tabs by mistake and Python didn’t warn about this (10-15 years later, it grew a warning, but I think it’s still opt in) and so control flow didn’t go the way that it appeared to because Python interpreted a tab as a certain number of spaces (instead of, you know, the thing that the character is defined to mean: one indent level) and his editor gave it a different width.
I’ve since written some Python because Pandas is a very nice library and I’m willing to put up with a bad language for a good library, but the language is awful and the tooling isn’t much better. For example, after a recent pipenv upgrade, I started getting deprecation warnings from Pandas. They show up with a line number pointing to a fairly long expression. They don’t tell me what the deprecated method that I’m calling is, just that one of the things in that expression called something that needed to be done differently. Yay. So now I know that my code will break at some point in the future, but I don’t know what I need to change to fix it. That pretty much sums up Python for me.
Ugh, that
while..elseis a total wart indeed, and every time I see it used, I have to read up on it and even then it still makes no sense to me. But that’s not the point of the post here, AFAIK.I think it’s part of the problem. The author is talking about how hard it is to ship Python code that actually works for your users, when they have a different environment to you. The
while...elsething is part of the reason that this is the case: it’s very easy to write code that looks correct and, as long as you always enter the while loop at least once and exit normally will work for you. It then won’t work for your consumer. The same is true of the exciting significant whitespace (which could have been trivially fixed by requiring that, per file, every line either starts with spaces xor tags, followed by a non-whitespace character, and doing a quick scan on file load to enforce this). The longer-term problems are exacerbated by the useless deprecation warnings.Once on the Python ideas mailinglist, someone was defending the feature… while misunderstanding the semantics and thinking it was an if-empty clause. :-)
There are multiple approaches to distributing Python applications in a way that bundles all their dependencies.
But none of them are standard. I think it goes to the article’s point: individual devs have solutions to this problem, but there is no blessed solution from the Python Foundation because the emphasis is on solving individual needs, not creating a working ecosystem.
A large part of Python’s success has been that the esosystem is broad enough to contain a wide range of competing alternatives to problems, and the Python Foundation has explictly avoided “blessed” solutions as they see their role as maintaining the language rather than the ecosystem. I’d go as far as to say that not being controlled by a single entity have having competing approaches a defining feature of a programming language “ecosystem” — without that you just have a bundle of software from a single organisation.
In this specific case the emphasis is correctly on solving “individual” needs as there’s no single approach that would work for all users, and so many more users have solutions to their problems than they would if there was one single way to package Python applications.
Yeah, these things are always tradeoffs. Obviously, Python is doing a lot right because it’s becoming the biggest language in a lot of areas.
There is, and has been a standard solution for a very long time. But for some reason it wasn’t picked up.
Unfortunately this only works if your code is 100% pure Python… As soon as you rely on a dependency that requires some native code,
zipappstops helping with the portability…(Granted, the same happens also with compiled languages, unless you are able to statically link your executable, which isn’t always that simple…)
Heh, I’ve never heard of that one! I use pex pretty regularly though.
And here I am over here using
.parfiles…Static built tools come with their own issues. When they work, they’re great and you don’t have to think about them, but the moment they fall over you’re SOL trying to track down the cause. Are debugging symbols available? can you track down the matching source tree? can you gdb your way through templated code? What if you need to make a tweak to the tools? what if the tooling is a living part of your source tree?
I feel that packaging tooling is a job best done by a standardised tool rather than the language itself. Half the problem is that you have bespoke packaging tools being written for every language, and they don’t all come with the same set of restrictions and caveats. Using nix or apt/yum/dnf or snap/flatpak puts the burden of comparability checking on the packager. as the end user, if you’re installing random unpackaged tools into a running system, you are purposefully breaking the system and you get what you asked for.
Chris Siebenmann has made the point before on his blog that it’s nice with Python that you have all the source in one-ish place. As a developer, I’ve inherited Apache servers with ancient PHP apps on them, and it’s nice that I can just rsync the stuff back to my computer, run
git initand start hacking away without knowing if the developer who was there before me even used a VCS at all.But I think a static executable could have this advantage: there’s no reason you couldn’t just embed all the source code into the binary and have a command flag or a tool that can extract it all again. Go programs already have debug information with line numbers for panics, so probably it’s not a big step to just include the whole source. It’s just that no one ever does that. It would be interesting if that were added as a mode though.
Proposal: we allow the “unkind” flag to apply to submissions as well. Deliberately inflammatory pieces like this do little to improve discussion.
The OP says the happy path is that the developer can do it all at build time, and that it can be automated. Both of these things are true for Python as well, just not “By Default”. So of course, developers are lazy and make it your problem if you want to use Python software.
It would be nice if Python fixed their tools to have a sane default around deployment, but it’s obvious nobody is willing to do that hard work(since it’s not a technical problem, but a political problem).
Non-Lazy developers, like Calibre developers and early versions of Dropbox decided to not be lazy, there are many other instances where Python developers did the work, and automated it. I do it at $WORK, because I have to support Windows and MacOS end-users with a Python codebase. My deliverables are MSI’s and .DMG’s, from Python. It’s not really even that hard to do, but it doesn’t come out of the box by default.
So one could argue, the problem is lazy developers, and not really Python’s ecosystem. Python’s perspective is, delivering code is the developers problem, not Python’s and it’s a reasonable perspective, as there is always debate about the best way(s) to deploy software.
The OP is arguing that because binaries are not enabled by default, one should avoid the entire Python ecosystem, which to me is not really reasonable. Perhaps they should just avoid the Python tools that don’t have a good deployment story, because the developers were lazy. Or contribute upstream to those tools a good deployment story yourself, if one is so inclined.
I’m of two minds about this.
On the one hand, I have no problem with criticizing Python as a language. At my current job I generally write in Python, I’ve spent a lot of time dealing with Python build system issues, and I’m well-aware of the ways in which your program can fail because the Python environment is somehow not set up correctly, even when you use more modern tools like poetry to help. I know what it’s like to contaminate your system python install because a venv wasn’t set up correctly, and I recognize that having to have a separate venv for every Python project (or do something even more exotic like embedding a Python interpreter in your final distributable) is inelegant and bloated, to say the least.
I’ve also used plenty of software written in Python that I don’t work on myself, just as an ordinary desktop linux user. I know what it looks like when some tool someone wrote in python fails because something about your OS’s standard python environment isn’t what the particular tool you’re using expects. I don’t think I would go so far as to say that Python is an anti-social language; but I’m well aware of what a pain it can be to debug software written with it for reasons that fundamentally have to do with the environment rather than a bug in the code per se.
At the same time, I think the best place to solve these problems is at the OS level, rather than a vague commitment on the part of developers to not use python. Even if we grant for the sake of argument that using Python meaningfully harms other developers or computer users (which, to be clear, I reject), computer users should expect their operating system to facilitate using it anyway. My views on this are largely informed by using Nix (and, for that matter, debugging environment-related issues in languages that have nothing to do with Python, like Node or for that matter good old fashioned C that still needs to link to the right version of a dynamic library for your program not to crash in an unhelpful way). When you get software packaged by your OS, your OS should make sure to set up the environment such that that software can run - Nix does this in a clever way of course, but other operating systems too still need to do something to make sure that what the program expects to be available exists on the system. Simply refusing to use Python doesn’t solve this problem, and if this problem were solved there’d be no reason to avoid Python in particular.
The author says “This might provoke Python programmers, but Python is an anti-social language. It focuses primarily on the needs of developers and not as much on the user of the software.” and I don’t totally disagree, but I think you could just as easily rephrase it as “Python focuses primarily on the needs of developers as part of empowering all users to be developers of their own software”
We use Python extensively at the NRAO. Most of my group’s software used to be written in Java, and we’ve been in the process of converting it to Python since about 2015 or 2016. The team has mixed feelings about this; some really detest Python’s whitespace syntax (I find this to be a fairly superficial opinion) and others rather miss the typing of Java, but I don’t think anyone misses Java per-se, rather than either the curly-brace affordance or the static typing affordance.
We have made a point of using the new optional typing system and there are tests, although I can confess the coverage is not as high as I would like, on the newer projects our discipline has been better and we have settled into a routine with typing and testing that makes Python sufficiently robust for our purposes.
The majority of our code is delivered as web applications. For this, the venv story has been acceptable, but I certainly wouldn’t call it “good.” Some of our applications must be delivered as scripts to be executed on cluster nodes. For this domain, Python has been basically unacceptable. We are using pex to produce standalone executables, but they are essentially self-extracting executables with venvs inside them. As you can imagine, these become large very quickly; we have simple scripts with a few dependencies that turn into pexes that number in the hundreds of megabytes. This is pretty unhelpful, but we live in a world where clusters are frequently managed by external partners and thus we cannot assume much about the executing machine. For these specific programs, we are migrating to Go, because it is easier to produce static executables that work on a wide variety of platforms. We audited many languages for this task but ultimately concluded that, for this purpose, Go was probably the best choice for us. If this domain counts as “tooling” then I suppose it supports the post’s conclusion. I certainly agree with the author insofar as Python developers, for the most part, become accustomed to the pain of certain things, and either internalize habits that do not create pain or else foist problems on the end users. For web apps, the end user doesn’t have to run the code, so it’s fine. But for executables it is a pain.
I do not see us migrating away from Python wholesale in the foreseeable future. This is partly because the scientific libraries available for Python are not matched by Go or much of anything else, and the scientific community, if anything, appears to be investing even more heavily in Python (although there is motion towards Julia).
You should really use something like Go where it’s statically compiled into one binary and you don’t have to be a pip wizard to fix weird problems when they happen (not if, when)
I dunno, all this stuff is weird to me since I know that people have been building sysadmin-y scripts and tools and daemons and other things in Python for literal decades and it’s basically just in the past couple years that people seem to have decided that none of it ever happened or worked. And weirdly there’s an inverse relationship between the quality of Python’s packaging ecosystem – which used to be horrible and now is pretty darned good – and the loudness of the complaints or the strenuousness of the people insisting Python packaging is unfit for any purpose.
Like, I remember vividly what it was like to try to package and distribute Python in 2008. It was awful! Django literally maintained a “no external dependencies” policy for years because of how awful it was. And now it’s easy! It’s as if all the complaints that ought to have been happening back then have somehow been timeskipped forward to now.
Maybe it’s even the problem that packaging became simpler and people start to include more external packages into their code.
When I think about my Python experience at work (with too many badly tracked dependencies) I immediately want to avoid Python for any simple tooling, even though it is my primary language. But on the other hand, if I just use stdlib Python it’s so good for small scripts: Copy to my server, execute, fix bugs on the server, copy back to local PC.
Jevon’s paradox but for dependencies.
I think part of the issue is that packaging ecosystems keep evolving. Once Python had nothing, but it was okay, because everything had nothing. Then Python got pip and PyPI, which was a big step up! But around the same time Node came out, and while it was a mess with unpinned and nested dependencies, the concept of a node_modules folder was a big step forward and eventually they started pinning all the dependencies and it became pretty easy to use. (Of course, since it was easy to use, people made crap like leftpad and iseven.) Now compared to Go modules or Cargo, Python has fallen pretty badly behind.
It’s interesting to imagine what will happen next to raise the bar again. Probably something inspired by Nix?
I have similar experiences. However, I realised that I really fret if I have to package and share any python code if it’s more than a single file (some of this is rooted in how we do things at work). For sharing a single file, even with several dependencies, I have no fears, as I usually wrap it with Nix, just as described here: https://notes.yukiisbo.red/posts/2021/07/Spice_up_with_Nix_Scripts.html
It’s cool in some circles to hate on what’s popular.
That’s probably an indication of how much easier it got for other languages. It seems like Go and Rust are defining the standard language tooling. Even if python got a lot friendlier it’s not as friendly as those languages.
Or maybe it’s an indication of pythons rise in popularity. When you are a niche language you get dedicated developers who care to learn the whole process. When you are popular everyone has to interact with you even if they didn’t take the time to really learn it.
Go and Rust are also kind of cheating, because they both, for the moment, are static-linking-only. Which means you only have to worry about having local copies of dependencies in a build directory as you compile. As soon as you introduce dynamic linking, you have to make decisions about where to look for your dynamically-linked dependencies, and you’re back into the thing people specifically are complaining most about in Python, including in this thread (the need for
venvor a similar construct to isolate potentially-conflicting sets of dependencies from each other).Rust at least seems open to the idea of eventually supporting dynamic linking, but it requires committing to an ABI and solving a lot of hard problems that Cargo currently doesn’t have to deal with. Go, I don’t know.
Or, just maybe, we should use well-supported languages we’re good at and that we enjoy.
My experience with Python tooling in regards to embedded development is good, especially because in the world of embedded development where you might need to rapidly iterate on how your tooling works you can dive into the source and modify it without much stress. It’s inane to me the author hasn’t heard of poetry or pipenv or such yet, maybe a couple minutes of Google would save him weeks yearly.
Wow are you an arrogant asshole.
Or maybe it means that you’re an arrogant asshole.
Anyway it seems his language of choice is go; I remember when Hugo switched from Python pygments to Go chroma for syntax highlighting. I work in a couple of niche languages without default highlighters. To add a custom highlighter to pygments, I used the built-in extension points and could make tweaks on the fly. To iterate on a custom highlighter to chroma, I had to recompile the entire Hugo binary every single time.
Ended up being easier to stay on an older version of Hugo.
FWIW you can now do this with Chroma (which I created) via XML lexers, though I’m not sure if this has been exposed through Hugo yet.
Go has always done caching per package (and it got better a couple years ago), so it would be weird if you were really compiling the whole binary each time. It was probably an incremental build unless you used Docker or something without a persistent cache.
It is slightly weird to me that the thing the author brings up here is the tendency of dynamically linked software to break (which isn’t really unique to Python) rather than, say, application startup times.
The reason I think this is weird is that the dynamic linking thing is tractable to solve, by a couple methods. One is packaging things up as [deb, rpm, brew formulas, nix derivations]. The application startup time problem, on the other hand, doesn’t seem very tractable to me without a lot of changes to the interpreter.
I’ve had good luck using PyInstaller to package Python scripts into a single binary. I’ve seen lots of open source projects that use it as well. It sometimes takes a bit of figuring out and it’s definitely not part of the standard tooling, though.
pipx is pretty great for “I want to run this script that’s on PyPI and I absolutely do not care about installing its dependencies or setting up a virtual environment for it” but doesn’t help at all for stuff that’s not packaged as a module.
Another quite annoying property of (large) python programs is their dreadful startup time. For example, google-cloud-sdk (which I thankfully don’t have to use much) takes a full second, on a beefy machine, to display
--help. A page of text. I wish that thing was written in Go (or rust), which at least is very nice on the program’s users.That’s not a property of python. If an app takes that much time to display the help text, that’s entirely on the app. (and the decision to load everything up front)
It absolutely is a property of python that it loads very slowly. The same thing written in Go would start in a few milliseconds.
I guess google could be bothered in rearchitecturing their program around this, but it’d be to work around the very real issue.
There are multiple stages here. If you’re saying “it should take less than 100ms”, then sure, that’s on the language. But “less than 1 second” then that’s on the app itself.
Yeah, this is a pervasive pain. I get the same with Azure’s CLI and some of the AWS tools. I think it’s particularly bad with things like cloud services CLIs because over time they end up with hundreds to thousands of endpoints that they load code for on process start.
I’m also used to seeing tools written in Ruby exhibit the same problem for the same reason.
This is so true, I hate when some tool is written in python and I have to navigate the pain and suffering that is virtualenv or pip.
https://sedimental.org/the_packaging_gradient.html
Packaging python can be a wild ride. For anyone feeling the pain, this guide can help you better understand. I love python and agree that the distribution is painful.