For those who are curious about what’s going on and lack context…
Let’s start with a quick refresher on Python packaging. When you use Python’s own default toolchain to package up and distribute some code, you generally have a choice of either or both of two types of artifacts: a “source distribution”, which is a .tar.gz archive whose installation on the target system may involve a build step and thus require the target system to have the requisite compiler toolchain and non-Python dependencies (such as C libraries it needs to link against). Or a binary distribution, which is a file with the extension .whl [note 1], and contains everything already compiled and packed up for a specific target operating system and CPU architecture, so that installation only requires unpacking the .whl and putting files in the correct location.
And although Python technically exposes only a C API, that API is robust enough, and other languages’ interop with C is similarly robust enough, that it’s possible to have Python use compiled extensions that communicate with a variety of other languages. The numeric/scientific Python stack, for example, interfaces with linear-algebra libraries which are written in Fortran (and thus building from source requires a Fortran compiler; it’s much nicer to install from a precompiled .whl).
So. This library – which is not part of the Python standard library, and also not required for common tasks like speaking HTTPS or writing scripts that will control other machines through SSH – had some compiled extensions which were written in C. Now, the library is in the process of moving the extensions from C to Rust. The latest release builds the Rust by default but does not have a hard dependency on it (you can disable with a build-time environment flag). The next release will have a hard dependency on it, and as such will require the now-ported-to-Rust extension code in order to work at all.
There appear to be two groups affected by this.
One group is people who were building Python applications in containers based on Alpine Linux. Python’s .whl format supports precompiled packages for Linux distros, but is based on a lowest-common-denominator definition of “Linux” that Alpine – with its alternative libc – does not meet. As a result, people using Alpine as their base image must build from source, and apparently only Alpine 3.13 and later have a sufficiently recent Rust to be able to do this. Inserting an aside of my own recommendation: in general, if you are looking to containerize Python applications, Alpine is a poor choice of base image. This article does a good job explaining why, but the tl;dr is the alleged benefits of Alpine mostly manifest when using other languages like Go, and actually become downsides when doing Python (since now you get to compile the world every time you rely on a Python package with an extension written in C or other non-Python language).
The other group is people who are on certain types of systems that are unsupported by Rust itself, and thus cannot build the ported-to-Rust extension modules now used in this package.
Now, that out of the way, I think the Cryptography team made the right choice. Their options are to stay in C and keep the attendant risks that brings, thus exposing every user to nebulous but likely future security issues, or switch to something memory-safe but low-level and performant enough to be a reasonable replacement, but at the cost of making Alpine users’ lives a bit harder and leaving a much smaller number of users completely unsupported. For a security-critical library which treats security as its first priority, I think this is an unpleasant (because nobody ever likes cutting someone else off, and because it generates heated complaint threads) but not particularly difficult choice to make.
Side arguments about “$THING would have fixed this” where $THING is one of “alternative versioning scheme”, “alternative/additional announcement channel”, and so on also don’t really work for reasons that become clear after reading the GitHub thread.
[note 1] This is a tortured joke. Python was named after Monty Python, and much of its early documentation contained references to Monty Python sketches. The original name of the Python Package Index was the Python Cheese Shop, in homage to one such sketch. Many types of cheese are distributed in large round blocks called wheels, so this type of Python package is called a “wheel” and has extension .whl.
For those ecosystems without Rust, might something like mrustc be an option? It’d be a somewhat torturous path, but transpile Rust to C, then you could use the Python C bindings …
Generalised, this could be a good option for dealing with Rust dependencies on a wide range of languages.
Wouldn’t it be possible to ship with Rust by default, but also accept a C fallback for cases where Rust is not (easily) available? I agree that in principle Rust is probably the better choice, but it’s not like the C code has become insecure or “bad” all of the sudden, and practicality beats purity IMO.
This would meet both needs; it was mentioned in passing in one comment, but probably got lost in all the noise/rather off-topic complaining/ranting. The biggest downside is the increased maintenance burden, but I’d just offer a “if you want it, you can maintain it” and/or “if you want it, this is how much it will cost”. If no one (or too few people) take up the offer then it probably wasn’t all that important after all.
Practically speaking if there is enough interest somebody will fork it to keep the C backend alive. The level of disgruntlement alone will drive that along for a while, long term I don’t think that will carry through. In some cases it makes more sense to just fix the Rust toolchain to support cross compiling to your favorite architecture. Once a few more platforms take than initiative the will to maintain a legacy fork will die off. Better for the health of the main project to be free of the burden early rather than later.
The problem with a fork is that it won’t fix cases where this library is an indirect dependency (i.e. my app imports libfoo which imports this). I’m not sure if Python provides a good way to override this?
it makes more sense to just fix the Rust toolchain to support cross compiling to your favorite architecture
Yeah, this would be the best long-term solution, but also not something that will be done overnight. I’m just trying to think of a solution that will fix the problems of today, so that there’s time to fix the longer term issues without too much friction.
Python package maintainers rarely use semantic versioning and often break backwards compatibility in minor releases. One of several reasons that dependency management is a nightmare in Python world.
I generally consider semantic versioning to be a well-intentioned falsehood. I don’t think that package vendors can have effective insight into which of their changes break compatibility when they can’t have a full bottom-up consumer graph for everyone who uses it.
I don’t think that Python gets this any worse than any other language.
I’ve heard this opinion expressed before… I find it to be either dangerously naive or outright dishonest. There’s a world of difference between a) the rare bug fix release or nominally-orthogonal-feature-add release that unintentionally breaks downstream code and b) intentionally changing and deprecating API’s in “minor” releases.
In my view, adopting SemVer is a statement of values and intention. It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.
In my view, adopting SemVer is a statement of values and intention. It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.
A “statement of values and intention” carries no binding commitment. And the fact that you have to hedge with “as much as is reasonably possible” and “only knowingly break” kind of gives away what the real problem is: every change potentially alters the observable behavior of the software in a way that will break someone’s reliance on the previous behavior, and therefore the only way to truly follow SemVer is to increment major on every commit. Which is the same as declaring the version number to be meaningless, since if every change is a compatibility break, there’s no useful information to be gleaned from seeing the version number increment.
And that’s without getting into some of my own direct experience. For example, I’ve been on the Django security team for many years, and from time to time someone has found a security issue in Django that cannot be fixed in a backwards-compatible way. Thankfully fewer of those in recent years since many of them related to weird old functionality dating to Django’s days as a newspaper CMS, but they do happen. Anyway, SemVer’s answer to this is “then either don’t fix it, or do but no matter how you fix it you’ve broken SemVer and people on the internet will scream at you and tell you that you ought to be following SemVer”. Not being a fan of no-win situations, I am content that Django has never and likely never will commit to following SemVer.
Anyway, SemVer’s answer to this is “then either don’t fix it, or do but no matter how you fix it you’ve broken SemVer and people on the internet will scream at you and tell you that you ought to be following SemVer”.
What do you mean? SemVer’s answer to “this bug can’t be fixed in a backwards-compatible way” is to increment the major version to indicate a breaking change. You probably also want to get the message across to your users by pushing a new release of the old major version which prints some noisy “this version of blah is deprecated and has security issues” messages to the logs.
It’s not perfect, I’m not saying SemVer is a silver bullet. I’m especially worried about the effects of basing automated tooling on the assumption that no package would ever push a minor or patch release with a breaking change; it seems to cause ecosystems like the NPM to be highly fragile. But when taken as a statement of intent rather than a guarantee, I think SemVer has value, and I don’t understand why you think your security issue anecdote requires breaking SemVer.
What do you mean? SemVer’s answer to “this bug can’t be fixed in a backwards-compatible way” is to increment the major version to indicate a breaking change.
So, let’s consider Django, because I know that well (as mentioned above). Typically Django does a feature release (minor version bump) every 8 months or so, and every third one bumps the major version and completes a deprecation cycle. So right now Django 3.1 is the latest release; next will be 3.2 (every X.2 is an LTS), then 4.0.
And the support matrix consists of the most recent feature release (full bugfix and security support), the one before that (security support only), and usually one LTS (but there’s a period at the end of each where two of them overlap). The policy is that if you run on a given LTS with no deprecation warnings issued from your code, you’re good to upgrade to the next (which will be a major version bump; for example, if you’re on 2.2 LTS right now, your next LTS will be 3.2).
But… what happens when a bug is found in an LTS that can’t be fixed in a backwards-compatible way? Especially a security issue? “Support for that LTS is cut off effective immediately, everybody upgrade across a major version right now” is a non-starter, but is what you propose as the correct answer. The only option is to break SemVer and do the backwards-incompatible change as a bugfix release of the LTS. Which then leads to “why don’t you follow SemVer” complaints. Well, because following SemVer would actually be worse for users than this option is.
But… what happens when a bug is found in an LTS that can’t be fixed in a backwards-compatible way?
Why do people run an LTS version, if not for being able to avoid worrying about it as a dependency? If you’re making incompatible changes: forget about semver, you’re breaking the LTS contract, and you may as well tell drop the LTS tag and people to run the latest.
you may as well tell drop the LTS tag and people to run the latest
I can think of only a couple instances in the history of Django where it happened that a security issue couldn’t be fixed in a completely backwards-compatible way. Minimizing the breakage for people – by shipping the fix into supported releases – was the best available option. It’s also completely incompatible with SemVer, and is a great example of why SemVer is at best a “nice in theory, fails in practice” idea.
Why not just tell them to upgrade? After all, your argument is essentially that stable APIs are impossible, so why bother with LTS? Every argument against semver also applies against LTS releases.
After all, your argument is essentially that stable APIs are impossible
My argument is that absolute perfect 100% binding commitment to never causing a change to observable behavior ever under any circumstance, unless also incrementing the major version at the same time and immediately dropping support for all users of previous versions, is not practicable in the real world, but is what SemVer requires. Not committing to SemVer gives flexibility to do things like long-term support releases, and generally people have been quite happy with them and also accepting of the single-digit number of times something had to change to fix a security issue.
“Support for that LTS is cut off effective immediately, everybody upgrade across a major version right now” is a non-starter
If it’s a non-starter then nobody should be getting the critical security patch. You’re upgrading from 2.2 to 3.0 and calling it 2.2.1 instead. That doesn’t change the fact that a breaking change happened and you didn’t bump the major version number.
You can’t issue promises like “2.2.X will have long term support” because that’s akin to knowing the future. Use a codename or something.
It’s pretty clear you’re committed to perfect technical adherence to a rule, without really giving consideration to why the rule exists. Especially if you’re at the point of “don’t commit to supporting things, because supporting things leads to breaking SemVer”.
They should probably use something like SemVer but with four parts, e.g.
Feature.Major.Minor.Patch
Feature version changes -> We’ve made significant changes / a new release (considered breaking)
Major version change -> We’ve made breaking changes
Minor version change -> Non breaking new features
Patch version change -> Other non-breaking changes
That way 2.*.*.* could be an LTS release, which would only get bug fixes, but if there was an unavoidable breaking change to fix a bug, you’d signal this in the version by e.g. going from 2.0.5.12 to 2.1.0.0. Users will have to deal with the breaking changes required to fix the bug, but they don’t have to deal with all the other major changes which have gone into the next ‘Feature’ release, 3.*.*.*. The promise that 2.*.*.*, as an LTS, will get bug fixes is honored. The promise that the major version must change on a breaking change is also honored.
SemVer doesn’t work if you try to imbue the numbers with additional meanings that can contradict the SemVer meanings.
every change potentially alters the observable behavior of the software
This is trivially false. Adding a new helper function to a module, for example, will never break backwards compatibility.
In contrast, changing a function’s input or output type is always a breaking change.
By failing to even attempt to distinguish between non-breaking and breaking changes, you’re offloading work onto the package’s users.
Optimize for what should be the common case: non-breaking changes.
Edit: to expand on this, examples abound in the Python ecosystem of unnecessary and intentional breaking changes in “minor” releases. Take a look at the numpy release notes for plenty of examples.
Python’s dynamic nature makes “adding a helper function” a potentially breaking change. What if someone was querying, say, all definitions of a module and relying on the length somehow? I know this is a bit of a stretch, but it is possible that such a change would break code. I still value semver though.
If you can access it at run-time, then someone will depend on it, and it’s a bit late to call it “not public”. Blame Python for exposing stuff like the call stack to introspection.
You seriously never know what kinds of things people might be relying on, and a mere definition of compatibility in terms of input and output types is woefully insufficient to capture the things people will expect in terms of backwards compatibility.
No, it does not descripbe a violation of SemVer, because spacebar heating is not a public API. SemVer is very clear about this. You are right people will still complain about backward compatibility even if you are keeping 100% correct SemVer.
I don’t think I have a naïve view of versioning; putting on my professional hat here, I have a decade of experience dealing with a dependency modeling system that handles the versions of hundreds of thousands of interrelated software artifacts that are versioned more or less independently of each other, across dozens of programming languages and runtimes. So… some experience here.
In all of this time, I’ve seen every single kind of breaking change I could imagine beforehand, and many I could not. They occurred independent of how the vendor of the code thought of it; a vendor of a versioned library might think that their change is minor, or even just a non-impacting patch, but outside of pure README changes, it turns out that they can definitely be wrong. They certainly had good intentions to communicate the nature of the change, but that intention can run hard into reality. In the end, the only way to be sure is to pin your dependencies, all the way down, and to test assiduously. And then upgrade them frequently, intentionally, and on a cadence that you can manage.
I would agree if violations were rare. Every time I’ve tried to solve dependency issues on Python, about 75% of the packages I look into have broken semver on some level. Granted, I probably have a biased sampling technique, but I find it extremely hard to believe that it’s a rare issue.
Backwards compatibility is hard to reason about, and the skill is by no means pervasive. Even having a lot of experience looking for compatibility breaks, I still let things slip, because it can be hard to detect. One of my gripes with semver is that it doesn’t scale. It assumes that tens of thousands of open source devs with no common training program or management structure all understand what a backwards breaking change is, and how to fix it.
Testing for compatibility breaks is rare. I can’t think of any Python frameworks that help here. Nor can I think of any other languages that address this (Erlang might, but I haven’t worked with it first-hand). The most likely projects to test for compatibility between releases are those that manage data on disk or network packets. Even among those, many rely on code & design review to spot issues.
It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.
It’s more likely that current package managers force you into semver regardless if you understand how it’s supposed to be used. The “statement of values” angle is appealing, but without much evidence. Semver is merely popular.
I guess this depends on a specific ecosystem? Rust projects use a lot of dependencies, all those deps use semver, and, in practice, issues rarely arise. This I think is a combination of:
the fact that semver is the only option in Rust
the combination of guideline to not commit Cargo.lock for libraries + cargo picking maximal versions by default. This way, accidental incompatibilities are quickly discovered & packages are yanked.
the guideline to commit Cargo.lock for binaries and otherwise final artifacts: that way folks who use Rust and who have the most of deps are shielded from incompatible updates.
the fact that “library” is a first-class language construct (crate) and not merely a package manager convention + associated visibility rules makes it easier to distinguish between public & private API.
Built-in support for writing test from the outside, as-if you are consumer of the library, which also catches semver-incompatible changes.
This is not to say that semver issues do not happen, just that they are rare enough. I’ve worked with Rust projects with 200-500 different deps, and didn’t pensive semver breakage being a problem.
I would add that the Rust type system is expressive enough that many backwards incompatible changes require type signature changes which are much more obvious than violations of some implicit contract.
to be either dangerously naive or outright dishonest
This phrase gets bandied around the internet so much I’m surprised its not a meme.
SemVer is … okay, but you make it sound like lives depend on it. There’s a lot of software running mission critical systems without using SemVer and people aren’t dying everyday because of it. I think we can calm down.
Thats the problem of the package management being so old. Back then semantic versioning wasnt that common and it never really caught on. In my opinion the PyPA should make a push to make more packages use semantic versioning. I‘m seeing this trend already, but its too slow…
I’m baffled that none of the frustrated developers use a lock file to pin transitive dependencies. Some, including a Nokia developer, attempt to build and run a staging environment containing “latest” code from the internet in their transitive dependencies. This hardly seems like the first time a transitive dependency might break you — and at least in this instance, it’s a VERY clearly worded build-time breakage! What if there’s a runtime issue caused by transitive dependency silent version update?
Why am I not surprised to see drewdevault making a ranty blogpost and leading to issue locking by starting to talk about stuff that isn’t related to the rust developers or bad decisions of developers at all. Or just against the decisions of the package developers…
What is the actual path forward fixing the problem? Bringing Rust/LLVM support to all of those platforms? I can understand the reasoning by the maintainers that C is inherently insecure, but not being able to use the package for the foreseeable future isn‘t really an option either. Well it might spark some innovation :D
Speaking in realistic terms and fully acknowledging that it is in some ways a sad state of affairs. Most of those platforms are dying and continuing to keep the alive is effectively volunteering to keep porting advancements from the rest of the world onto the platform. If you want to use third party packages on an AIX box you sort of just have to expect that you’ll have to share the labor of keeping that third party package working on AIX. The maintainers are unlikely to be thinking about you and for good reason.
For users of Alpine linux the choice to use a distro that is sufficiently outside the mainstream means you are also effectively committing to help port advancements from the rest of the world onto the platform if you want to consistently use them.
For both categories you can avoid that implicit commitment by moving to more current and mainstream platforms.
Exactly. And as a long term implication, if Rust is here to stay, the inevitable fate of those platforms is “Gain Rust support or die”. Maintaining a C implementation of everything in the name of backward compatibility is only delaying the inevitable, and is ultimately a waste of time.
I’m not saying that you shouldn’t to maintain a C implementation. It’s never going to be as memory safe as Rust is out of the box so you are going to have to expend way more effort than a rust implementation to do so and your users will be less safe. However expecting a third party to maintain that implementation for you because you decided to use a platform on the edges is never going to work. Using a platform on the edges means you are signing up to do that work yourself in a lot of cases. And that’s fine as long as you realize what you are signing up for.
This definitely seems to be true. I helped drive Rust support for a relatively niche platform (illumos), and while it took a while to get it squared away everybody I spoke with was friendly and helpful along the way. We’re a Tier 2 platform now, and lots of Rust-based software just works. We had a similar experience with Go, another critical runtime in 2021, which also required bootstrapping, being no longer written in C and so on.
Rust crates tend to rely on recently added features and/or library functions. Given that GCC releases are far less frequent, I think there will be a lot of friction when using the GCC frontend.
It’s tagged [rust], but the project seems to be in Python?
Someone (the maintainers?) added a Rust dependency, and now the software won’t build on some architectures?
I’ll unflag this as off-topic if I can a good explanation to why this on-topic, and not some misguided attempt to get more upvotes on a random GH issue.
As from the sheer amount of comments and reaction on the issue, this is a large deal. I’ve never seen a Github issue this active. It is a Python project, but the main problem is that LLVM/Rust is not available on some platforms. So I think both tags are justified.
It’s a real-life case of a practical problem at an intersection of quite a few technical and social topics important for programmers (package management, backwards compatibility, security, cross-platform-ness, FFI), with no clear “perfect” solution, which makes various compromises in how to solve it worth pondering and discussing as to possible consequences/pros and cons.
A high “value density” case-study which has potential to bear learnings that can later be used to educate smaller day-to-day cases, by giving insights as to potential consequences of specific decisions.
Yes, the discussion here has been enlightening. The discussion on GH was way too in media res for someone who wasn’t already caught up on the issue to get much out of it.
For example, I’m still unclear about exactly what “pyca” is. It’s not a standard part of Python? If it isn’t, why is this project specifically so interesting that people who build docker images use it?
A thoughtful blog post (like the top comment by @ubernostrum) would have been a much better submission imho.
pyca stands for Python Cryptographic Authority.
It’s an organization that regroups python libraries and tools related to cryptography. They are regrouped to allow better interoperability between them. The term “Authority” is, in my opinion, a little bit excessive as nobody requires you to use them; but as they are used by building block tools, you still use them transitively, hence the shitload of comments in the issue.
This is not the only “Authority” in the python world, there is also the Python Packaging Authority (pypa), the Python Code Quality Authority (PyCQA), and maybe other ones that I don’t know about.
As they often involve python core developers, their “Authority” status looks genuine.
I found it to be relevant and informative, but that’s probably just because my day job includes writing and maintaining Python and because I use Rust in several of my side projects.
As far as I understand it, Gentoo will be able to work around this for now, but ultimately Gentoo can only run on architectures that are supported by Python. And while dev-python/cryptography may not be strictly required when bootstrapping a Gentoo system, cryptography plays an important role how distributions, like Gentoo, manage the system.
Eventually, I think is premature for pyca/cryptography to require Rust as hard dependency when it reduces the set of supported hardware architectures.
Yes, type- and memory-safe languages have a clear advantage over C, especially when sensible code for cryptography or network services is involved. But dropping cryptography support for architectures that previously where supported is probably far worse. I’d really expected (and wished) pyca/cryptography would go with Rust as optional alternative, while supporting the native C code for a little longer.
I like Alphas, but people aren’t using them in production (read: people like me using them for hyucks), or they’re using them for a specific application, which this is irrelevant because chances are their BLISS program on VMS doesn’t care about it.
For the former, the nerds have to care about this, and for the latter, VSI has to care enough when enough customers care. Some platforms, the nerds just don’t care - see Itanium or s390. 68k, nerds care about, and I’m sure IBM/Oracle care enough about their own platforms to keep things working.
Rust is probably the canary for if a platform has enough critical mass for people to start fixing things, or if it’s there out of inertia.
Disclaimer: I maintain stuff for AIX, but I’m not associated with IBM.
Rust is probably the canary for if a platform has enough critical mass for people to start fixing things, or if it’s there out of inertia.
Then communicate this and don’t have it shake out implicitly from the change. Again, I’m just asking for empathy. It’s the same reason anyone gets angry at change. Look, I’ve been on this side of the issue at $WORK myself. My team pushes a migration and we catch others unaware. The change is necessary in our view, so we’re not willing to pull the change back. But the last thing I’m going to do is to flippantly tell them to migrate or be left behind. I’ll commiserate, empathize, and try to work on a path forward. I realize that the maintainers are not getting paid and so there’s less of an incentive to be “nice”, but I still maintain this isn’t the best look. As moral as it is to write cryptographic code in a memory-safe language, it’s equally moral to help guide distressed members of the community, or at least help engage them.
I realize that the maintainers are not getting paid and so there’s less of an incentive to be “nice”, but I still maintain this isn’t the best look.
Insisting that the volunteers whose work you get for free should be doing more isn’t a real great look either.
Maybe it’s a cultural thing, but “Migrate or be left behind” is exactly the kind of messaging I want from my upstream; the last thing I want is to dig through a wall of prose to find the actual information.
Insisting that the volunteers whose work you get for free should be doing more isn’t a real great look either.
Everyone here is working for free. The cryptography authors are affecting distro maintainers. Neither of these parties are working for pay. The paid employees of corporations have entire days to do this, they can afford to spend an entire day pinning down their transitive dependencies for this. It’s the distro maintainers that are scrambling now. The only ones that lose with cryptography’s stance and lackluster messaging is other unpaid volunteers.
Maybe it’s a cultural thing, but “Migrate or be left behind” is exactly the kind of messaging I want from my upstream; the last thing I want is to dig through a wall of prose to find the actual information.
That is most definitely not what I want as a distro maintainer, or even as a proprietary software developer. I want well communicated timelines. The Linux kernel does an amazing job of this, even though much like with cryptography, you can argue that the authors and maintainers of the kernel can really do whatever they want.
The paid employees of corporations have entire days to do this, they can afford to spend an entire day pinning down their transitive dependencies for this
90% of the complaints in that thread are paid employees getting mad at volunteer maintainers for breaking their builds.
Distro maintainers like yourself are doing great work & I can’t thank you enough.
I want well communicated timelines.
What does well communicated look like to you? As far as I was aware, it was on their site, twitter and mailing list.
Is the underlying problem that transitive dependencies aren’t “on the map” as something everyone needs to stay on top of? I feel like I’ve never seen a project (paid or otherwise) seriously look at them.
90% of the complaints in that thread are paid employees getting mad at volunteer maintainers for breaking their builds.
While I haven’t bothered keeping score, I wouldn’t doubt it. And ddevault didn’t do any favors either with his childish little rant in that thread. That being said, I couldn’t care less about paid employees. Having been a paid employee myself who has had to rev pipelines due to dependency changes, yes it’s annoying, but it is what I get paid to do.
What does well communicated look like to you? As far as I was aware, it was on their site, twitter and mailing list.
Honestly, I wish this version would have been released with a big change in the Readme or Changelog telling us that Rust was incoming in the next version, then revving in the subsequent version. Personally, I think that’s enough notification for us to at least get the Tier 2 bootstrapping in motion, and even if we can’t get it all done in time for the next release, it’ll at least allow us to stay pinned for only a small amount of time. I welcome using Rust in our low-level tools personally, because I think Rust really shines when it comes to writing CLI tools, and I’d love to see some of our heavy lifting Python scripts be eventually replaced by Rust tools.
Is the underlying problem that transitive dependencies aren’t “on the map” as something everyone needs to stay on top of? I feel like I’ve never seen a project (paid or otherwise) seriously look at them.
Sadly yeah, this is about how transitive dependencies play with package ecosystems at large. It’s a complicated problem and, personally, I really hate pip and the Python ecosystem as a whole with its inconsistent view of versioning.
The Linux kernel does an amazing job of this, even though much like with cryptography, you can argue that the authors and maintainers of the kernel can really do whatever they want.
The Linux kernel is backed by contributors paid to work on it and has a far far larger community of contributors. The Python Cryptographic Authority, as best I can tell, runs entirely on the efforts of a very small number of volunteers without any funding. It seems very weird to me to expect any open source project to provide the same level of quality as the worlds largest and probably best funded open source project.
I’m not disagreeing with you, I just want to note that the Linux kernel is more of a “platinum standard” in this case. We would all like to judge ourselves according to how close or far we get to coordinating and cooperating like the Linux kernel does (though not Linus!).
Look, I get it. The maintainers don’t want to keep maintaining C anymore, they feel Rust is the clear/clean option going forward. Moreover those architectures really are moribund, on their way out. I don’t think we should hold the maintainers to maintain/write code they don’t care about for a few moribund platforms either. But I don’t understand why there’s so much Rust aggression both in the GH issue and in this thread. Distro maintainers are having a bad day today, and honestly if both Gentoo and Alpine were affected, that means the messaging was not clear enough about this change. We can continue to advocate for the rights of maintainers/authors and push the Rust language forward while being respectful of the real pain it causes distro maintainers and other users alike. Moribund though those architectures may be, those folks are running around figuring out how to patch this properly on their systems and how to deal with this going forward. The lack of empathy from a lot of these posts is not a good look for the Rust community.
Sure, that’s the tricky question. In my experience, the best time to deprecate a feature/stop supporting something is a few years after you believe that it is the right time to do so. Add a little more time if it’s something crucial, like something cryptography related.
OTOH, I can’t expect anything from a third party I have no contract with. And I see the appeal of Rust.
I think supporting older architectures is a laudable goal in theory, but after a certain amount of time none of these machines are being used in production and there’s less and less reasons for them to be running the latest versions of software.
I’m really grateful for being able to run OpenBSD on an old AlphaStation, but I never relied on that machine for anything other than fun.
Lots of people seems to be updating dependencies without looking into what has changed, nor any kind of audit. And we are talking about a projects that use cryptography. This is negligence.
No, most people are running into this with transitive dependencies, where a 2nd or 3rd level dependency specified something like cryptography<=2.3.0 in that project’s setup.py. I would guess that most people generally only pin first level dependencies as a matter of course. I further posit that python packaging/tooling is doing folks few favors here.
All these problems disappear like tears in the rain if we stop insisting everybody re-compile everything all the time, and native extensions to python|node|whatevs came in wasm form.
For those who are curious about what’s going on and lack context…
Let’s start with a quick refresher on Python packaging. When you use Python’s own default toolchain to package up and distribute some code, you generally have a choice of either or both of two types of artifacts: a “source distribution”, which is a
.tar.gzarchive whose installation on the target system may involve a build step and thus require the target system to have the requisite compiler toolchain and non-Python dependencies (such as C libraries it needs to link against). Or a binary distribution, which is a file with the extension.whl[note 1], and contains everything already compiled and packed up for a specific target operating system and CPU architecture, so that installation only requires unpacking the.whland putting files in the correct location.And although Python technically exposes only a C API, that API is robust enough, and other languages’ interop with C is similarly robust enough, that it’s possible to have Python use compiled extensions that communicate with a variety of other languages. The numeric/scientific Python stack, for example, interfaces with linear-algebra libraries which are written in Fortran (and thus building from source requires a Fortran compiler; it’s much nicer to install from a precompiled
.whl).So. This library – which is not part of the Python standard library, and also not required for common tasks like speaking HTTPS or writing scripts that will control other machines through SSH – had some compiled extensions which were written in C. Now, the library is in the process of moving the extensions from C to Rust. The latest release builds the Rust by default but does not have a hard dependency on it (you can disable with a build-time environment flag). The next release will have a hard dependency on it, and as such will require the now-ported-to-Rust extension code in order to work at all.
There appear to be two groups affected by this.
One group is people who were building Python applications in containers based on Alpine Linux. Python’s
.whlformat supports precompiled packages for Linux distros, but is based on a lowest-common-denominator definition of “Linux” that Alpine – with its alternative libc – does not meet. As a result, people using Alpine as their base image must build from source, and apparently only Alpine 3.13 and later have a sufficiently recent Rust to be able to do this. Inserting an aside of my own recommendation: in general, if you are looking to containerize Python applications, Alpine is a poor choice of base image. This article does a good job explaining why, but the tl;dr is the alleged benefits of Alpine mostly manifest when using other languages like Go, and actually become downsides when doing Python (since now you get to compile the world every time you rely on a Python package with an extension written in C or other non-Python language).The other group is people who are on certain types of systems that are unsupported by Rust itself, and thus cannot build the ported-to-Rust extension modules now used in this package.
Now, that out of the way, I think the Cryptography team made the right choice. Their options are to stay in C and keep the attendant risks that brings, thus exposing every user to nebulous but likely future security issues, or switch to something memory-safe but low-level and performant enough to be a reasonable replacement, but at the cost of making Alpine users’ lives a bit harder and leaving a much smaller number of users completely unsupported. For a security-critical library which treats security as its first priority, I think this is an unpleasant (because nobody ever likes cutting someone else off, and because it generates heated complaint threads) but not particularly difficult choice to make.
Side arguments about “
$THINGwould have fixed this” where$THINGis one of “alternative versioning scheme”, “alternative/additional announcement channel”, and so on also don’t really work for reasons that become clear after reading the GitHub thread.[note 1] This is a tortured joke. Python was named after Monty Python, and much of its early documentation contained references to Monty Python sketches. The original name of the Python Package Index was the Python Cheese Shop, in homage to one such sketch. Many types of cheese are distributed in large round blocks called wheels, so this type of Python package is called a “wheel” and has extension
.whl.For those ecosystems without Rust, might something like mrustc be an option? It’d be a somewhat torturous path, but transpile Rust to C, then you could use the Python C bindings …
Generalised, this could be a good option for dealing with Rust dependencies on a wide range of languages.
This actually works pretty well. Someone brought up Rust ESP8266 ecosystem using mrustc.
I really appreciate the in depth write up. Great stuff
Wouldn’t it be possible to ship with Rust by default, but also accept a C fallback for cases where Rust is not (easily) available? I agree that in principle Rust is probably the better choice, but it’s not like the C code has become insecure or “bad” all of the sudden, and practicality beats purity IMO.
This would meet both needs; it was mentioned in passing in one comment, but probably got lost in all the noise/rather off-topic complaining/ranting. The biggest downside is the increased maintenance burden, but I’d just offer a “if you want it, you can maintain it” and/or “if you want it, this is how much it will cost”. If no one (or too few people) take up the offer then it probably wasn’t all that important after all.
Practically speaking if there is enough interest somebody will fork it to keep the C backend alive. The level of disgruntlement alone will drive that along for a while, long term I don’t think that will carry through. In some cases it makes more sense to just fix the Rust toolchain to support cross compiling to your favorite architecture. Once a few more platforms take than initiative the will to maintain a legacy fork will die off. Better for the health of the main project to be free of the burden early rather than later.
The problem with a fork is that it won’t fix cases where this library is an indirect dependency (i.e. my app imports
libfoowhich imports this). I’m not sure if Python provides a good way to override this?Yeah, this would be the best long-term solution, but also not something that will be done overnight. I’m just trying to think of a solution that will fix the problems of today, so that there’s time to fix the longer term issues without too much friction.
Python package maintainers rarely use semantic versioning and often break backwards compatibility in minor releases. One of several reasons that dependency management is a nightmare in Python world.
I generally consider semantic versioning to be a well-intentioned falsehood. I don’t think that package vendors can have effective insight into which of their changes break compatibility when they can’t have a full bottom-up consumer graph for everyone who uses it.
I don’t think that Python gets this any worse than any other language.
I’ve heard this opinion expressed before… I find it to be either dangerously naive or outright dishonest. There’s a world of difference between a) the rare bug fix release or nominally-orthogonal-feature-add release that unintentionally breaks downstream code and b) intentionally changing and deprecating API’s in “minor” releases.
In my view, adopting SemVer is a statement of values and intention. It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.
A “statement of values and intention” carries no binding commitment. And the fact that you have to hedge with “as much as is reasonably possible” and “only knowingly break” kind of gives away what the real problem is: every change potentially alters the observable behavior of the software in a way that will break someone’s reliance on the previous behavior, and therefore the only way to truly follow SemVer is to increment major on every commit. Which is the same as declaring the version number to be meaningless, since if every change is a compatibility break, there’s no useful information to be gleaned from seeing the version number increment.
And that’s without getting into some of my own direct experience. For example, I’ve been on the Django security team for many years, and from time to time someone has found a security issue in Django that cannot be fixed in a backwards-compatible way. Thankfully fewer of those in recent years since many of them related to weird old functionality dating to Django’s days as a newspaper CMS, but they do happen. Anyway, SemVer’s answer to this is “then either don’t fix it, or do but no matter how you fix it you’ve broken SemVer and people on the internet will scream at you and tell you that you ought to be following SemVer”. Not being a fan of no-win situations, I am content that Django has never and likely never will commit to following SemVer.
A label on a jar carries no binding commitment to the contents of the jar. I still appreciate that my salt and sugar are labelled differently.
Selling the jar with that label on it in many countries is a binding commitment and puts you under the coverage of food safety laws, though.
What do you mean? SemVer’s answer to “this bug can’t be fixed in a backwards-compatible way” is to increment the major version to indicate a breaking change. You probably also want to get the message across to your users by pushing a new release of the old major version which prints some noisy “this version of blah is deprecated and has security issues” messages to the logs.
It’s not perfect, I’m not saying SemVer is a silver bullet. I’m especially worried about the effects of basing automated tooling on the assumption that no package would ever push a minor or patch release with a breaking change; it seems to cause ecosystems like the NPM to be highly fragile. But when taken as a statement of intent rather than a guarantee, I think SemVer has value, and I don’t understand why you think your security issue anecdote requires breaking SemVer.
So, let’s consider Django, because I know that well (as mentioned above). Typically Django does a feature release (minor version bump) every 8 months or so, and every third one bumps the major version and completes a deprecation cycle. So right now Django 3.1 is the latest release; next will be 3.2 (every X.2 is an LTS), then 4.0.
And the support matrix consists of the most recent feature release (full bugfix and security support), the one before that (security support only), and usually one LTS (but there’s a period at the end of each where two of them overlap). The policy is that if you run on a given LTS with no deprecation warnings issued from your code, you’re good to upgrade to the next (which will be a major version bump; for example, if you’re on 2.2 LTS right now, your next LTS will be 3.2).
But… what happens when a bug is found in an LTS that can’t be fixed in a backwards-compatible way? Especially a security issue? “Support for that LTS is cut off effective immediately, everybody upgrade across a major version right now” is a non-starter, but is what you propose as the correct answer. The only option is to break SemVer and do the backwards-incompatible change as a bugfix release of the LTS. Which then leads to “why don’t you follow SemVer” complaints. Well, because following SemVer would actually be worse for users than this option is.
Why do people run an LTS version, if not for being able to avoid worrying about it as a dependency? If you’re making incompatible changes: forget about semver, you’re breaking the LTS contract, and you may as well tell drop the LTS tag and people to run the latest.
I can think of only a couple instances in the history of Django where it happened that a security issue couldn’t be fixed in a completely backwards-compatible way. Minimizing the breakage for people – by shipping the fix into supported releases – was the best available option. It’s also completely incompatible with SemVer, and is a great example of why SemVer is at best a “nice in theory, fails in practice” idea.
Why not just tell them to upgrade? After all, your argument is essentially that stable APIs are impossible, so why bother with LTS? Every argument against semver also applies against LTS releases.
My argument is that absolute perfect 100% binding commitment to never causing a change to observable behavior ever under any circumstance, unless also incrementing the major version at the same time and immediately dropping support for all users of previous versions, is not practicable in the real world, but is what SemVer requires. Not committing to SemVer gives flexibility to do things like long-term support releases, and generally people have been quite happy with them and also accepting of the single-digit number of times something had to change to fix a security issue.
If it’s a non-starter then nobody should be getting the critical security patch. You’re upgrading from 2.2 to 3.0 and calling it 2.2.1 instead. That doesn’t change the fact that a breaking change happened and you didn’t bump the major version number.
You can’t issue promises like “2.2.X will have long term support” because that’s akin to knowing the future. Use a codename or something.
It’s pretty clear you’re committed to perfect technical adherence to a rule, without really giving consideration to why the rule exists. Especially if you’re at the point of “don’t commit to supporting things, because supporting things leads to breaking SemVer”.
They should probably use something like SemVer but with four parts, e.g. Feature.Major.Minor.Patch
That way 2.*.*.* could be an LTS release, which would only get bug fixes, but if there was an unavoidable breaking change to fix a bug, you’d signal this in the version by e.g. going from 2.0.5.12 to 2.1.0.0. Users will have to deal with the breaking changes required to fix the bug, but they don’t have to deal with all the other major changes which have gone into the next ‘Feature’ release, 3.*.*.*. The promise that 2.*.*.*, as an LTS, will get bug fixes is honored. The promise that the major version must change on a breaking change is also honored.
SemVer doesn’t work if you try to imbue the numbers with additional meanings that can contradict the SemVer meanings.
This scheme is very similar to Haskell’s Package Versioning Policy (PVP).
I’m saying supporting things and adhering to SemVer should be orthogonal.
This is trivially false. Adding a new helper function to a module, for example, will never break backwards compatibility.
In contrast, changing a function’s input or output type is always a breaking change.
By failing to even attempt to distinguish between non-breaking and breaking changes, you’re offloading work onto the package’s users.
Optimize for what should be the common case: non-breaking changes.
Edit: to expand on this, examples abound in the Python ecosystem of unnecessary and intentional breaking changes in “minor” releases. Take a look at the numpy release notes for plenty of examples.
Python’s dynamic nature makes “adding a helper function” a potentially breaking change. What if someone was querying, say, all definitions of a module and relying on the length somehow? I know this is a bit of a stretch, but it is possible that such a change would break code. I still value semver though.
The number of definitions in a module is not a public API. SemVer only applies to public APIs.
If you can access it at run-time, then someone will depend on it, and it’s a bit late to call it “not public”. Blame Python for exposing stuff like the call stack to introspection.
Eh no? SemVer is very clear about this. Public API is whatever software declares it to be. Undeclared things can’t be public API, by definition.
Python has no concept of public vs private. It’s all there all the time. As they say in python land, “We’re all consenting adults here”.
I’m sure, by the way, when Hettinger coined that phrase he didn’t purposely leave out those under the age of 18. Language is hard. :P
Does this comic describe a violation of SemVer?
You seriously never know what kinds of things people might be relying on, and a mere definition of compatibility in terms of input and output types is woefully insufficient to capture the things people will expect in terms of backwards compatibility.
No, it does not descripbe a violation of SemVer, because spacebar heating is not a public API. SemVer is very clear about this. You are right people will still complain about backward compatibility even if you are keeping 100% correct SemVer.
I don’t think I have a naïve view of versioning; putting on my professional hat here, I have a decade of experience dealing with a dependency modeling system that handles the versions of hundreds of thousands of interrelated software artifacts that are versioned more or less independently of each other, across dozens of programming languages and runtimes. So… some experience here.
In all of this time, I’ve seen every single kind of breaking change I could imagine beforehand, and many I could not. They occurred independent of how the vendor of the code thought of it; a vendor of a versioned library might think that their change is minor, or even just a non-impacting patch, but outside of pure README changes, it turns out that they can definitely be wrong. They certainly had good intentions to communicate the nature of the change, but that intention can run hard into reality. In the end, the only way to be sure is to pin your dependencies, all the way down, and to test assiduously. And then upgrade them frequently, intentionally, and on a cadence that you can manage.
Here here. My experience isn’t exactly like @offby1’s but I can vouch for the rest.
I would agree if violations were rare. Every time I’ve tried to solve dependency issues on Python, about 75% of the packages I look into have broken semver on some level. Granted, I probably have a biased sampling technique, but I find it extremely hard to believe that it’s a rare issue.
Backwards compatibility is hard to reason about, and the skill is by no means pervasive. Even having a lot of experience looking for compatibility breaks, I still let things slip, because it can be hard to detect. One of my gripes with semver is that it doesn’t scale. It assumes that tens of thousands of open source devs with no common training program or management structure all understand what a backwards breaking change is, and how to fix it.
Testing for compatibility breaks is rare. I can’t think of any Python frameworks that help here. Nor can I think of any other languages that address this (Erlang might, but I haven’t worked with it first-hand). The most likely projects to test for compatibility between releases are those that manage data on disk or network packets. Even among those, many rely on code & design review to spot issues.
It’s more likely that current package managers force you into semver regardless if you understand how it’s supposed to be used. The “statement of values” angle is appealing, but without much evidence. Semver is merely popular.
I guess this depends on a specific ecosystem? Rust projects use a lot of dependencies, all those deps use semver, and, in practice, issues rarely arise. This I think is a combination of:
This is not to say that semver issues do not happen, just that they are rare enough. I’ve worked with Rust projects with 200-500 different deps, and didn’t pensive semver breakage being a problem.
I would add that the Rust type system is expressive enough that many backwards incompatible changes require type signature changes which are much more obvious than violations of some implicit contract.
This phrase gets bandied around the internet so much I’m surprised its not a meme.
SemVer is … okay, but you make it sound like lives depend on it. There’s a lot of software running mission critical systems without using SemVer and people aren’t dying everyday because of it. I think we can calm down.
Thats the problem of the package management being so old. Back then semantic versioning wasnt that common and it never really caught on. In my opinion the PyPA should make a push to make more packages use semantic versioning. I‘m seeing this trend already, but its too slow…
I’m baffled that none of the frustrated developers use a lock file to pin transitive dependencies. Some, including a Nokia developer, attempt to build and run a staging environment containing “latest” code from the internet in their transitive dependencies. This hardly seems like the first time a transitive dependency might break you — and at least in this instance, it’s a VERY clearly worded build-time breakage! What if there’s a runtime issue caused by transitive dependency silent version update?
For some technical details from the maintainers’ perspective I found Alex Gaynor’s comment, somewhere in the middle of the deluge, worth reading: https://github.com/pyca/cryptography/issues/5771#issuecomment-775176581
Why am I not surprised to see drewdevault making a ranty blogpost and leading to issue locking by starting to talk about stuff that isn’t related to the rust developers or bad decisions of developers at all. Or just against the decisions of the package developers…
What is the actual path forward fixing the problem? Bringing Rust/LLVM support to all of those platforms? I can understand the reasoning by the maintainers that C is inherently insecure, but not being able to use the package for the foreseeable future isn‘t really an option either. Well it might spark some innovation :D
Speaking in realistic terms and fully acknowledging that it is in some ways a sad state of affairs. Most of those platforms are dying and continuing to keep the alive is effectively volunteering to keep porting advancements from the rest of the world onto the platform. If you want to use third party packages on an AIX box you sort of just have to expect that you’ll have to share the labor of keeping that third party package working on AIX. The maintainers are unlikely to be thinking about you and for good reason.
For users of Alpine linux the choice to use a distro that is sufficiently outside the mainstream means you are also effectively committing to help port advancements from the rest of the world onto the platform if you want to consistently use them.
For both categories you can avoid that implicit commitment by moving to more current and mainstream platforms.
Exactly. And as a long term implication, if Rust is here to stay, the inevitable fate of those platforms is “Gain Rust support or die”. Maintaining a C implementation of everything in the name of backward compatibility is only delaying the inevitable, and is ultimately a waste of time.
[Comment removed by moderator pushcx: When I said stop posting in the thread where you told someone to kill themself, I meant it.]
I’m not saying that you shouldn’t to maintain a C implementation. It’s never going to be as memory safe as Rust is out of the box so you are going to have to expend way more effort than a rust implementation to do so and your users will be less safe. However expecting a third party to maintain that implementation for you because you decided to use a platform on the edges is never going to work. Using a platform on the edges means you are signing up to do that work yourself in a lot of cases. And that’s fine as long as you realize what you are signing up for.
I see it like this: Rust is not going away. This won’t be the last project introducing rust support.
Either your plattform supports rust or the world will stop supporting your plattform.
This definitely seems to be true. I helped drive Rust support for a relatively niche platform (illumos), and while it took a while to get it squared away everybody I spoke with was friendly and helpful along the way. We’re a Tier 2 platform now, and lots of Rust-based software just works. We had a similar experience with Go, another critical runtime in 2021, which also required bootstrapping, being no longer written in C and so on.
I mean, you can always pin your dependency to the version before this one. No way that could come back and bite you </sarcasm>
I think GCC frontend for Rust, which recently got funded, will solve this problem.
Rust crates tend to rely on recently added features and/or library functions. Given that GCC releases are far less frequent, I think there will be a lot of friction when using the GCC frontend.
Maintaining compatibility with a 6/9/12 month old rust compiler is a much smaller ask of the maintainers than maintaining a C library indefinitely.
I don’t get this.
It’s tagged [rust], but the project seems to be in Python?
Someone (the maintainers?) added a Rust dependency, and now the software won’t build on some architectures?
I’ll unflag this as off-topic if I can a good explanation to why this on-topic, and not some misguided attempt to get more upvotes on a random GH issue.
As from the sheer amount of comments and reaction on the issue, this is a large deal. I’ve never seen a Github issue this active. It is a Python project, but the main problem is that LLVM/Rust is not available on some platforms. So I think both tags are justified.
It’s a real-life case of a practical problem at an intersection of quite a few technical and social topics important for programmers (package management, backwards compatibility, security, cross-platform-ness, FFI), with no clear “perfect” solution, which makes various compromises in how to solve it worth pondering and discussing as to possible consequences/pros and cons.
A high “value density” case-study which has potential to bear learnings that can later be used to educate smaller day-to-day cases, by giving insights as to potential consequences of specific decisions.
Yes, the discussion here has been enlightening. The discussion on GH was way too in media res for someone who wasn’t already caught up on the issue to get much out of it.
For example, I’m still unclear about exactly what “pyca” is. It’s not a standard part of Python? If it isn’t, why is this project specifically so interesting that people who build docker images use it?
A thoughtful blog post (like the top comment by @ubernostrum) would have been a much better submission imho.
But I’ve removed my flag now, anyway.
pycastands for Python Cryptographic Authority. It’s an organization that regroups python libraries and tools related to cryptography. They are regrouped to allow better interoperability between them. The term “Authority” is, in my opinion, a little bit excessive as nobody requires you to use them; but as they are used by building block tools, you still use them transitively, hence the shitload of comments in the issue.This is not the only “Authority” in the python world, there is also the Python Packaging Authority (pypa), the Python Code Quality Authority (PyCQA), and maybe other ones that I don’t know about. As they often involve python core developers, their “Authority” status looks genuine.
I found it to be relevant and informative, but that’s probably just because my day job includes writing and maintaining Python and because I use Rust in several of my side projects.
Gentoo is also affected by this:
As far as I understand it, Gentoo will be able to work around this for now, but ultimately Gentoo can only run on architectures that are supported by Python. And while dev-python/cryptography may not be strictly required when bootstrapping a Gentoo system, cryptography plays an important role how distributions, like Gentoo, manage the system.
Eventually, I think is premature for pyca/cryptography to require Rust as hard dependency when it reduces the set of supported hardware architectures.
Yes, type- and memory-safe languages have a clear advantage over C, especially when sensible code for cryptography or network services is involved. But dropping cryptography support for architectures that previously where supported is probably far worse. I’d really expected (and wished) pyca/cryptography would go with Rust as optional alternative, while supporting the native C code for a little longer.
How much longer? Until Rust is ported to Alpha? Who will do the port? Until all Alpha machines are broken?
I like Alphas, but people aren’t using them in production (read: people like me using them for hyucks), or they’re using them for a specific application, which this is irrelevant because chances are their BLISS program on VMS doesn’t care about it.
For the former, the nerds have to care about this, and for the latter, VSI has to care enough when enough customers care. Some platforms, the nerds just don’t care - see Itanium or s390. 68k, nerds care about, and I’m sure IBM/Oracle care enough about their own platforms to keep things working.
Rust is probably the canary for if a platform has enough critical mass for people to start fixing things, or if it’s there out of inertia.
Disclaimer: I maintain stuff for AIX, but I’m not associated with IBM.
Then communicate this and don’t have it shake out implicitly from the change. Again, I’m just asking for empathy. It’s the same reason anyone gets angry at change. Look, I’ve been on this side of the issue at $WORK myself. My team pushes a migration and we catch others unaware. The change is necessary in our view, so we’re not willing to pull the change back. But the last thing I’m going to do is to flippantly tell them to migrate or be left behind. I’ll commiserate, empathize, and try to work on a path forward. I realize that the maintainers are not getting paid and so there’s less of an incentive to be “nice”, but I still maintain this isn’t the best look. As moral as it is to write cryptographic code in a memory-safe language, it’s equally moral to help guide distressed members of the community, or at least help engage them.
Insisting that the volunteers whose work you get for free should be doing more isn’t a real great look either.
Maybe it’s a cultural thing, but “Migrate or be left behind” is exactly the kind of messaging I want from my upstream; the last thing I want is to dig through a wall of prose to find the actual information.
Everyone here is working for free. The cryptography authors are affecting distro maintainers. Neither of these parties are working for pay. The paid employees of corporations have entire days to do this, they can afford to spend an entire day pinning down their transitive dependencies for this. It’s the distro maintainers that are scrambling now. The only ones that lose with cryptography’s stance and lackluster messaging is other unpaid volunteers.
That is most definitely not what I want as a distro maintainer, or even as a proprietary software developer. I want well communicated timelines. The Linux kernel does an amazing job of this, even though much like with
cryptography, you can argue that the authors and maintainers of the kernel can really do whatever they want.90% of the complaints in that thread are paid employees getting mad at volunteer maintainers for breaking their builds.
Distro maintainers like yourself are doing great work & I can’t thank you enough.
What does well communicated look like to you? As far as I was aware, it was on their site, twitter and mailing list.
Is the underlying problem that transitive dependencies aren’t “on the map” as something everyone needs to stay on top of? I feel like I’ve never seen a project (paid or otherwise) seriously look at them.
While I haven’t bothered keeping score, I wouldn’t doubt it. And ddevault didn’t do any favors either with his childish little rant in that thread. That being said, I couldn’t care less about paid employees. Having been a paid employee myself who has had to rev pipelines due to dependency changes, yes it’s annoying, but it is what I get paid to do.
Honestly, I wish this version would have been released with a big change in the Readme or Changelog telling us that Rust was incoming in the next version, then revving in the subsequent version. Personally, I think that’s enough notification for us to at least get the Tier 2 bootstrapping in motion, and even if we can’t get it all done in time for the next release, it’ll at least allow us to stay pinned for only a small amount of time. I welcome using Rust in our low-level tools personally, because I think Rust really shines when it comes to writing CLI tools, and I’d love to see some of our heavy lifting Python scripts be eventually replaced by Rust tools.
Sadly yeah, this is about how transitive dependencies play with package ecosystems at large. It’s a complicated problem and, personally, I really hate pip and the Python ecosystem as a whole with its inconsistent view of versioning.
EDIT: I’d like to reepmphasize how hard of a time distros are having. https://archives.gentoo.org/gentoo-dev/message/167c5156a80d6860e935a2ea92c6b6b5.
The Linux kernel is backed by contributors paid to work on it and has a far far larger community of contributors. The Python Cryptographic Authority, as best I can tell, runs entirely on the efforts of a very small number of volunteers without any funding. It seems very weird to me to expect any open source project to provide the same level of quality as the worlds largest and probably best funded open source project.
I’m not disagreeing with you, I just want to note that the Linux kernel is more of a “platinum standard” in this case. We would all like to judge ourselves according to how close or far we get to coordinating and cooperating like the Linux kernel does (though not Linus!).
Look, I get it. The maintainers don’t want to keep maintaining C anymore, they feel Rust is the clear/clean option going forward. Moreover those architectures really are moribund, on their way out. I don’t think we should hold the maintainers to maintain/write code they don’t care about for a few moribund platforms either. But I don’t understand why there’s so much Rust aggression both in the GH issue and in this thread. Distro maintainers are having a bad day today, and honestly if both Gentoo and Alpine were affected, that means the messaging was not clear enough about this change. We can continue to advocate for the rights of maintainers/authors and push the Rust language forward while being respectful of the real pain it causes distro maintainers and other users alike. Moribund though those architectures may be, those folks are running around figuring out how to patch this properly on their systems and how to deal with this going forward. The lack of empathy from a lot of these posts is not a good look for the Rust community.
Sure, that’s the tricky question. In my experience, the best time to deprecate a feature/stop supporting something is a few years after you believe that it is the right time to do so. Add a little more time if it’s something crucial, like something cryptography related.
OTOH, I can’t expect anything from a third party I have no contract with. And I see the appeal of Rust.
I think supporting older architectures is a laudable goal in theory, but after a certain amount of time none of these machines are being used in production and there’s less and less reasons for them to be running the latest versions of software.
I’m really grateful for being able to run OpenBSD on an old AlphaStation, but I never relied on that machine for anything other than fun.
Lots of people seems to be updating dependencies without looking into what has changed, nor any kind of audit. And we are talking about a projects that use cryptography. This is negligence.
They only depend on PyO3. If Rust is a problematic dependency, we have bigger problems ;).
No, most people are running into this with transitive dependencies, where a 2nd or 3rd level dependency specified something like
cryptography<=2.3.0in that project’ssetup.py. I would guess that most people generally only pin first level dependencies as a matter of course. I further posit that python packaging/tooling is doing folks few favors here.I wish rust had just added a ‘emit portable C’ option from day one.
That was never the goal. However if you want, then there is alternative compiler mrustc that supports that.
All these problems disappear like tears in the rain if we stop insisting everybody re-compile everything all the time, and native extensions to python|node|whatevs came in wasm form.
Yes, this has been great fun so far this morning.
/me waiting for a fork…