1. 28
  1.  

  2. 27

    Another difficulty is that many Rust crates can’t meaningfully exist as standalone dynamically-linked libraries, even if they wanted to, even if Rust blessed its ABI as stable. Many of Rust’s abstractions rely heavily on inlining, and generics are based on monomorphisation (basically code generated at compile time).

    You can’t realistically have serde as a standalone dynamic library, because it implements deserialization code for your types in your crate. This code doesn’t exist in serde, and even if you update bulk of the serde library, your derived code won’t.

    1. 12

      We go through this every time some language designer comes up with a way of packaging and distributing: CPAN,pip,gems,npm,crates and it goes on and on. It seems like everybody likes re-inventing the distribution wheel.

      Short version is, RPMs and debs have been around for 25 years, and, while they were originally designed with C in mind, they’re flexible enough to incorporate programs written in any language.

      Yes, they have restrictions, and some of these restrictions are uncomfortable to people who do all their work in one or two particular languages, but distros like Debian and Redhat target people who just want to have a system that works out of the box, with programs written in many different languages co-existing in one reasonably coherent system. This is the problem distro packages like RPM and deb are trying to solve.

      I appreciate that if I’m working in ruby, I usually have something like rvm, and bundler, and other utilities for keeping multiple ruby development environments in a sane way, and I appreciate that others like these tools for programming in their preferred environment.

      However. If I just want to be able to install and use a script written in python (take for example, “Terminator” that is written in python), as a distro, I just want to be able to install the script, and ensure that it works cleanly with the rest of the system, I don’t care about setting up a special environment and managing dependencies, and all of the other baggage that these other distribution methods involve.

      1. 18

        Short version is, RPMs and debs have been around for 25 years, and, while they were originally designed with C in mind, they’re flexible enough to incorporate programs written in any language.

        No, they don’t. They don’t even work well for C++, and have been hobbling the development of template libraries for over 10 years now because of how bad they are at handling API-stable-ABI-unstable libraries.

        1. 2

          What’s the problem? It’s been a while since I looked at these in Linux environments, but in the FreeBSD ports collections if you upgrade a C++ library, it bumps the PORTREVISION of all of the ports that depend on that library. The next time a package set is built, the packages use the new version of the library. The FreeBSD project explicitly moved away from building packages outside of package sets to address this kind of thing - it’s easy for things to get out of sync and computers are so fast now that building everything (30,000+ packages) on a single machine is feasible in about a day.

        2. 10
          • apt doesn’t handle multiple versions of the same library well. You can do it by renaming packages and an indirection layer of virtual packages, but this is tortured compared to auto deduplicating dependencies according to semver.

          • People in charge of Linux package managers generally don’t care about Windows. There is nothing technically preventing RPM/deb from working on Windows, except that it “sucks” (for shits and giggles I’ve made cargo-deb Windows compatible, and it happily builds Windows-only deb packages).

            npm has got a huge traction as a method of distributing dev tools, because it supports Windows as a first-class platform.

          • RPM/deb as an archive format doesn’t make that much of a difference. Rust/Python/JS could use .deb as their format, after all it’s just a bunch of files together. The real issue is where do you publish to, and how other people find it. Linux distros set themselves as gatekeepers, which has its own value, but it’s very different from npm/cargo, etc. that have npm publish free for all with zero hassle, way easier than maintaining PPAs.

          Having written all that, I realize it all comes to usability. The distinction between “it can do it” vs “it just works”.

          1. 8

            I never want to care about “the rest of the system”. I want to write complete programs, not shells of programs that work if slotted into the right system.

            The more programs are like this, the less a system that works out of the box is a problem to think about.

            1. 5

              I’d argue the packaging system put up by Linux distros is actually not flexible, but rather the human sitting between applications to be packaged and the Linux distribution sinking hours into patching software to fit into the packaging system is. If rpm/deb were actually flexible we would not have these conflicts.

              It seems like everybody likes re-inventing the distribution wheel.

              I can say the same about the state of Linux distributions. It seems that I as an application developer can only rely on the featureset that is the intersection of all the distro’s package managers if I were to follow your advice.

              1. 2

                rpm/deb is fairly flexible. The reason you get these arguments is mostly because of distribution policies, not because deb/rpm can’t do it.

                https://lwn.net/Articles/843313/rss

                1. 2

                  I did not claim rpm/deb cannot deal with large bundled apps (that’s fairly trivial, curl|sh can do that too). I’m saying rpm/deb cannot deal with dependency management to the granularity npm/cargo can, and then not in an efficient manner. Kornel already replied with other examples where rpm/deb can’t do things.

              2. 7

                Then the distribution maintainers should solve that problem within the constraints of the language and ecosystem the tool was developed in. The language and ecosystem are not going to change nor should they. If RPMs and debs can handle the change then they should just package them and move on. Complaining that new ways of doing development and deployment make the job harder helps no one. Either the distributions will adapt or they will lose the next generation of software developed in these new paradigms.

                1. 9

                  A CVE gets assigned some widely popular library. For fun we will say they have a monthly release cadence and the bug is published mid release cycle. Upstream is not done with their integration tests and don’t want the release just for a 3 line patch, even if the issue is severe. Lets say it’s used for around 30 binaries.

                  What do you do?

                  If the solution here is to do some wonkey patching of Cargo.toml and Cargo.lock across 30 packages to ensure they are pulling the correct patch (is it even possible), how does this scale?

                  This isn’t the question of distributions adapting to anything, this is “the next generation of software” digging a deep grave and ignoring almost 3 decades worth of experience. This isn’t me claiming we should pretend Rust is C and package all crates like individual packages. Nobody has the time for that. But pretending this isn’t a problem is deeply unsettling.

                  1. 10

                    I don’t know, it’s not my problem, but if it were, I guess I would try solving it rather than trying to wedge everything into this old C paradigm.

                    1. 4

                      Not having a solution doesn’t mean you can just paint an old solution as terrible.

                      Lots of smart people are working on separating language ecosystems from system administration, and we have these problems. So now, what do we do?

                    2. 4

                      Upstream is not done with their integration tests and don’t want the release just for a 3 line patch, even if the issue is severe.

                      In the case of uncooperative and irresponsible upstreams, what Debian does is say “we will package this, but it is not covered by the security support that we provide for the rest of the OS”. They used to do this for webkit and node.

                      What else can you do? At some point packaging implies cooperation.

                      1. 3

                        Cargo has multiple features for replacing dependencies. For individual packages you drop [patch.crates-io] (it works across the whole tree, so no need to patch deps-of-deps recursively). To do it at a distro scale, you can globally configure Cargo to use a local copy of the index instead of crates-io, and replace deps in your copy (it’s just a git repo with JSON, easy to modify).

                        Binaries are built from lockfiles, so you can read them to know exactly what to rebuild. There are tools like cargo-audit that already deal with this.

                        1. 1

                          So there are then 30 patches to modify cargo. You would also need a secondary package repository to provide the patched cargo package? Does cargo build everything from source or would it require the patched packages to be built?

                          1. 3

                            You can tell Cargo to get code from a local directory. Distros already replace crates-io with their own packages, so they already have it all set up.

                            Distros have managed to tame dozens of weird C build systems. Cargo is quite simple in comparison.

                            1. 2

                              Well, no. Most doesn’t. Just the two largest ones because of distro policies. But if you look at the raw numbers most distribution bundle everything vendored and does not provide this at all. I’m not even sure if Ubuntu follows the Debian guidelines?

                              This is why I bring it up to begin with.

                        2. 2

                          I get it. Distro’s are downstream of everything. They get the sewage and the refuse and whatever else that results from trying to get hundreds if not thousands of different applications and libraries to work together and keep them patched appropriately. But that’s the job. If you don’t want that job then don’t work on a distro.

                          In your particular example I would feel free to blacklist a project that doesn’t want to patch and test their code when it has a CVE. If the code is absolutely necessary and blacklisting it isn’t an option then patch it locally and move on. This isn’t substantially different from a terrible maintainers of a C application. Distributions have been carrying patches forward for libraries and applications for as long as I’ve been using distributions and longer.

                          1. 7

                            Back in the C days, it was considered basic etiquette to make sure your Makefile worked with DESTDIR properly.

                            What happens now is simply Parkinson’s law in it’s finest. Flatpak, Snap and Docker included.

                            It puzzles me that nobody is worried about our inability to maintain a coherent culture given the influx of programmers, but then again… Eternal September, right?

                            Must be weird for old geezers to live through that the second time. I am far too young for that so I can’t tell.

                            We need to negotiate a packaging standard that would not suck for most and then push it hard so that it gets adopted far and wide from Rust to ES. Funny times.

                            I’m especially curious whether it can be done without effectively merging all distros into one. But hey, project maintainers seem to know the best how is their code supposed to be built and configured. Maybe it’s time.

                            1. 3

                              I am one of the old geezers and I’m fully on board with the new way of doing things. Let the software maintainers bear the burden of patching and securing their software when CVEs happen. In a way this could reduce the burden on distro packagers. Download or build the latest binaries and distribute that. If upstream won’t patch their stuff then warn the user and refer them to upstream. Yes this means packagers have to build more stuff. But we unlock a whole lot of benefits in the process. Less issues with shared libraries. More reliable software with these new tools.

                            2. 4

                              This isn’t substantially different from a terrible maintainers of a C application. Distributions have been carrying patches forward for libraries and applications for as long as I’ve been using distributions and longer.

                              Previously we just needed to patch the shared library and move on. Now we suddenly need to care about what a number of upstreams vendor with complete disregard for what that implies.

                              The comments reads as extremely unsympathetic to distributions. But why? This is a model Rust/Go/npm picked. This wasn’t decided by us, and you still need to deal with the issue regardless if there is a distribution involved or not. We are told “take this and deal with it”. Upstreams are not the one going to deal with user inquiries why XYZ isn’t fixed and what we are going to do about it. We are understaffed and given more problems to deal with.

                              If you don’t want us to package the “next generation of software” say so… but users are going to disagree.

                              1. 1

                                I acknowledge the fact that you have to run cargo build more times than before. But that is the price you pay for packinging in a distro. If your users want the rust application then package it for them. Rust isn’t going to adapt for a whole host of really good reasons. And I as both a developer and someone who deploys some of these in production get a lot of benefit out of those reasons and as the primary user of the language would resist any such change.

                                For the security issues if upstream won’t patch then remove them from the security policy and tell the user they need to take CVEs up with maintainer of the software. This isn’t hard and complaining about it gives no value to any end-user.

                                1. 2

                                  Are we going to claim everything Rust touches to be unsupported? Seriously?

                                  I don’t think Rust is the C replacement the community thinks it is.

                                  1. 1

                                    If the distro needs something that Rust touches then they need to build the tooling to be able to package it. It’s more expensive to build it all but if that’s what you need to do then do that.

                              2. 3

                                But that’s the job. If you don’t want that job then don’t work on a distro.

                                Note that Debian security team found the job so onerous that they decided to remove security support for all Go packages in Debian. See https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#golang-static-linking.

                                1. 1

                                  This is a perfectly valid approach. If you can’t make your tooling do the job with that software then drop support for it.

                                  1. 2

                                    This works for Go, but does not work for Rust, because GNOME C libraries are starting to depend on Rust and distributions don’t want to drop support for GNOME.

                                    1. 1

                                      Then I guess in the case of GNOME it is by definition not too onerous. It is not the case that you can’t package and ship updates for the software. It’s just harder and more complicated. But that’s why we write tooling. I don’t get to complain when my business wants a feature that requires me to write complicated hard code. I just role up my sleeves and start designing/typing. This is no different.

                        3. 5

                          Wow this whole thread happened without any shilling for nix and guix.

                          Guix!

                          I am not really that familiar with the problems fedora and debian have with rust but afaict they’re solved by next-gen package managers so no need to worry about them. ( <— me being wrong on the internet, hoping to learn something)

                          1. 4

                            Distributions traditionally had the role of keeping programs secure. Monitor CVEs and patch programs as fast as possible. And this requires having good visibility of all the dependencies.

                            One of the core issues with that approach is that SAT-solving all the dependencies don’t scale on the system level. It was only possible before because the dependency graph was smaller. C programs don’t have good dependency resolution and rely on ABI compatibility and packagers doing the tree shaking.

                            Now that programs tend to bundle more and more dependencies it just means that this responsibility also gets shifted to the developers. And the platforms that host the source code. Instead of having distributions keeping the programs up to date, we have GitHub dependabot notifying the project of security updates.

                            It’s not necessarily a problem, it means that times are changing. Distributions have to adapt to this.

                            1. 6

                              I am in strong agreement with one of the theses of this post, which is that “application distribution” is the thing that matters to end-users. I’ve long thought myself that distribution packaging focuses too much on packaging of the composite parts instead of whole applications.

                              I believe I understand many of the [historical] reasons for why this is. But I can’t shake the feeling that many of the policies/decisions are rooted in the state of technology from 20+ years ago and if we were starting from first principles today we would devise a different set of policies. We’ve even seen this experiment play out with technologies like Docker, snap, and flatpack (as the author notes in the post), so there is some precedent to evaluate and understand the viability of alternative distribution mechanisms/policies. The thing that hasn’t yet changed is the mindset of distribution packagers, which is still focused on the composing parts instead of higher-level applications.

                              On one hand, I admire their commitment to the established distribution packaging policies because towing that line is difficult. On the other, I believe that other technologies have clearly demonstrated end-user value for application distribution and the world would be a better, more standardized place if distribution packaging updated its policies and allowed us to converge on fewer, more officially supported solutions. Of course, this is an insanely complex problem space with many 2nd and 3rd order effects to any policy change. So it is understandable why things have remained why they have for so long.

                              1. 2

                                many of the policies/decisions are rooted in the state of technology from 20+ years ago

                                Maybe that’s just indicative how little has changed. If you look at ecosystems like npm, gems, cargo, docker, flatpack and snap, it feels like they regressed more than 20 years, instead of bringing anything new to the table.

                                It feels like kids reinventing the very old ways, completely unaware how bad it was and that the current state is the result of a lot of effort into migrating way from it.

                              2. 3

                                I wonder if a middle ground is possible? Can a distro maintain an alternative cargo registry, subset of crates.io, which is used to resolve dependencies in distro-packaged rust binaries? I do think there’s some value in having all apps to be build against the same set of crate versions with less duplication, which hopefully gets at least cursory review by distro maintainers.

                                1. 4

                                  Debian wants to be able to rebuild itself from its package repository. I think an alternative Cargo registry would be workable if only Rust binaries would use it, but it doesn’t work that way, because GNOME includes GNOME shell and GNOME shell uses SpiderMonkey and SpiderMonkey uses Rust. To be able to rebuild GNOME from the repository, the repository needs to include Rust.

                                  1. 1

                                    I am not sure I entirely understand (or maybe I am just being unclear). In my mind, that alternative registry is a part of Debian package repository. That is, it includes both the meta info in crates.io format (“which versions of crates are available”) and the actual source code of crates in the form of .crate archives.

                                    1. 1

                                      Eh… in that case, that’s what Debian is already doing…

                                      1. 4

                                        It seems that not exactly? It looks like they are trying to package each individual library as a separate package, or even several packages (according to this doc), with explicitly declared dependencies. This’ll run into impedance mismatch between how Rust dependencies actually work, and whatever universal dependency resolution algorithm Debian’s package manager is using.

                                        The middle ground I have in mind is packaging a whole bunch of crates as a single thing. I believe Facebook did this (maybe still so)? There was a giant Cargo.toml/Cargo.lock, which listed all crates.io packages Facebook depended on. All these packages were vendored. Internal Facebook crates than were allowed to depend only on the packages from this set.

                                        That way, there’s no re-invention of dependency specification mechanism, but there’s still a coherent set of libraries you can audit and upgrade.

                                2. 2

                                  I appreciate most of the arguments, but the counter-point around security is missing the spot. For distributions, it is far easier to apply a patch on a single package. Rebuilding or not is not really the difficulty. Now, if many applications are bundling/pinning specific versions, distributions need to patch each version. Some of these versions may be very old and the patch may be more difficult to apply. This is a lot of work. Distributions cannot just bump the dependency as it goes against the stability promise and introduce bugs and changes. Distributions have to support what they ship for around 5 years (because many users use distributions for this exact purpose) while developers usually like to support things for a few months.

                                  Unfortunately both sides do not want to move an inch. When packaging for Debian, I would appreciate being able to bundle dependencies instead of packaging each single dependency, but there must be some ways to guarantee we are not just multiplying the amount of work we need to provide in the future. However, this is not new. Even with C, many devs do not like distributions freezing their software for 5 years.

                                  1. 11

                                    The real “issue” from the distro perspective is that they’re now trying to package ecosystems that work completely differently than the stuff they’re used to packaging, and specifically ecosystems where the build process is tied tightly to the language’s own tooling, rather than the distro’s tooling.

                                    This is why people keep talking about distros being stuck on twenty-years-ago’s way of building software. Or, really, stuck on C’s way of building software. C doesn’t come with a compiler, or a build configuration tool, or a standard way to specify dependencies and make sure they’re present and available either during build or at runtime. C is more or less just a spec for what the code ought to do when it’s run. So distros, and everybody else doing development in C, have come up with their own implementations for all of that, and grown used to that way of doing things.

                                    More recently-developed languages, though, treat a compiler and build tool and dependencies/packaging as a basic requirement, and tightly integrate to their standard tooling. Which then means that the distro’s existing and allegedly language-agnostic tooling doesn’t work, or at least doesn’t work as well, and may not have been as language-agnostic as they hoped.

                                    Which is why so many of the arguments in these threads have been red herrings. It’s not that “what dependencies does this have” is some mysterious unanswerable question in Rust, it’s that the answer to the question is available in a toolchain that isn’t the one the distro wants to use. It’s not that “rebuild the stuff that had the vulnerable dependency” is some nightmare of tracking down impossible-to-know information and hoping you caught and patched everything, it’s that it’s meant to be done using a toolchain and a build approach that isn’t the one the distro wants to use.

                                    And there’s not really a distro-friendly thing the upstream developers can do, because each distro has its own separate preferred way of doing this stuff, so that’s basically pushing the combinatorial nightmare upstream and saying “take the information you already provide in your language’s standard toolchain, and also provide and maintain one additional copy of it for each distro, in that distro’s preferred format”. The only solution is for the distros to evolve their tooling to be able to handle these languages, because the build approach used in Rust, Go, etc. isn’t going away anytime soon, and in fact is likely to become more popular over time.

                                    1. 5

                                      The only solution is for the distros to evolve their tooling to be able to handle these languages

                                      The nixpkgs community has been doing this a lot. Their response to the existence of other build tools has been to write things like bundix, cabal2nix and cargo2nix. IIRC people (used to) use cabal2nix to make the whole of hackage usable in nixpkgs?

                                      From the outside it looks like the nix community’s culture emphasizes a strategy of enforcing policy by making automations whose outputs follow it.

                                      1. 4

                                        Or, really, stuck on C’s way of building software.

                                        I think it’s at least slightly more nuanced than that. Most Linux distributions, in particular, have been handling Perl modules since their earliest days. Debian/Ubuntu use them fairly extensively even in base system software. Perl has its own language ecosystem for building modules, distributing them in CPAN, etc., yet distros have generally been able to bundle Perl modules and their dependencies into their own package system. End users are of course free to use Perl’s own CPAN tooling, but if you apt-get install something on Debian that uses Perl, it doesn’t go that route, and instead pulls in various libxxx-perl packages. I don’t know enough of the details to know why Rust is proving more intractable than Perl though.

                                        1. 6

                                          I don’t know enough of the details to know why Rust is proving more intractable than Perl though

                                          There is a big difference between C, Perl, Python on the one side and Rust on the other.

                                          The former have a concept of “search path”: there’s a global namespace where all libraries live. That’s include path for C, PYTHONPATH for Python and @INC (?) for Perl. To install a library, you put it into some blessed directory on the file system, and it becomes globally available. The corollary here is that every one is using the same version of library. If you try to install two different versions, you’ll get a name conflict.

                                          Rust doesn’t have global search path / global namespace. “Installing rust library” is not a thing. Instead, when you build a piece of Rust software, you need to explicitly specify path for every dependency. Naturally, doing this “by hand” is hard, so the build system (Cargo) has a lot of machinery for wiring a set of interdependent crates together.

                                          1. 2

                                            there’s a global namespace where all libraries live

                                            Yes, this is one of the biggest differences. Python, Perl, etc. come out of the Unix-y C-based tradition of not having a concept of an “application” you run or a “project” you work on, but instead only of a library search path that’s assumed to be shared by all programs in that language, or at best per-user unique so that one user’s set of libraries doesn’t pollute everyone else’s.

                                            Python has trended away from this and toward isolating each application/project – that’s the point of virtual environments – but does so by just creating a per-virtualenv search path.

                                            More recently-developed languages like Rust have avoided ever using the shared-search-path approach in the first place, and instead isolate everything by default, with its own project-local copies of all dependencies.

                                            (the amount of code generation/specialization that happens at compile time in Rust for things like generics is a separate issue, but one that distros – with their ability to already handle C++ – should in theory not have trouble with)

                                        2. 4

                                          This is why people keep talking about distros being stuck on twenty-years-ago’s way of building software. Or, really, stuck on C’s way of building software. C doesn’t come with a compiler, or a build configuration tool, or a standard way to specify dependencies and make sure they’re present and available either during build or at runtime. C is more or less just a spec for what the code ought to do when it’s run. So distros, and everybody else doing development in C, have come up with their own implementations for all of that, and grown used to that way of doing things.

                                          More than that, I’d say they’re language-specific package managers around autotools+C.