Semver allows me to break compatibility in major releases. The author recommends not to, and to support old interfaces for years. The latter is out of the question for hobby projects. The package manager of the language I use (e.g. Cargo, npm) is able to deal with this just fine. Apt is not.
Cargo or npm allow me to lock all versions in my dependency tree and to restrict version ranges. Especially the latter is necessary for API breakages (in compliance with semver) to not be a problem. Again, it’s only apt that is not able to deal with this. Cargo or npm can resolve this conflict automatically and find appropriate versions, or do version splits if necessary. In Debian packaging, version splits are a manual task done by a person, which, to me, is the actual friction here.
As an application developer I don’t see why I should need to work around the deficiencies of distribution’s package management systems, especially when those deficiencies are admitted by the author. I am not convinced that I, as developer, should bother with Debian. All of the arguments in “why Debian?” are from the perspective of an end user.
The existence of Debian “stable” is actually a reason why I avoid Debian as a developer. Because what Debian considers “stable” comes at a price I don’t want to pay.
In that sense I reasonate with the post by Joey Hess much more than with this one. Because I really think it’s the distros who have a problem here. Not me. I can just statically link everything, put the resulting binary in a PPA and never need to bother with any of this.
Everything made a lot more sense to me when I started to think about the fact that the npm model assumes that code is being deployed by a team of full-time developers who are paid to stay on the upgrade treadmill and work thru all the integration issues of pulling in different pieces that have never been tested together. In this context, you can’t afford to wait for a stable release of a whole distro; you’ve got the bandwidth and expertise and test infrastructure to handle making it work with just the pieces you know you need. But forcing the end-user to be responsible for that kind of integration would be a nightmare.
The package manager of the language I use (e.g. Cargo, npm) is able to deal with this just fine. Apt is not.
Apt is perfectly able to cope with complex versioned dependencies, including “not compatible with libs < this version” and “this random point release actually changed the API so all the dependencies need to care about it, even though the developer claims otherwise”.
Version range restrictions (particularly upper bounds) are the default in Cargo and npm, while in apt they are only used if actually necessary. They’re not necessary if no breakage happens. That is the friction in apt for actually using semver to its full extent.
It’s more of a policy or best practice question than a technical one, but it doesn’t matter.
A major reason I use Debian is that, as a user, I consider 90% of software lifecycles to be utterly insane and actively hostile to me, and Debian forces them into some semblance of a reasonable, manageable, release pattern (namely, Debian’s). If I get the option to choose between upstream and a Debian package, I will take the latter every single time, because it immediately has a bunch of policy guarantees that make it friendlier to me as a user. And if I don’t get the option, I will avoid the software if I possibly can.
(Firefox is the only major exception, and its excessively fast release cadence and short support windows are by far my biggest issue with it as a piece of software.)
I never really understood why short release cycles is a problem for people, but then I don’t use Debian because of their too long ones. For example, the majority of Firefox’s releases don’t contain user-visible changes.
Could you elaborate what your problems with Firefox on Debian are? Or why software lifecycles can even be hostile to you?
I’m with you. I update my personal devices ~weekly via a rolling release model (going on 10 years now), and I virtually never run into problems. The policies employed by Debian stable provide literally no advantage to me because of that. Maybe the calculus changes in a production environment with more machines to manage, but as far as personal devices go, Debian stable’s policies would lead to a net drain on my time because I’d be constantly pushing against the grain to figure out how to update my software to the latest version provided by upstream.
I’ve had quite a few problems myself, mostly around language-specific package managers that break something under me. This is probably partly my fault because I have a lot of one-off scripts with unversioned dependencies, but at least in the languages I use most (Python, Perl, R, shell, etc.), those kinds of unversioned dependencies seem to be the norm. Most recent example: an update to R on my Mac somehow broke some of my data-visualization scripts while I was working on a paper (seemingly due to a change in ggplot, which was managed through R’s own package manager). Not very convenient timing.
For a desktop I mostly put up with that anyway, but for a server I prefer Debian stable because I can leave it unattended with auto-updates on, not having to worry that something is going to break. For example I have some old Perl CGI stuff lying around, and have been happy that if I manage dependencies via Debian stable’s libdevel-xxx-perl packages instead of CPAN, I can auto-update and pull in security updates without my scripts breaking. I also like major Postfix upgrades (which sometimes require manual intervention) to be scheduled rather than rolling.
Yeah I don’t deal with R myself, but based on what my wife tells me (she works with R a lot), I’m not at all surprised that it would be a headache to deal with!
Every time a major update happens to a piece of software, I need to spend a bunch of time figuring out and adapting to the changes. As a user, my goal is to use software, rather than learn how to use it, so that time is almost invariably wasted. If I can minimize the frequency, and ideally do all my major updates at the same time, that at least constrains the pain.
My problem with Firefox on Debian is that due to sheer code volume and complexity, third-party security support is impossible; its upstream release and support windows are incompatible with Debian’s; and it’s too important to be dropped from the distro. Due to all that, it has an exception to the release lifecycle, and every now and then with little warning it will go through a major update, breaking everything and wasting a whole bunch of my time.
Due to all that, it has an exception to the release lifecycle, and every now and then with little warning it will go through a major update, breaking everything and wasting a whole bunch of my time.
I had this happen with Chromium; they replaced the renderer in upstream, and a security flaw was found which couldn’t be backported due to how insanely complicated the codebase is and the fact that Chromium doesn’t have a proper stable branch, so one day I woke up and suddenly I couldn’t run Chromium over X forwarding any more, which was literally the only thing I was using it for.
Because you need to invest into upgrading too much of your time. I maintain 4 personal devices with Fedora and I almost manage to upgrade yearly. I am very happy for RHEL at work. 150 servers would be insane. Even with automation. Just the investment into decent ops is years.
For me there is an equivalence between Debian stable releases and Ubuntu LTE ones, they both run at around 2 years.
But the advantage (in my eyes) that Debian has is the rolling update process for the “testing” distribution, which gets a good balance between stability and movement.
We are currently switching our servers from Ubuntu LTE to Debian stable. Driven mostly by lack of confidence in the future trajectory of Ubuntu.
I myself has noticed that Debian packaging is really hard to get right as a newcomer.
It’d be great if their was both simpler tooling a more concise guides on getting started. A PDF with 50+ pages/slides of information isn’t a good quick start format.
A coworker once spent a couple of months trying to get a package upsteamed to Debian. By far the hardest part was waiting for the upstream maintainers to comment on the package. It took weeks to get any kind of feedback at all. By that time my teammate had completely forgotten all the packaging details but fixed the comments anyway. After not hearing back about the fixes for another couple weeks he just gave up. Someone else ended up finishing the packaging.
A few of these are mentioned in the article under the “Things for Debian to improve” section. I think improvements here could help reduce friction at the margins, but not sure they address the more fundamental disconnect. For example tooling/docs improvements would make it easier for the subset of upstream developers who value a Debian-style packaging system but currently don’t bother with it due to annoyances on that front, which is definitely not nothing. But it wouldn’t fix the problem with regard to upstream developers whose release cycle just doesn’t fit the Debian packaging model at all.
I wish more folks involved in packaging for Linux distros were familiar with Homebrew. Obviously not everything Homebrew does is applicable to Debian, but the ability for folks to show up and easily contribute new versions with a simple PR is game changing. Last night I noticed that the python-paramiko package in Debian is severely out of date, but the thought of trying to learn the various intricacies of contributing to Debian well enough to update it is turns me right off.
As an upstream dev of code that’s packaged with Homebrew, I have noticed that Homebrew is by far the sloppiest of any packagers; there is basically no QA, and often the packagers don’t even read the instructions I’ve provided for them. I’ve never tried it myself, but it’s caused me a lot of headaches all the same.
Unfortunately, Debian has still a strong ownership model. Unless a package is team-maintained, an unwilling maintainer can stall any effort to update a package, sometimes actively, sometimes passively. In the particular case of Paramiko, the maintainer has very strong opinions on this matter (I know that first hand).
How does this setup even work in case of a security vulnerability?
Bugs tagged as security problems (esp. if also tagged with a CVE) get extra attention from the security team. How that plays out depends on the package/bug, but it can range from someone from the security team prodding the maintainer, all the way to directly uploading a fix themselves (as a non-maintainer upload).
But yeah in general most Debian packages have 1-2 maintainers, which can be a bottleneck if the maintainer loses interest or gets busy. For packages with a lot of interest, such a maintainer will end up replaced by someone else. For more obscure packages it might just languish unmaintained until someone removes the package from Debian for having unfixed major issues.
Great article. Solid overview of the pros and cons of the quite different approaches of the Debian-packaging and npm-style worldviews. Lots has been discussed elsewhere, but I think this pulls together a measured overview that tries to seriously understand why each of them is the way it is, what could be improved in each without undermining their core benefits, and what are more fundamental difficulties in getting the two approaches to work together well.
There is also a commentary on this from Joey Hess, one time Debian release manager (I think)
See http://joeyh.name/blog/entry/futures_of_distributions/
As an application developer I don’t see why I should need to work around the deficiencies of distribution’s package management systems, especially when those deficiencies are admitted by the author. I am not convinced that I, as developer, should bother with Debian. All of the arguments in “why Debian?” are from the perspective of an end user.
The existence of Debian “stable” is actually a reason why I avoid Debian as a developer. Because what Debian considers “stable” comes at a price I don’t want to pay.
In that sense I reasonate with the post by Joey Hess much more than with this one. Because I really think it’s the distros who have a problem here. Not me. I can just statically link everything, put the resulting binary in a PPA and never need to bother with any of this.
Everything made a lot more sense to me when I started to think about the fact that the npm model assumes that code is being deployed by a team of full-time developers who are paid to stay on the upgrade treadmill and work thru all the integration issues of pulling in different pieces that have never been tested together. In this context, you can’t afford to wait for a stable release of a whole distro; you’ve got the bandwidth and expertise and test infrastructure to handle making it work with just the pieces you know you need. But forcing the end-user to be responsible for that kind of integration would be a nightmare.
The npm model works because it’s being deployed on top of a stable Debian system.
I’m late to the party, but can you elaborate on this?
The package manager of the language I use (e.g. Cargo, npm) is able to deal with this just fine. Apt is not.
Apt is perfectly able to cope with complex versioned dependencies, including “not compatible with libs < this version” and “this random point release actually changed the API so all the dependencies need to care about it, even though the developer claims otherwise”.
Exactly what feature do you think apt is missing?
Version range restrictions (particularly upper bounds) are the default in Cargo and npm, while in apt they are only used if actually necessary. They’re not necessary if no breakage happens. That is the friction in apt for actually using semver to its full extent.
It’s more of a policy or best practice question than a technical one, but it doesn’t matter.
A major reason I use Debian is that, as a user, I consider 90% of software lifecycles to be utterly insane and actively hostile to me, and Debian forces them into some semblance of a reasonable, manageable, release pattern (namely, Debian’s). If I get the option to choose between upstream and a Debian package, I will take the latter every single time, because it immediately has a bunch of policy guarantees that make it friendlier to me as a user. And if I don’t get the option, I will avoid the software if I possibly can.
(Firefox is the only major exception, and its excessively fast release cadence and short support windows are by far my biggest issue with it as a piece of software.)
I never really understood why short release cycles is a problem for people, but then I don’t use Debian because of their too long ones. For example, the majority of Firefox’s releases don’t contain user-visible changes.
Could you elaborate what your problems with Firefox on Debian are? Or why software lifecycles can even be hostile to you?
I’m with you. I update my personal devices ~weekly via a rolling release model (going on 10 years now), and I virtually never run into problems. The policies employed by Debian stable provide literally no advantage to me because of that. Maybe the calculus changes in a production environment with more machines to manage, but as far as personal devices go, Debian stable’s policies would lead to a net drain on my time because I’d be constantly pushing against the grain to figure out how to update my software to the latest version provided by upstream.
I’ve had quite a few problems myself, mostly around language-specific package managers that break something under me. This is probably partly my fault because I have a lot of one-off scripts with unversioned dependencies, but at least in the languages I use most (Python, Perl, R, shell, etc.), those kinds of unversioned dependencies seem to be the norm. Most recent example: an update to R on my Mac somehow broke some of my data-visualization scripts while I was working on a paper (seemingly due to a change in ggplot, which was managed through R’s own package manager). Not very convenient timing.
For a desktop I mostly put up with that anyway, but for a server I prefer Debian stable because I can leave it unattended with auto-updates on, not having to worry that something is going to break. For example I have some old Perl CGI stuff lying around, and have been happy that if I manage dependencies via Debian stable’s libdevel-xxx-perl packages instead of CPAN, I can auto-update and pull in security updates without my scripts breaking. I also like major Postfix upgrades (which sometimes require manual intervention) to be scheduled rather than rolling.
Yeah I don’t deal with R myself, but based on what my wife tells me (she works with R a lot), I’m not at all surprised that it would be a headache to deal with!
Every time a major update happens to a piece of software, I need to spend a bunch of time figuring out and adapting to the changes. As a user, my goal is to use software, rather than learn how to use it, so that time is almost invariably wasted. If I can minimize the frequency, and ideally do all my major updates at the same time, that at least constrains the pain.
I’ve ranted about this in a more restricted context before.
My problem with Firefox on Debian is that due to sheer code volume and complexity, third-party security support is impossible; its upstream release and support windows are incompatible with Debian’s; and it’s too important to be dropped from the distro. Due to all that, it has an exception to the release lifecycle, and every now and then with little warning it will go through a major update, breaking everything and wasting a whole bunch of my time.
I had this happen with Chromium; they replaced the renderer in upstream, and a security flaw was found which couldn’t be backported due to how insanely complicated the codebase is and the fact that Chromium doesn’t have a proper stable branch, so one day I woke up and suddenly I couldn’t run Chromium over X forwarding any more, which was literally the only thing I was using it for.
Ha, now I understand why I use emacs. It hasn’t changed the UX in years, if not decades.
Because you need to invest into upgrading too much of your time. I maintain 4 personal devices with Fedora and I almost manage to upgrade yearly. I am very happy for RHEL at work. 150 servers would be insane. Even with automation. Just the investment into decent ops is years.
For me there is an equivalence between Debian stable releases and Ubuntu LTE ones, they both run at around 2 years.
But the advantage (in my eyes) that Debian has is the rolling update process for the “testing” distribution, which gets a good balance between stability and movement.
We are currently switching our servers from Ubuntu LTE to Debian stable. Driven mostly by lack of confidence in the future trajectory of Ubuntu.
I myself has noticed that Debian packaging is really hard to get right as a newcomer.
It’d be great if their was both simpler tooling a more concise guides on getting started. A PDF with 50+ pages/slides of information isn’t a good quick start format.
A coworker once spent a couple of months trying to get a package upsteamed to Debian. By far the hardest part was waiting for the upstream maintainers to comment on the package. It took weeks to get any kind of feedback at all. By that time my teammate had completely forgotten all the packaging details but fixed the comments anyway. After not hearing back about the fixes for another couple weeks he just gave up. Someone else ended up finishing the packaging.
A few of these are mentioned in the article under the “Things for Debian to improve” section. I think improvements here could help reduce friction at the margins, but not sure they address the more fundamental disconnect. For example tooling/docs improvements would make it easier for the subset of upstream developers who value a Debian-style packaging system but currently don’t bother with it due to annoyances on that front, which is definitely not nothing. But it wouldn’t fix the problem with regard to upstream developers whose release cycle just doesn’t fit the Debian packaging model at all.
I wish more folks involved in packaging for Linux distros were familiar with Homebrew. Obviously not everything Homebrew does is applicable to Debian, but the ability for folks to show up and easily contribute new versions with a simple PR is game changing. Last night I noticed that the python-paramiko package in Debian is severely out of date, but the thought of trying to learn the various intricacies of contributing to Debian well enough to update it is turns me right off.
As an upstream dev of code that’s packaged with Homebrew, I have noticed that Homebrew is by far the sloppiest of any packagers; there is basically no QA, and often the packagers don’t even read the instructions I’ve provided for them. I’ve never tried it myself, but it’s caused me a lot of headaches all the same.
I just looked at the packaging information for paramiko and I have more questions than before:
How does this setup even work in case of a security vulnerability?
Unfortunately, Debian has still a strong ownership model. Unless a package is team-maintained, an unwilling maintainer can stall any effort to update a package, sometimes actively, sometimes passively. In the particular case of Paramiko, the maintainer has very strong opinions on this matter (I know that first hand).
Strong opinions are not necessarily bad. Does he believe paramiko should not be updated?
Bugs tagged as security problems (esp. if also tagged with a CVE) get extra attention from the security team. How that plays out depends on the package/bug, but it can range from someone from the security team prodding the maintainer, all the way to directly uploading a fix themselves (as a non-maintainer upload).
But yeah in general most Debian packages have 1-2 maintainers, which can be a bottleneck if the maintainer loses interest or gets busy. For packages with a lot of interest, such a maintainer will end up replaced by someone else. For more obscure packages it might just languish unmaintained until someone removes the package from Debian for having unfixed major issues.
Great article. Solid overview of the pros and cons of the quite different approaches of the Debian-packaging and npm-style worldviews. Lots has been discussed elsewhere, but I think this pulls together a measured overview that tries to seriously understand why each of them is the way it is, what could be improved in each without undermining their core benefits, and what are more fundamental difficulties in getting the two approaches to work together well.