I’d like to attach my personal experience of attempting to contribute to Debian here because it seems somewhat relevant to the author’s assertion that “Debian has long been experiencing a decline in the amount of people willing to participate in the project.”
I’ve tried to contribute on three different occasions; two were plain old bug reports, one was a patch. A common thread for each of these contributions was a demoralizing absence of feedback. One of the two bug reports has remained unanswered for over a year now, and my patch submission for about nine months.
Now, I sent the patch I wanted to contribute directly to the relevant maintainer mailing list. I wasn’t sure whether this was the correct way of submitting a patch for a downstream package, so I apologized preemptively in case this was not proper form and asked cordially to be pointed in the right direction if this wasn’t it.
No “thanks for your contribution,” no requests for additional information, no human response whatsoever.
What I did get was spam. Some of it was addressed to the alias I’d used for my bug report, and some of it came straight through the nginx package maintainer list I’d eventually unsubscribe from after giving up hope of receiving feedback and getting sick of receiving spam every couple of days.
Credit where it’s due, my other bug report was responded to and fixed, though also after a several months long period of radio silence.
I have no idea what to take away from this experience other than the impression that contributions are generally unwelcome, and that I should probably not bother trying to contribute in the future. Maybe this experience is not entirely uncommon, or maybe it’s just me. I really don’t know.
I have no idea what to take away from this experience other than the impression that contributions are generally unwelcome, and that I should probably not bother trying to contribute in the future.
What I would take away from this (and from my similar experiences) is that Debian maintainers are overworked volunteers with too little time and we need more of them. Of course, getting to be one involves somehow getting time from existing ones, so it’s a vicious cycle.
I was a Debian developer in the late 90s (I think from 1999-2003?) and the original application process was quite quick. I later went into hiatus for a period and tried to reactivate my access after about ten years away but the whole process was so onerous that I’ve just not taken it further. A pity, as I’d love to contribute again.
Yeah the spam is definitely real. I get at least five messages a day (might need to adjust my filter as well, but oh well). I’m happy I used a semi-throwaway address for that list - I can always delete the address and be rid of that spam forever.
I wanted to become a Debian Developer around 2003-2005 and back then you could expect the whole process to take about a year. Spoiler alert: I never even officially started the process.
From what I heard this was improved immensely but I never followed up and instead have been contributing to other projects. But personally Github has actually been a huge boon here, the friction is just so much less, I’ve also had problems contributing to other older projects before, at one time an important patch of mine hung for so long that someone else with commit rights fixed it before the turnaround time…
On the other hand I don’t think this can be solved, but the more open the project is (and yes, accepting PRs on Github or public other forge) usually makes it a 10x better experience…
I think there’s not enough data attached to this rant. I don’t disagree that the maintainers are spread very thin. But as presented, it doesn’t look overwhelming - 60-odd packages per person for LTS is not a huge amount. Quite a few of them will have no issues at all. Some may not even publish new versions between releases. There will be some that need a lot of attention of course (browsers, SSL libraries, popular daemons, etc).
But if I’m reading this right, author seems to also expect the maintainers to be actually auditing the package sources. This may happen to some extent to help the global effort, but their main security responsibility is to apply/backport/create patches based on reports, not to actively develop third party software. I’m not sure where that expectation came from.
The “This is further made worse by the fact that the current leadership is more worried about…” bit is also not supported by anything. Even if you assumed the slides represent the priorities, the order is opposite to what the blog post claims.
There are issues with available manpower in distributions, but I found this post a bit misinformed/unfair.
It’s always seemed a bit insane to me that you needed to have this additional layer of manpower for maintaining packages in addition to the manpower required to just keep many of these open source projects afloat.
Maintaining packages for a single distribution seems like an enormous overhead, and it seems to me that we are seeing the consequences of this. I would be really surprised if this was a problem with debian alone.
That said, I do understand why it has to be like this, but it is clearly the highest core operational cost of running a linux distribution.
Debian policies do themselves no favors in maintenance. Take go applications for instance. Every go library must be packaged and maintained separately. This means potentially dozens or hundreds of dependency packages for a go application to receive its own package.
Never mind that Debian has one of the most complex packaging standards of all the distributions. Adding to that complex standard there are probably a dozen different toolsets to meet the standard.
I think when Linux was much smaller than it is now this was seen as feasible. You “only” needed to maintain the relationship between kernel, libc, and some utilities (that were handled by GNU, mostly). Plus it was a hobby OS mostly run on desktops.
Fast-forward a few years and your Linux server is a mission-critical, high-value target, but you still rely on the same volunteer organization to ensure it’s up-to-date.
Yeah, it’s not meant to be disingenious, rather just sloppily made :)
I think the main takeaway, if you disregard 2011 specifically is that the amount of packages grows with 1000-2000 ever release, so it’s not like the growth is in any way explosive, but rather it is possible for the debian team to anticipate the ballpark level of packages in each subsequent release.
Yup. I think it’s fine. I only picked 2011 at random because I don’t actually remember when I transitioned my stuff to amd64. 10-12y sounds about right…
Also, a lot of software was probably a lot simpler back then and development perhaps even moved at a slower pace because open source was such a niche thing (like you said, hobby OS), so it was both more doable to keep packages up to date and also less of an issue if packages lagged behind.
Coming from a BSD perspective, I’ve always struggled with the lack of any kind of distinction in systems like Debian between a base system and third-party things. On FreeBSD, there is a base system that has long-term ABI stability guarantees. You can be very sure that software (including kernel modules) built on FreeBSD 10.0 will still work on FreeBSD 10.x, as long as its only dependencies are in the base system. There is also third-party software, which is on a rolling-release schedule. The project does not accept things into the base system unless they either have strong stability guarantees upstream, or the project has the expertise to do its own security back ports. In contrast, if there’s a security vulnerability in a port, then it’s just bumped to the latest version that includes the fix and the next package set will include it (and all of the things that depend on it will be rebuilt as part of this package-build cycle).
FreeBSD has recently also moved to not packaging libraries for languages with their own package managers. For example, programs written in Haskell are still packaged, but they expect to get their dependencies via Cabal or similar. The package builds still run offline and so the ports infrastructure will cache the files that Cabal wants and will pre-seed the build directory with the downloads, so you still have a stable version. The same is true for Rust, Go, and so on. I don’t think it is for Python / Ruby yet, but maybe things like pipenv will make that the case as well. This avoids the need to create 30 ports for Haskell libraries (which no one will ever use because they’ll build their stuff using Cabal), just so that one port that actually uses them all can exist.
The problem with the FreeBSD model is that most software that you want to write depends on things beyond the base system. If you ship something that depends on, say, QtWebKit, then you need to either depend on the one in ports (currently the Qt 5 version only) and hope that it stays there and gets security updates (it looks as if this version is from 20017, so good luck!) and isn’t updated to an incompatible version, or bundle your own one and be responsible for ensuring that you are not shipping an insecure version (which, for a browser engine, typically means doing a new release every week or two). In the latter case, you might succeed if you can keep your dependencies to projects that have good security policies and do LTS releases but unfortunately that’s the minority of open source projects (proprietary libraries are far more rare in comparison but typically come with an SLA requiring security fixes in a timely fashion, or you wouldn’t be willing to pay for them at all).
“On FreeBSD, there is a base system that has long-term ABI stability guarantees. You can be very sure that software (including kernel modules) built on FreeBSD 10.0 will still work on FreeBSD 10.x, as long as its only dependencies are in the base system.”
On Debian, that’s almost exactly the same. If you install Debian 10.0, everything from the Debian repos will build and work. 10.1, 10.2, 10.3 don’t actually mean much except as markers that the baseline for installs has changed - all due to bug fixes, not new features, and with no breaking changes.
No new releases for the sake of new features happen until a major version release, at which point “stable” means 11 instead of 10. You are mostly guaranteed to be able to upgrade from 10 to 11 without a reinstall, but individual programs may have decided to change configuration formats (Apache 2.2 => 2.4) or have been dropped. The most catastrophic scenario is if Debian decided to no longer support your machine architecture, in which case there is no upgrade available.
I agree that Debian does not have a good solution to the language ecosystem problem. In order to get a reproducible build, you need to have a frozen repo of all the dependencies. The options that I can think of right now are:
package each library individually when needed (current practice)
produce superpackages of commonly used libraries and package others individually
snapshot go, perl, python,ruby, haskell… package repos and don’t do any curation, just
provide them as the known point in time. Apply non-breaking bug fixes as resources allow, but
each one triggers a rebuild of every package marked with a “go-repo” dependency.
On Debian, that’s almost exactly the same. If you install Debian 10.0, everything from the Debian repos will build and work. 10.1, 10.2, 10.3 don’t actually mean much except as markers that the baseline for installs has changed - all due to bug fixes, not new features, and with no breaking changes.
The difference is that Debian tries to make this claim for however many tens of thousands of packages exist in their repos. FreeBSD does not, it makes this claim only for things that are either maintained by the FreeBSD project directly (e.g. the kernel, libc) or maintained by an upstream that has strong ABI stability guarantees that we can use (e.g. libc++). Anything that can’t make that kind of stability guarantee is not in the base system and so there is a clear dividing line between ‘this thing is ABI stable and will get security and bug fixes for the duration of a major release’ and ‘this thing is packaged as a convenience for you but security and bug fixes are the responsibility of the upstream project’.
Debian conflates these two, which leads to the problem that the article describes. There is no clear divide between things that the Debian project has enough expertise and manpower to be able to provide long-term guarantees (including back-porting security fixes if the shipped branch is no longer supported upstream), versus things that may have security vulnerabilities over the lifetime of a Debian release but won’t be patched because upstream no longer does releases from a compatible branch and the Debian project doesn’t have any contributors who know how to back-port the fixes. Web browsers are the canonical example of a place where this is a problem: even the Chrome team gave up trying to back-port security fixes to older Chrome releases and now have a policy of ‘you’re on the latest release or you’re insecure’. No third-party packaging infrastructure is going to have the resources to back-port these fixes (the code is aggressively refactored upstream, so even determining if the vulnerable code is in the version that you ship is a difficult problem), yet Debian tries to apply the same support policy to Chromium as to glibc.
I’d like to attach my personal experience of attempting to contribute to Debian here because it seems somewhat relevant to the author’s assertion that “Debian has long been experiencing a decline in the amount of people willing to participate in the project.”
I’ve tried to contribute on three different occasions; two were plain old bug reports, one was a patch. A common thread for each of these contributions was a demoralizing absence of feedback. One of the two bug reports has remained unanswered for over a year now, and my patch submission for about nine months.
Now, I sent the patch I wanted to contribute directly to the relevant maintainer mailing list. I wasn’t sure whether this was the correct way of submitting a patch for a downstream package, so I apologized preemptively in case this was not proper form and asked cordially to be pointed in the right direction if this wasn’t it.
No “thanks for your contribution,” no requests for additional information, no human response whatsoever.
What I did get was spam. Some of it was addressed to the alias I’d used for my bug report, and some of it came straight through the nginx package maintainer list I’d eventually unsubscribe from after giving up hope of receiving feedback and getting sick of receiving spam every couple of days.
Credit where it’s due, my other bug report was responded to and fixed, though also after a several months long period of radio silence.
I have no idea what to take away from this experience other than the impression that contributions are generally unwelcome, and that I should probably not bother trying to contribute in the future. Maybe this experience is not entirely uncommon, or maybe it’s just me. I really don’t know.
What I would take away from this (and from my similar experiences) is that Debian maintainers are overworked volunteers with too little time and we need more of them. Of course, getting to be one involves somehow getting time from existing ones, so it’s a vicious cycle.
I was a Debian developer in the late 90s (I think from 1999-2003?) and the original application process was quite quick. I later went into hiatus for a period and tried to reactivate my access after about ten years away but the whole process was so onerous that I’ve just not taken it further. A pity, as I’d love to contribute again.
Yeah the spam is definitely real. I get at least five messages a day (might need to adjust my filter as well, but oh well). I’m happy I used a semi-throwaway address for that list - I can always delete the address and be rid of that spam forever.
I wanted to become a Debian Developer around 2003-2005 and back then you could expect the whole process to take about a year. Spoiler alert: I never even officially started the process.
From what I heard this was improved immensely but I never followed up and instead have been contributing to other projects. But personally Github has actually been a huge boon here, the friction is just so much less, I’ve also had problems contributing to other older projects before, at one time an important patch of mine hung for so long that someone else with commit rights fixed it before the turnaround time…
On the other hand I don’t think this can be solved, but the more open the project is (and yes, accepting PRs on Github or public other forge) usually makes it a 10x better experience…
I think there’s not enough data attached to this rant. I don’t disagree that the maintainers are spread very thin. But as presented, it doesn’t look overwhelming - 60-odd packages per person for LTS is not a huge amount. Quite a few of them will have no issues at all. Some may not even publish new versions between releases. There will be some that need a lot of attention of course (browsers, SSL libraries, popular daemons, etc).
But if I’m reading this right, author seems to also expect the maintainers to be actually auditing the package sources. This may happen to some extent to help the global effort, but their main security responsibility is to apply/backport/create patches based on reports, not to actively develop third party software. I’m not sure where that expectation came from.
The “This is further made worse by the fact that the current leadership is more worried about…” bit is also not supported by anything. Even if you assumed the slides represent the priorities, the order is opposite to what the blog post claims.
There are issues with available manpower in distributions, but I found this post a bit misinformed/unfair.
It’s always seemed a bit insane to me that you needed to have this additional layer of manpower for maintaining packages in addition to the manpower required to just keep many of these open source projects afloat.
Maintaining packages for a single distribution seems like an enormous overhead, and it seems to me that we are seeing the consequences of this. I would be really surprised if this was a problem with debian alone.
That said, I do understand why it has to be like this, but it is clearly the highest core operational cost of running a linux distribution.
Never underestimate the costs of system integration.
Debian policies do themselves no favors in maintenance. Take go applications for instance. Every go library must be packaged and maintained separately. This means potentially dozens or hundreds of dependency packages for a go application to receive its own package.
Never mind that Debian has one of the most complex packaging standards of all the distributions. Adding to that complex standard there are probably a dozen different toolsets to meet the standard.
I think when Linux was much smaller than it is now this was seen as feasible. You “only” needed to maintain the relationship between kernel, libc, and some utilities (that were handled by GNU, mostly). Plus it was a hobby OS mostly run on desktops.
Fast-forward a few years and your Linux server is a mission-critical, high-value target, but you still rely on the same volunteer organization to ensure it’s up-to-date.
Well, linux as production-critical OS isn’t really very new to be fair. OP claims to have run debian since 1998, and that’s 24 years ago (sorry…).
Just because I spend 15 minutes procrastinating with this, here’s the growth of packages within debian historically:
Fun fact: Debian provides direct access to a postgresql with a lot of statistics about everything debian related. I think that is kind of neat.
Nice graphics, but a bit disingenious because you probably should’ve used
max(amd64, i386), especially for 2011 ;)Yeah, it’s not meant to be disingenious, rather just sloppily made :)
I think the main takeaway, if you disregard 2011 specifically is that the amount of packages grows with 1000-2000 ever release, so it’s not like the growth is in any way explosive, but rather it is possible for the debian team to anticipate the ballpark level of packages in each subsequent release.
Yup. I think it’s fine. I only picked 2011 at random because I don’t actually remember when I transitioned my stuff to amd64. 10-12y sounds about right…
Also, a lot of software was probably a lot simpler back then and development perhaps even moved at a slower pace because open source was such a niche thing (like you said, hobby OS), so it was both more doable to keep packages up to date and also less of an issue if packages lagged behind.
Coming from a BSD perspective, I’ve always struggled with the lack of any kind of distinction in systems like Debian between a base system and third-party things. On FreeBSD, there is a base system that has long-term ABI stability guarantees. You can be very sure that software (including kernel modules) built on FreeBSD 10.0 will still work on FreeBSD 10.x, as long as its only dependencies are in the base system. There is also third-party software, which is on a rolling-release schedule. The project does not accept things into the base system unless they either have strong stability guarantees upstream, or the project has the expertise to do its own security back ports. In contrast, if there’s a security vulnerability in a port, then it’s just bumped to the latest version that includes the fix and the next package set will include it (and all of the things that depend on it will be rebuilt as part of this package-build cycle).
FreeBSD has recently also moved to not packaging libraries for languages with their own package managers. For example, programs written in Haskell are still packaged, but they expect to get their dependencies via Cabal or similar. The package builds still run offline and so the ports infrastructure will cache the files that Cabal wants and will pre-seed the build directory with the downloads, so you still have a stable version. The same is true for Rust, Go, and so on. I don’t think it is for Python / Ruby yet, but maybe things like pipenv will make that the case as well. This avoids the need to create 30 ports for Haskell libraries (which no one will ever use because they’ll build their stuff using Cabal), just so that one port that actually uses them all can exist.
The problem with the FreeBSD model is that most software that you want to write depends on things beyond the base system. If you ship something that depends on, say, QtWebKit, then you need to either depend on the one in ports (currently the Qt 5 version only) and hope that it stays there and gets security updates (it looks as if this version is from 20017, so good luck!) and isn’t updated to an incompatible version, or bundle your own one and be responsible for ensuring that you are not shipping an insecure version (which, for a browser engine, typically means doing a new release every week or two). In the latter case, you might succeed if you can keep your dependencies to projects that have good security policies and do LTS releases but unfortunately that’s the minority of open source projects (proprietary libraries are far more rare in comparison but typically come with an SLA requiring security fixes in a timely fashion, or you wouldn’t be willing to pay for them at all).
“On FreeBSD, there is a base system that has long-term ABI stability guarantees. You can be very sure that software (including kernel modules) built on FreeBSD 10.0 will still work on FreeBSD 10.x, as long as its only dependencies are in the base system.”
On Debian, that’s almost exactly the same. If you install Debian 10.0, everything from the Debian repos will build and work. 10.1, 10.2, 10.3 don’t actually mean much except as markers that the baseline for installs has changed - all due to bug fixes, not new features, and with no breaking changes.
No new releases for the sake of new features happen until a major version release, at which point “stable” means 11 instead of 10. You are mostly guaranteed to be able to upgrade from 10 to 11 without a reinstall, but individual programs may have decided to change configuration formats (Apache 2.2 => 2.4) or have been dropped. The most catastrophic scenario is if Debian decided to no longer support your machine architecture, in which case there is no upgrade available.
I agree that Debian does not have a good solution to the language ecosystem problem. In order to get a reproducible build, you need to have a frozen repo of all the dependencies. The options that I can think of right now are:
The difference is that Debian tries to make this claim for however many tens of thousands of packages exist in their repos. FreeBSD does not, it makes this claim only for things that are either maintained by the FreeBSD project directly (e.g. the kernel, libc) or maintained by an upstream that has strong ABI stability guarantees that we can use (e.g. libc++). Anything that can’t make that kind of stability guarantee is not in the base system and so there is a clear dividing line between ‘this thing is ABI stable and will get security and bug fixes for the duration of a major release’ and ‘this thing is packaged as a convenience for you but security and bug fixes are the responsibility of the upstream project’.
Debian conflates these two, which leads to the problem that the article describes. There is no clear divide between things that the Debian project has enough expertise and manpower to be able to provide long-term guarantees (including back-porting security fixes if the shipped branch is no longer supported upstream), versus things that may have security vulnerabilities over the lifetime of a Debian release but won’t be patched because upstream no longer does releases from a compatible branch and the Debian project doesn’t have any contributors who know how to back-port the fixes. Web browsers are the canonical example of a place where this is a problem: even the Chrome team gave up trying to back-port security fixes to older Chrome releases and now have a policy of ‘you’re on the latest release or you’re insecure’. No third-party packaging infrastructure is going to have the resources to back-port these fixes (the code is aggressively refactored upstream, so even determining if the vulnerable code is in the version that you ship is a difficult problem), yet Debian tries to apply the same support policy to Chromium as to glibc.
Thanks, that helped explain it to me. I think you’re correct about the deficiency.