When Jeff decided to fork rather than continue to contributing to illumos, I was initially disappointed – but ultimately I think he made the right decision. Not every project can be everything to all comers, as some decisions are fundamental and preclude coexistence with the alternative; e.g., you cannot simultaneously maintain a stable ABI at the same time as making breaking changes to improve on old design errors.
A project is generally a reflection of the values of the developer and user community built up around it. While our community is small, many users are people with customers. I used to work at Joyent where we sold public and private cloud software based on illumos, and if we were to take changes that changed an ABI or broke an on-disk format of some data file, we would end up with support calls and possibly lost customers. This is also true of the folks who work on OmniOS, who support a mixture of community and paid commercial users. Even as a user who is not currently paying anybody for support, I appreciate that I can upgrade and reboot without fear that my existing binaries will need to be recompiled.
As far as I can tell, Jeff has little to no interest in backwards compatibility – as with many in the various BSD communities, he feels it’s acceptable to recompile everything to run on each new release. That’s definitely a legitimate perspective that I respect, but it is not something we have in common; when you don’t fundamentally agree on a premise like that it can be difficult to arrive at a compromise when taking patches. It is often more work to support a backwards compatible evolution of what we have already, but the alternative is pushing that work (often as a surprise at upgrade time) onto consumers of the software. Even in a small project like ours there are many more users than there are engineers, and it feels like the right trade-off to us.
Though we haven’t got an iron clad rule about how to make breaking changes, we do make them. We stopped building the 32-bit x86 kernel at all for new year’s in 2018. There were people who were still using it, but we felt the maintenance burden was too great. We announced in 2017 that it was coming, to give one distribution some time to make a final 32-bit release, and then we ripped it out.
We’ve been reducing friction in the process we use to accept patches for some time. We used to use membership in the core decision making group to gate direct push access, but we have since relaxed that posture and regular contributors are able to push their own changes themselves. This has made it easier to take patches, and I hope has increased the sense of shared ownership of the software that many of us hold.
Though with my apologies as it is still woefully underdocumented, we have been working on a process for tighter design discussions: illumos Project Discussions. The goal is to enable longer term projects that might have multiple phases, and to give people an opportunity to work in advance with the core team to get to “yes”, rather than experience a “no” after you’ve already put in a bunch of effort. We aim to respond promptly and to not allow discussion to drag on forever in mailing list bike shed hell. I expect us to be able to use this process to make decisive changes in the face of uncertainty of impact, and indeed one of the IPDs I see when looking at the list is the removal of the kind of ancient compatibility interface that Jeff talks about in his post.
I don’t disagree that there have at times been hostile and unwelcoming interactions in our public spaces (mailing lists, IRC, the bug tracker) over the years. I am sure I have at various times become frustrated with disagreement and not handled it well. I and others in the project leadership have been working to improve this situation on several fronts, including institution of a code of conduct and confidential reporting alias last year. If people leave the project I definitely don’t want it to be because they were made to feel unwelcome.
More work is going on this year to move the way we take patches away from e-mail altogether, and to provide a better story for official CI/CD infrastructure for the project. We can always do better, and I have certainly read Jeff’s post in the spirit of finding things to continue to improve on our end. My door is always open via IRC, or Twitter, or e-mail, if people want to talk about the project.
Jeff is always welcome to come back, but I wish him well in whatever he does next!
As far as I can tell, Jeff has little to no interest in backwards compatibility …
I actually do. I’m just opposed to backwards compatibility above all else. I am opposed to backwards compatibility with hw/binary combinations that aren’t possible to use.
(libbc was a dumpster fire, it needed to go - I’m glad it’s gone now but, IMO, it shouldn’t have taken 5 years especially given how easy the change was to make.)
Though we haven’t got an iron clad rule about how to make breaking changes, we do make them.
I strongly suggest you codify it somehow. Otherwise, you run the risk of accepting some and rejecting other changes arbitrarily. At least historically, it seemed that Joyent contributors could do no wrong, while us lowly non-Joyeurs had to fight extra just to get changes reviewed. Needless to say, this wasn’t exactly motivating.
We can always do better, and I have certainly read Jeff’s post in the spirit of finding things to continue to improve on our end.
Every time a removal of something antique is mentioned, people that haven’t contributed a single line of code would come out of the woodwork with various forms of “I use it” or even a hypothetical “someone could be using it to do $foo”.
Perhaps somewhat ironically considering Jeff’s thesis, we removed calendar(1) in 2017. Somebody did come out of the woodwork to suggest they still use it, but we gave them notice and I believe somebody helped them find an alternative, and then we pulled it out.
My least favourite is the commit that reverts. No prior discussion or review, just an ad-hoc FreeBSD isn't an encyclopedia, and suddenly a tool in surely more than a couple .login files is just gone.
There’s an argument for getting rid of cruft, but that’s not the way you do it.
Every time a removal of something antique is mentioned, people that haven’t contributed a single line of code would come out of the woodwork with various forms of “I use it” or even a hypothetical “someone could be using it to do $foo”.
This part of the article told me a lot. They fail to realize that different people do actually have different values. And that having different values does not make these people Toxic.
The fork might have failed due to lack of empathy. Or the author’s belief that those with the same views as them are some sort of “silent majority” that would follow them along with the fork, whereas those with different values are “obnoxiously loud”, “toxic” or even “trolls” outright.
It’s fine that those people have values. It’s even fine that those people to occasionally talk about those values.
It’s not fine that those people shit up threads and mailing lists with arcane requests that derail development and confuse other contributors. If the situation is as the author suggests, I can’t blame them for wanting to fork.
My take is that if a minority uses a thing, and the mainstream project that embodies that thing wants to move away from it, then it’s the responsibility of the minority to either fork or even better do the work to make whatever it is they desperately want and need into an optional add-on so the main project can stop thinking about it.
Definitely. This seems to come up a lot with support for niche, vintage or otherwise uncommon architectures in all sorts of projects; if someone is going to insist on using a program on a thrity-year-old processor, they surely should have the technical nous to maintain it themselves, rather than having the project slowly accumulate hacky code purely there for the purpose of backwards compatibility.
shit up threads and mailing lists with arcane requests that derail development
Maybe that’s a very narrow viewpoint. Maybe this sentiment of caring for not breaking working hardware is actually widespread among the community, and it would have been a better use of the author’s time to learn to work with the values of the community, rather than starting a fork with little support and putting a massive amount of effort on it, just to end up in disappointment.
Assertiveness goes a long way. This does include trying to understand others and work with them, rather than bulldozing your own view.
It takes a serious amount of prepotency to refer to those with opposing views as “peanut gallery”, and based on this, I am not surprised they failed to gather the support to make their fork successful.
In one instance (in August 2015), when I tried to remove an ancient specialized kernel module binary compatibility workaround that Sun added in Solaris 9 (released in 2002), I was asked about what turned out to be a completely ridiculous situation—you’d need to try to load a kernel module built against Solaris 8 that used this rather specialized API, an API that has been changed twice in incompatible ways since it was added (once in 2005 by Sun, and once in April 2015 by Joyent).
I like stable platforms just as much as the next engineer who has had to deal with, say, webshit.
rather than starting a fork with little support and putting a massive amount of effort on it, just to end up in disappointment.
Again. from the article:
I know I’m biased, but I think we’ve made some good changes in Unleashed and I’d love to see at least some of them make their way into illumos-gate or one of the illumos distros. So, I hope that someone will have the time and interest to integrate some of these changes.
Finally, I’d like to encourage anyone who may be interested not to be afraid of forking an open source project. Even if it doesn’t work out, it is extremely educational.
That was of course preceded by the things that they accomplished. I won’t speak for the author (who is already in this thread), but I don’t agree with reading “just ending in disappointment” in it.
When Jeff decided to fork rather than continue to contributing to illumos, I was initially disappointed – but ultimately I think he made the right decision. Not every project can be everything to all comers, as some decisions are fundamental and preclude coexistence with the alternative; e.g., you cannot simultaneously maintain a stable ABI at the same time as making breaking changes to improve on old design errors.
A project is generally a reflection of the values of the developer and user community built up around it. While our community is small, many users are people with customers. I used to work at Joyent where we sold public and private cloud software based on illumos, and if we were to take changes that changed an ABI or broke an on-disk format of some data file, we would end up with support calls and possibly lost customers. This is also true of the folks who work on OmniOS, who support a mixture of community and paid commercial users. Even as a user who is not currently paying anybody for support, I appreciate that I can upgrade and reboot without fear that my existing binaries will need to be recompiled.
As far as I can tell, Jeff has little to no interest in backwards compatibility – as with many in the various BSD communities, he feels it’s acceptable to recompile everything to run on each new release. That’s definitely a legitimate perspective that I respect, but it is not something we have in common; when you don’t fundamentally agree on a premise like that it can be difficult to arrive at a compromise when taking patches. It is often more work to support a backwards compatible evolution of what we have already, but the alternative is pushing that work (often as a surprise at upgrade time) onto consumers of the software. Even in a small project like ours there are many more users than there are engineers, and it feels like the right trade-off to us.
Though we haven’t got an iron clad rule about how to make breaking changes, we do make them. We stopped building the 32-bit x86 kernel at all for new year’s in 2018. There were people who were still using it, but we felt the maintenance burden was too great. We announced in 2017 that it was coming, to give one distribution some time to make a final 32-bit release, and then we ripped it out.
We’ve been reducing friction in the process we use to accept patches for some time. We used to use membership in the core decision making group to gate direct push access, but we have since relaxed that posture and regular contributors are able to push their own changes themselves. This has made it easier to take patches, and I hope has increased the sense of shared ownership of the software that many of us hold.
Though with my apologies as it is still woefully underdocumented, we have been working on a process for tighter design discussions: illumos Project Discussions. The goal is to enable longer term projects that might have multiple phases, and to give people an opportunity to work in advance with the core team to get to “yes”, rather than experience a “no” after you’ve already put in a bunch of effort. We aim to respond promptly and to not allow discussion to drag on forever in mailing list bike shed hell. I expect us to be able to use this process to make decisive changes in the face of uncertainty of impact, and indeed one of the IPDs I see when looking at the list is the removal of the kind of ancient compatibility interface that Jeff talks about in his post.
I don’t disagree that there have at times been hostile and unwelcoming interactions in our public spaces (mailing lists, IRC, the bug tracker) over the years. I am sure I have at various times become frustrated with disagreement and not handled it well. I and others in the project leadership have been working to improve this situation on several fronts, including institution of a code of conduct and confidential reporting alias last year. If people leave the project I definitely don’t want it to be because they were made to feel unwelcome.
More work is going on this year to move the way we take patches away from e-mail altogether, and to provide a better story for official CI/CD infrastructure for the project. We can always do better, and I have certainly read Jeff’s post in the spirit of finding things to continue to improve on our end. My door is always open via IRC, or Twitter, or e-mail, if people want to talk about the project.
Jeff is always welcome to come back, but I wish him well in whatever he does next!
I actually do. I’m just opposed to backwards compatibility above all else. I am opposed to backwards compatibility with hw/binary combinations that aren’t possible to use.
(libbc was a dumpster fire, it needed to go - I’m glad it’s gone now but, IMO, it shouldn’t have taken 5 years especially given how easy the change was to make.)
I strongly suggest you codify it somehow. Otherwise, you run the risk of accepting some and rejecting other changes arbitrarily. At least historically, it seemed that Joyent contributors could do no wrong, while us lowly non-Joyeurs had to fight extra just to get changes reviewed. Needless to say, this wasn’t exactly motivating.
Glad to hear it!
My favorite recent example: “Add extremely useful calendar(1) application to FreeBSD”
Perhaps somewhat ironically considering Jeff’s thesis, we removed calendar(1) in 2017. Somebody did come out of the woodwork to suggest they still use it, but we gave them notice and I believe somebody helped them find an alternative, and then we pulled it out.
My least favourite is the commit that reverts. No prior discussion or review, just an ad-hoc
FreeBSD isn't an encyclopedia, and suddenly a tool in surely more than a couple.loginfiles is just gone.There’s an argument for getting rid of cruft, but that’s not the way you do it.
This part of the article told me a lot. They fail to realize that different people do actually have different values. And that having different values does not make these people Toxic.
The fork might have failed due to lack of empathy. Or the author’s belief that those with the same views as them are some sort of “silent majority” that would follow them along with the fork, whereas those with different values are “obnoxiously loud”, “toxic” or even “trolls” outright.
It’s fine that those people have values. It’s even fine that those people to occasionally talk about those values.
It’s not fine that those people shit up threads and mailing lists with arcane requests that derail development and confuse other contributors. If the situation is as the author suggests, I can’t blame them for wanting to fork.
I agree.
My take is that if a minority uses a thing, and the mainstream project that embodies that thing wants to move away from it, then it’s the responsibility of the minority to either fork or even better do the work to make whatever it is they desperately want and need into an optional add-on so the main project can stop thinking about it.
Definitely. This seems to come up a lot with support for niche, vintage or otherwise uncommon architectures in all sorts of projects; if someone is going to insist on using a program on a thrity-year-old processor, they surely should have the technical nous to maintain it themselves, rather than having the project slowly accumulate hacky code purely there for the purpose of backwards compatibility.
Maybe that’s a very narrow viewpoint. Maybe this sentiment of caring for not breaking working hardware is actually widespread among the community, and it would have been a better use of the author’s time to learn to work with the values of the community, rather than starting a fork with little support and putting a massive amount of effort on it, just to end up in disappointment.
Assertiveness goes a long way. This does include trying to understand others and work with them, rather than bulldozing your own view.
It takes a serious amount of prepotency to refer to those with opposing views as “peanut gallery”, and based on this, I am not surprised they failed to gather the support to make their fork successful.
From the submission:
I like stable platforms just as much as the next engineer who has had to deal with, say, webshit.
Again. from the article:
That was of course preceded by the things that they accomplished. I won’t speak for the author (who is already in this thread), but I don’t agree with reading “just ending in disappointment” in it.
We didn’t take over the world - that was a bit disappointing, but otherwise you are correct. :)
Stable APIs for drivers is something everybody who’s ever pushed Illumos to me mentioned. It’s not something nobody cares about.
And I’m sorry re: webshit. I’ve had to deal with webshit too (it’s inescapable) and I can more or less relate.