Elsewhere, he also explains why he reacted so strongly in his earlier reply, and apologises:
This is a good and important move for him. People are more important than things. Treating people with dignity when they make mistakes is how we encourage open source development. I think too, behind most angry outbursts is a gap in understanding that isn’t be adequately communicated.
Email is a pretty bad communication channel in general, even if you don’t factor in stuff like language and cultural differences. Who knows what the effects of communicating almost all the time via email does to the human psyche…
Well treating people with dignity should be possible even through email. The important part is that he apologized, and explained why he was upset, and what he thought the community could do better to prevent this kind of issue in the future. It’s a billion times more effective than exploding. If you are about to write an explosive email to a colleague, maybe hit save as draft and come back a few hours later.
My rule: read it out loud as if I were saying it to someone’s face. Does it work? If it doesn’t, then it’s probably not ok. This still allows for strong dissent, but keeps it polite (for most normal people at least).
“For a developer, the hardening effort could be a great boon, in that
it could show nasty bugs early, it could make them easier to report,
and it could add a lot of useful information to that report that makes
them easier to fix too.”
This is actually one of the point fans of Design-by-Contract have been making since it takes you right to the bug. Memory-safe languages can prevent them. You don’t see Linus adopting many things like that in this quest to squash all the bugs. I say he’s mostly talking.
Now, let’s say I tried to commit something with hardening. He wants it to show the bug with a report. It can sometimes be obvious where something was hit but not always. So, a app gets hit with a non-obvious one eventually triggering some containment code. I’m guessing the Linux kernel already has support for pulling the code and data in the app from memory to analyze it in a way that shows where the attack is? Or does he expect me to dump all of that in a file to pull off the machine for manual analysis? Or just the writable parts in memory? I’m just wondering what’s standard in terms of support infrastructure for those doing it his way. There could even be opportunities to design mitigations around it.
You don’t see Linus adopting many things like that in this quest to squash all the bugs. I say he’s mostly talking.
You don’t see Linus adopting many things like that in this quest to squash all the bugs. I say he’s mostly talking.
I say this a lot whenever the new userspace rant crops up.
And not even in the context of memory safe languages. It’s far more basic than that. Linux doesn’t really have an extensive set of API/regression tests or a test infrastructure.
Without any of that, “don’t break userspace” is completely hollow. It’s really “don’t let me see you breaking userspace”; if folks actually cared about that that much then they would test for it.
This is also why I mostly consider attempts to rewrite linux in a safer language premature; without good testing it’s just not going to be doable.
Browsers are quite similar to operating systems in many ways (specifically, that they expose a large API/ecosystem within which you can program, and have a huge base of programs written for them). Browsers have extensive tests which go everywhere from testing the basic behavior of a feature to its million edge cases, including “nobody should write code that relies on this but we’re going to test it anyway” edge cases. When we did the Stylo work for Firefox a large, possibly majority, component of the work was just getting all these tests to pass, because we had lots of edge cases we missed. I can’t even begin to imagine how we’d do it without tests. I can’t even begin to imagine how a project like Linux would do it without tests.
I didn’t know they were lacking a test infrastructure. Yeah, that’s even worse than what I was saying. I especially like your characterization here:
“Without any of that, “don’t break userspace” is completely hollow. It’s really “don’t let me see you breaking userspace”; if folks actually cared about that that much then they would test for it.”
Yeah, this stuff is Linus’ ego until they get tests or contracts helping ensure that behavior. I also remember CompSci people bug-hunting the API’s had problems due to under or no specification of some components. They had to reverse engineer it a bit while they did the formal specs. They all found bugs, too.
It’s not like the kernel doesn’t get tested, though: https://stackoverflow.com/a/3180642/942130
I expected a little testing like that. Manishearth and I’s point is that this is a huge, critical project with more contributors than most whose leader is supposedly all about protecting the stability of the userspace. Yet, there’s no testing infrastructure for doing that. Yet, smaller projects and startups routinely pull that off for their growing codebases.
So, Linus is a hypocrite to not be doing what he can on testing side. There’s also a benefit to submitters where they could run the tests to spot breaks before submitting.
Could it be, that Linus is not the evil abuser that people paint him like? gasp
I find this comment equally useless as the other reply I commented on.
It’s is very usual that people resorting to abuse regularly don’t do it all the time. It doesn’t make it less abusive.
Discuss the interpretation of Linus outbursts in all directions, but all these discussions have to happen in a wider context, not based on single emails. Linus is criticised for regular outburts, no one is saying that he’s like that all the time.
The emails linked here (both post subject and the one linked by @pgl) paint a more nuanced picture than the one painted in the thread here. I find them interesting as I try to research the entire situation around the Linux kernel mailing list and Linus’ stewardship of the kernel development process.
Obviously if a public figure is abusive and contributes to a working environment that’s toxic, it’s a serious issue. Maybe doubly so when it’s a flagship open source product like the Linux kernel. I apologize if I sound as if I’m flippant and dismissive.
However, browsing through the comments in the previous thread I found a lot of speculation that this has harmed the kernel development process, made people less interested in contributing to open source, or generally being looked down upon. I did not find any links to actual first- or second-person accounts.
I realize that demanding such accounts may in itself be insensitive, however I’d prefer to judge Linus’ behavior based on them rather than unsourced speculation.
Edit I also realize that the speculation may in fact not be unsourced, but it’s in fact common knowledge (but unknown to me). If so, I should probably have just politely asked for links instead of attempting satire…
I was actually more annoyed by this comment then by yours. I was just seriously not understanding what you wanted to imply. Thank you for this elaborate answer, though, I’ll ramble in return.
I totally appreciate that Linus has a lot of experience with a project at that scale. And this email here perfectly illustrates it. It’s stern. Stern is great. I think the proper feedback to give here would be “that’s a great email” and not immediately bring up the standard debate. What annoys me is that people criticising specific Linus behaviour are painted like they can’t appreciate a good word from Linus or are just enraged by anything. This is also done by bringing up this debate in unrelated places.
No, often, criticism comes from people that invest a lot of time in FOSS project management, too. I’ve lost more then one contributor on projects I was involved in because some project lead jumped on them in fashions like Linus does, some specifically citing the Linus way. That’s why I take a lot of care about communication nowadays, especially over mediums that detach you from the speaker, such as email.
What gets me: Linus is also not the only person in such a position. It’s often postulated that his position is singular. It isn’t. There’s an ample amount of very nice people that pulled off similarly huge projects that pull a lot of weight. Matz, Knuth, Lamport, Larry Wall. And doing a thousand good things does not earn you the privilege of being an ass from time to time. That’s a classic abuse pattern that gets enabled by not pushing back when people cross boundaries.
The proper feedback to Linus outbreaks would therefore be “Linus, most of the work is great, but this is a boundary crossed”. That doesn’t mean immediately breaking ties or such, but now, Linus has shown that this is his habit and he wants it that way.
The fact that Linus felt the need to apologise in this case speaks for the people criticising his outburst at first. Maybe, things change? I’d be happy about it.
He inflicts a lot of harm with random outburts to people. Googles mangagement research in the recent years found that safety is important for creativity and a good work environment. Safety specifically doesn’t mean freedom from critcism, or even anger. But there are lines that should not be crossed, one of them being that you don’t call for people to be “retroactively aborted” (essentially wishing their death). This is an unambigous, spitting insult in all cultures on this planet and no way to treat contributors. I’m amazed of the number of people still defending such things. There’s perfectly fine ways to express anger without going on a rampage.
Retreating to “Finnish management style” also doesn’t cut it, unless you work with a project exclusively staffed by with finnish people.
I wonder where the idea that this discussion is just theory comes from. People have publicly left kernel development because of the nature of debate in the project, which comes from top, the most prominent example being Sage Sharp: http://sarah.thesharps.us/2015/10/05/closing-a-door/. There’s other accounts around, from people like Matthew Garrett and others. There’s ample number of people that specifically say they avoid Linus if possible, ask a couple of people on conferences. Expecting people to rehash all of those whenever the subject comes up again is also problematic. Also, word-of-mouth is a thing, because criticising Linus in a public space might yield with your inbox getting emails like this. You know how no one talks about the bad managers in a company in public, but once you go to a bar with some colleagues, they start talking? Same effect.
This thing is real. I am convinced that Linux would be better if they had better communication from top. This whole situation would be much better if Linus would have written just this email and not the other one. Then again, I’m not part of the Linux kernel, I have no say there. If things should change, its the crew around him that must do that.
I want to put this comment in a golden frame and show it to everyone who thinks Linus should totally be hurling insults.
Thanks a lot of this extensive reply. It’s given me a lot to think about and read up on.
Thanks for prompting it :).
I have several friends who quit kernel development because of the culture on LKML in general, and Linus specifically; they are some of the most talented folks I know. One of their friends quit doing upstream kernel security work and now sells vulnerabilities to semi-shady semi-government organizations.
Where is all the foul language and berating new kernel developers I have been led to expect from every email from Linus? Am disappoint.
I’m not sure what you want to imply, no one has said that Linus is angry or abusive all the time.
He’s criticised for regular outbursts and that’s it. Also, his outburts are more widely reported, even if you factor out criticism, because that’s what people do.
I am waxing satirical, in light of the long discussions here: https://lobste.rs/s/mij1sz/some_security_people_are_f_cking_morons
It’s not very good satire if it’s literally indistinguishable from the thing it’s satiring. You gotta exaggerate at least a little.
I keep trying to make it as a pro satirist but everyone tells me not to quit the day job.