1. 43
  1.  

  2. 22

    Here’s the original email: http://lkml.iu.edu/hypermail/linux/kernel/1711.2/01701.html

    And here’s a bit of commentary from Errata Security: http://blog.erratasec.com/2017/11/why-linus-is-right-as-usual.html

    1. 6

      If you read the whole thread from start, it gives you an entirely new perspective: 1, 2, 3, 4 and the Linus’ reply - 5

      1. 2

        Later, he explains why he reacted so strongly in his earlier reply, and apologises:

        https://lkml.org/lkml/2017/11/21/315

      2. 14

        “Some security people have scoffed at me when I say that security problems are primarily “just bugs”.”

        I scoff at him telling me the covert, timing channels caused by Linux threads running on multicore CPU’s with caches are just bugs. They’re actually a side effect of a CPU design that gives Linux much of its desired performance in most scenarios. It’s simultaneously a vulnerability in some others. Stopping them, if even possible, would require more breaking changes to the Linux kernel. I even tried keeping the shared caches off in one or more designs. Anyone seeing the results knows he’d never modify the kernel to force that by default.

        Some other problems start at the API level where UNIX architecture itself adds risks that a security-focused kernel wouldn’t. Changing it would require changing Linux’s behavior in ways unacceptable to Linus. Yet again, he’d be full of shit if he equated dealing with those security risks with just fixing some bugs in existing, backward-compatible kernel.

        EDIT: Note that there is overlap between correctness/safety and security. Many if not most security problems do start out as mere bugs in requirements, design, or code. Others take intentional design, though, to work around a malicious, intelligent attacker trying to turn everything against you. Malice poses more issues than accidents. Some even require fundamentally different architectures.

        1. 5

          Etiquette aside, I’d still say such things are “just bugs”. I suppose different people use the word differently, but to me it’s basically “behaviour/outcome/situation generally agreed to be undesirable”. I think there’s general agreement that vulnerability to timing attacks is undesirable. In your example, it sounds like fixing those bugs would break backwards compatibility and lower performance; two things which are also generally agreed to be bad and hence are (just) bugs.

          When we have to make tradeoffs, the best approach is often to report the bug (so we have a historical reference to point to and add subsequent ideas to), mark it as “won’t fix” and move on. This seems to be what Linus is doing, but using more offensive language (which is an important, but orthogonal, debate).

          1. 15

            Traditionally, we called something a bug if it was a mistake we didn’t want. If it was there by design, it wasn’t a bug. The shared resources that give rise to covert channels weren’t bugs. They were straight-up design decisions thought to be beneficial. You’re right about making and documenting tradeoffs. We need to do both. I think the general usage of bug is more narrow to most developers, though, versus what you or Linus described. It’s almost always what they didn’t want.

          2. 3

            I’m not sure what your point is. All engineering involves tradeoffs. Performance versus covert channels enabled by shared caches is an easy tradeoff in a general production operating system. Using POSIX versus some non-standard API is a similarly easy tradeoff. The security issues POSIX introduces are a penalty for making an operating system that runs useful applications.

            1. 2

              He equated security vulnerabilities with bugs. I countered they’re different with some vulnerabilities not seen as bugs but intentional design decisions. That was the point.

              1. 4

                But in this case, the “hardening” was looking for things that were almost certainly mistakes - bugs. There are security problems caused by deliberate design decisions too I agree.

          3. 22

            Is anybody else getting tired of this guy? It seems like he gets a pass for being an asshole because he’s been an asshole for longer than people have cared about being an asshole. This behaviour would violate the code of conduct of any sensible project.

            1. 20

              Yeah you can be direct without being a dick. “I won’t merge something that breaks the kernel, please find some other way.” would have worked just fine.

              1. 18

                And in fact, that’s how it works most of the time.

                Linus’ reputation as an asshole is due, in part, to selection bias, and the high profile of Linux. Thousands and thousands of merges go into the kernel all the time without a problem, and without Linus going off on a rant.

                I don’t work on the kernel, but my observation has been that the big blow ups seem to only come after people repeatedly break the rules. I won’t say Linus handles it well, but I don’t think he’s as bad as some maintainers in some smaller open source communities.

                1. 6

                  It’s survivor bias, not selection bias. He also owes a lot of it to businesses that got his kernel out there plus make up a lot of contributions. It’s not as if him being an asshole combined with some FOSS contributors that loved that asshole equals success of Linux.

                  1. 6

                    Not that it makes a difference, but I believe I was correct in calling it selection bias. Nobody will post to Lobste.rs or write an article when Linus is being nice, so in general people only see the bitchy posts, hence the bad reputation.

                    1. 7

                      I don’t think that’s strictly true.

                      I think there are a few salient points here:

                      • If you just go by his posts that make it to lobste.rs/hacker news/reddit, you’ll get an extremely skewed view of Linus’s attitude. The vast majority of his communications are somewhere between polite and blunt. (Remember, his job basically entails reading and writing emails all day every day, and he writes something social-media worthy at most monthly.) To the best of my knowledge, he’s never exploded at a kernel newbie, only at long-time kernel hackers.
                      • That said, his attitude is still incredibly problematic. It’s been demonstrated to drive away talented developers. It drives away new developers, even if they are not themselves directly getting yelled at by Linus.
                      • Linux’s success is a complicated beast dependent on a whole host of factors, including to varying extents all of good timing (a few years later and BSD would have made it through its legal troubles), technical talent, corporate support, sheer dumb luck. Linus’s attitude certainly had an impact, but where it slots in that long list is impossible to say; I think it was a negative factor and thus, based on Linux’s evident success, had a relatively low impact, but obviously that’s pure speculation.
                      1. 2

                        Even adding in that first bullet from you and jlarocco, I think I still agree with about everything you said. It’s consistent with my position that he goes too far with the bad stuff.

                    2. 5

                      I have never ever behaved this way to my colleagues, and I suspect you haven’t either. So to call it selection bias is to ignore that he’s doing something that the vast majority of us would be fired for. It’s not okay to rarely shout down your coworkers. Sure it’s better to do it rarely than every single day, but the fact that we keep examples of this is a clear example that he has no checks and balances.

                      1. 1

                        And generally these are people who have a corporate position that makes them believe they are entitled to break the rules.

                    3. 45

                      The only thing I’m getting tired of is people pulling the odd email out of thousands and wringing hands over how mean Old Man Linus is.

                      Maybe folks should reflect on how, after 25 years of loud and blatant protestations by Linus, fucking morons keep trying to merge the same types of userspace breaking bugs.

                      Maybe, sometimes, a broader more accepting tent isn’t the answer.

                      1. 27

                        If Linus being famously mean for 25 years hasn’t produced a productive culture, perhaps it’s time to try a new approach.

                        1. 26

                          But it has produced a plenty productive culture - a culture that produces a better end product than many more professional environments, in fact.

                          1. 5

                            Professionally “rewarding”, still toxic at the personal end. It’s mentioned in this article mentioned at the main link.

                            1. 3

                              Professionally “rewarding”, still toxic at the personal end. It’s mentioned in this article mentioned at the main link.

                              And little of value was lost. This is how Sarah Sharp tried to publicly humiliate the guy with a wife and daughter - https://lwn.net/Articles/559077/ :

                              *Snort*. Perhaps we haven’t interacted very often, but I have never seen you be nice in person at KS. Well, there was that one time you came to me and very quietly explained you had a problem with your USB 3.0 ports, but you came off as “scared to talk to a girl kernel developer” more than “I’m trying to be polite”.

                              I disagree with labelling things and people as “toxic” in general, but I’ll choose Linus over Sarah any day: https://linux.slashdot.org/story/15/10/05/2031247/linux-kernel-dev-sarah-sharp-quits-citing-brutal-communications-style

                              1. 12

                                Did we read the same mail? Did you read any of the quoted parts from Linus? A guy that refuses to even consider treating people with respect is a clear-cut asshole. I’d much rather work with someone that talks about treating people with dignity than someone that refuses to consider the concept seriously.

                                1. [Comment from banned user removed]

                                  1. 16

                                    You got it backward. Linus is the special snowflake here if he can continue to be that unnecessarily-abusive publicly with no consequences just because his work just happened to get popular in that way. Expecting people to deliver constructive criticism or not chase away good talent is the default for those managing good teams in most places. A manager/leaser simply getting off on abusing those doing work is adding nothing of value to the project in doing so.

                                    Instead of a snowflake, people just expect to be treated with decency by default with shitflakes like Linus able to get away with being exceptional jerks.

                                    1. [Comment from banned user removed]

                                      1. 2

                                        That would be a good trait if he had it. Instead, he’s still pushing monoliths in unsafe languages with limited metaprogramming. Took forever to get it reliable versus Minix 3’s a few developers in a few years. So much for his decisions being merit-based. ;)

                                        1. 3

                                          he’s still pushing monoliths in unsafe languages with limited metaprogramming

                                          Linux is modular.

                                          There was no serious alternative to C back in 1991 and, as much as I love metaprogramming, it increases the amount of surprises for the programmer.

                                          Took forever to get it reliable versus Minix 3’s a few developers in a few years.

                                          It’s easy to be reliable when your biggest deployment is on Intel’s spy chip.

                                          Minix was little more than an emulator pet for a few CS students, before that. Low on drivers, low on performance, low on functionality. You might as well compare Linux with L4…

                                          1. 4

                                            It’s modular in kernel mode for full compromise and crash potential. There were a bunch of memory-safe languages used in other OS’s before 1991, esp from Wirth, whose safety could be selectively disabled. Worst case compile them to C to leverage compilers while dodging programmer-related problems like some projects did.

                                            “It’s easy to be reliable when your biggest deployment is on Intel’s spy chip.”

                                            DOD is one of Red Hat’s biggest customers and sources of funding for contributions to Linux. Lots of kernel bugs were also found by analysis and testing tools from CompSci similarly funded by US-government. I agree that helps but a company just freeloaded off Minix 3. Should’ve went with GPL.

                                            “Minix was little more than an emulator pet for a few CS students, before that. Low on drivers, low on performance, low on functionality. “

                                            You should’ve seen the first Linux. It was similar but crashed more. Meanwhile, several years earlier than 1991, QNX folks were building a microkernel-based UNIX that became reliable as hell, fast, and deterministic. The Playbook versus iPad comparisons were the first I got to see with multimedia after BeOS. In both, the multithreading without stalling abilities were mindboggling versus the better-funded, older competition. My Linux systems can still come to a crawl over misbehaved applications to this day. Things that the others made highly unlikely with better architecture.

                                            You’re arguments were who used it and features that came with labor put in. Either one of those put into better architecture would’ve made an even better Linux. So, they’re neutral points. Mine was Linus wouldn’t listen anyway. If you believed him in Linus vs Tannenbaum, things like the Playbook w/ QNX and BeOS would’ve been impossible to program easily or perform well. Way wrong cuz he’s about politics and arbitrary preferences as much as merit. Like most developers.

                          2. 18

                            It has, though?

                            What I meant was that newcomers seem to be ignoring 25 years of norms and others being surprised when those newcomers–who are doing dumb things–are told to knock it off.

                            1. 6

                              Yeah, With “productive”, which seems to have been a really poor word choice, I meant one that didn’t have to teach the same thing over and over in the way you described. Sorry to you and the other responders for the confusion.

                              1. 2

                                Thanks for the clarification, and agreed.

                            2. 13

                              Linux is the most successful, widespread operating system kernel of all time. You can say the man’s rude, but you can’t say the results demonstrate unproductivity.

                              1. 2

                                The others from Microsoft, Apple, and IBM also were driven by assholes who were greedy on top of it. Just throwing that in there even though Im anti-Linus in this debate.

                            3. 21

                              There’s honestly no good reason to be hostile. It doesn’t actually help reduce the problem, evidenced by the fact that what he has done hasn’t worked. Instead they need processes for check in, code reviews, and linters. Linus should be delegating more as well if this is bothering him so much.

                              1. 4

                                That’s not a theory supported by the evidence.

                                1. 3

                                  What he’s done hasn’t worked. Most contributions are from businesses. Many good talent say they avoid it. That seems to be evidence of something. Meanwhile, the Rust crowd managed to get piles of people early on for one of the hardest-to-learn languages I’ve seen in a while. They used the opposite approach. Now, two projects or even ten aren’t a lot of datapoints for an empirical assessment of which method is working. Oh, what can we do to see how much or how little damage Linus is doing to kernel in terms of lost contributions?

                                  Oh wait, it turns out researchers in universities have been doing both observational studies and surveys on large numbers of organizations and people for decades covering this very thing. A key question was which management styles have most positive impact. One thing that’s pretty consistent in the research is that people working for assholes were much more likely to half-ass their work on purpose, dodge doing work, or even sabotage that person where possible. People working for those that treated them with respect or constructive criticism did better work. That kept being a result of most studies. Crazy to ignore decades of consistency in human behavior when trying to decide how best to treat them in a FOSS project for achieving goals such as more contributors, higher-quality contributions, and so on.

                                  The theory supported by the evidence is that Linus’ style when doing what’s in the OP is unnecessarily rude and destructive. The evidence says he’ll loose a lot of talent since that talent just needs a worthwhile project to work on rather than his project. Just like he feels he doesn’t need them. Objectively, such a result is bad for the project if one wants it to improve. He might be willing to sacrifice features, QA, and so on for the personal enjoyment of those insults. That is what he’s doing. Anyone defending him shouldn’t pretend otherwise. Instead, they should shift to their actual argument of “I know we’re losing contributors that could’ve made the Linux kernel even better. The main reason is Linus’s personal preference. We think that’s a good status quo to maintain because…” That does look to be a harder position to defend, though, on either technical or moral grounds.

                                  1. 1

                                    Just to say, would be nice if you posted source of the research you’re referencing.

                                    1. 3

                                      I’m too much of an overloaded procrastinator to give it to you. I’d have to resurvey it as I bet the Web 1.0 sites are gone, new ones have formed, and I’ll have to dig through tons of noise. I do plan to either find or do another meta study on that in future since it’s so critical. For IT, I always told people to read the PeopleWare book and Dale Carnegie’s How to Win Friends and Influence People. Lots of managers hand out the latter believing it’s great advice. Implies they think blunt assholes are non-ideal. The No Asshole Rule book also cited a bunch of studies on effects of people being assholes downward or upward in an organizations recommending against it.

                                      I do need to recollect the studies, though. Plus do a new bookmarking solution I’ve been procrastinating on since Firefox is full to point it constantly looses bookmarks lol.

                            4. 8

                              Linux would not be what it is today if they would be “merge-first-fix-later” type code-conducted safe place for noobs to mess around in.

                              1. 16

                                If you’re going to be derogatory, safe space is properly mocking.

                                There is a near infinite gap between “let the noods do whatever they want to the codebase” and “don’t degrade people’s character because they submitted a PR you dislike”.

                                I guess some people are just more tolerant of a project leader taking their anger and frustration out on people trying to get involved?

                                1. 20

                                  The problem isn’t that he wouldn’t merge the person’s code. The problem is the unprofessional way that he treats other people. The fact that you think the problem is that he wouldn’t merge the code is either deeply concerning or purposefully avoiding the issue.

                                  1. 7

                                    If you actually read the damn thread, you see that Linus actually explained this pretty clearly: http://lkml.iu.edu/hypermail/linux/kernel/1711.2/01357.html

                                    The person decides to ignore Linus and Linus gets angry, I really don’t see a problem here.

                                    1. 2

                                      Ok, I read the full thread. It’s more reasonable in the other parts. Kees seems to have put some work into making it acceptable. Later on, I see someone do what Linus should’ve done in the first place in giving specific details about where he’s coming from in a way that wouldn’t have bothered me as a contributor:

                                      http://lkml.iu.edu/hypermail/linux/kernel/1711.2/03732.html

                                      After seeing that, I’m more annoyed by whoever was half-assing security contributions to the kernel so much that it will be hard for worthwhile contributions to get in.

                                      1. 1

                                        Yeah, same here - I think there are just special snowflakes who think that human psychology has anything to do with whether or not the kernel is going to continue running reliably for me, the kernel user. Guess what snowflakes, nobody cares about the feelings if the product doesn’t work.

                                        Not to mention, this is only the squeaky wheel - Linus has been nice and professional and accommodating many, many times over. Many more times over, in fact. It just never makes the news ..

                                    2. [Comment removed by author]

                                      1. 3

                                        I’m not used to navigating the CVE database, is there an easy way to restrict issues to just the Linux kernel?

                                    3. 6

                                      Nope. I think he’s great. And I’m very glad that he is stewarding the Linux project to this day. Whether you think its ‘nice’ or not, his management of the Linux kernel has produced superlative results - and sometimes, in the thick of the mob, you have to be an asshole to get people to work the way they need to work to continue producing quality results.

                                      What I am sick of, is petulant snowflakes who think they know better than Linus how to manage the 1000’s of developers that want to have their fingers in the pie. The kernel doesn’t care about your feelings, and neither do 99.9999% of the kernels really important people: its users.

                                      1. 4

                                        Since when did asking to be treated with the bare minimum of basic human decency become a “special snowflake” thing? Nobody wants Linus to write “You’re so wonderful, and special, and beautiful, but I cannot accept this patch because, despite how wonderful and unique it and you are, it just won’t work with Linux’s performance requirements.”

                                        NOBODY is asking for that. So I don’t get why I keep seeing “special snowflake” thrown around. I guess it’s just a strawman? (OH WAIT I GET IT NOW!)

                                        Notice how your comment is verging on “nobody can critique the way Linus runs the project (that we all rely on in myriad ways)”. Aren’t snowflakes the ones who want to shut people down and stop discussion? Isn’t it the “snowflakes” that want to prevent people from having to hear mean things? (Like, stop taking your anger out on contributors because you’re not 7 anymore).

                                        Doesn’t it kind of seem like–and bear with me here, I know it hurts–that you’ve become the special snowflake? Stifling discussion, needing a space where someone you look up to is immune to criticism, insulting people who are just trying to have a conversation?

                                        Isn’t it your post that seems to be the petulant one here?

                                        1. 2

                                          Since when did asking to be treated with the bare minimum of basic human decency become a “special snowflake” thing?

                                          Precisely at the point where well-established ground rules, respected by the rest of us, were continually broken with no regard for the work load incurred, nor the hassle of having to deal with all the noise. Or did you miss the part where known, functional, productive policies were repeatedly ignored in the rush to get this patch included in the next release?

                                          Its one thing for a contributor to feel like they should be treated with respect as a special snowflake whose feelings are more important than the work, or in this case non-work, that they are contributing to the lives of others; its another thing to respect the very foundations of the activity from which one is attempting to derive that respect in ones own life.

                                          Perhaps you missed the part where this could have been a disaster for the Linux kernel, and a lot of time was wasted having to deal with it, since the original developer decided to ignore the policies, well-since established as being necessary to the task of managing the Kernel patch integration process?

                                          “nobody can critique the way Linus runs the project (that we all rely on in myriad ways)”

                                          Well, whether you like it or not, its the truth: Linus has guided the way through decades of these kinds of events, and we have an extraordinarily powerful tool that has revolutionised computers as a result. Perhaps you ought to consider whether the quality of your own work and contributions might improve if you harden up a little and don’t take offence so easily. Time and again, this proves to be true - in the real world and in this fantasy land we’re currently sharing as participants in this thread.

                                          The poster involved in this incident seems to have accepted that they were, in fact, violating a fundamental policy of the Linux kernel developer group, and has addressed the issue in a way that moves things forward - how, exactly, would Linux kernel development be pushed forward by your insistence at being treated like a snowflake?

                                          A mistake was made - the policy was not followed - and Linus jumped on the guy. He’ll never do it again, many many others have also learned the importance of the check-in policy (Rule #1: Don’t Break The Kernel.) and he doesn’t seem at all worse for the wear, personally, as a consequence; its really only folks such as yourself who are getting so easily upset about this, because Linus somehow doesn’t conform to your particular cultural ideal.

                                          Perhaps you haven’t been following Linux kernel development for long, or with much attention - there are many, many counter-cases of Linus having great relations with the developer group, which don’t seem to figure into your equation that “Linus is rude”. He’s precisely rude when he needs to be, and an awesome, polite, respectful individual, all the while. Please try to avail yourself of that truth before you continue ad-hoc insults and insinuations against random Internet strangers. It hurts my feelings to be challenged by an ignoramus.

                                          Doesn’t it kind of seem like–and bear with me here, I know it hurts–that you’ve become the special snowflake?

                                          Are you assuming that I wouldn’t want to be called a snowflake when appropriate? Because, I’m quite a snowflake, and often, when its appropriate or otherwise. Absolutely nothing with being called one, when you are one. Or, is there some other kind of kettle we should be boiling for tea?

                                      2. 2

                                        If a security vulnerability is introduced by design it’s still a bug. It just means the mistake was made at design time as opposed to implementation time.

                                        1. 2

                                          In all sincerity here, what would it mean for a person to say, “I’m not going to tolerate this behavior?”

                                          Linus would still own the Linux trademark. He’d still control the mainline kernel repo. The “lieutenants” that manage various areas of the kernel would still control those areas and report to him. It seems very unlikely that they would support a coup. (Anyone who had a major problem with Linus’ behavior wouldn’t have lasted long enough to get one of the top positions.)

                                          As a user, you can choose not to use or support Linux. But as a user, you don’t get to change the way the project runs.

                                          I think the most extreme option you’d have would be to fork the source code and try to attract both a large developer community and a large user base on the basis of running a more inclusive community. But there’s a chicken-and-egg problem to that approach.

                                          There’s an implicit hypothesis that says, “A more inclusive community will produce a better kernel.” Let’s assume that proves to be true. Some users would switch on that basis alone, but most will wait to see practical benefits. Since it would still take time for a fork to produce tangible benefits, you’d have to attract developers and users with the promise alone. We have a small set of cases to examine, where a major open source project was forked with the intention of creating a better community. It appears that the majority of users will hang back with a “wait and see” approach.

                                          I really don’t know what kind of negative feedback anyone could apply to Linus that would have an effect.

                                          1. 1

                                            Working code doesn’t care about your feelings. Working code is completely orthogonal to human emotions. My computer runs whether I’m crying or not.

                                          2. 0

                                            This behaviour would violate the code of conduct of any sensible project.

                                            Maybe you should run a kernel made by the CoC crowd. I’ll stick with the foul-mouthed guy.

                                            1. 5

                                              The only one I know off top of head is Redox OS since it used Rust CoC. It’s got potential but is alpha software. All the rest that are good seem to be made with different philosophies with a range of civility.

                                              I am interested if anyone knows of another usable OS made with all activity enforced with a CoC a la Rust/Redox. At least the basic console or GUI apps so it’s usable for some day to day stuff.

                                                1. 1

                                                  Good catch. This one…

                                                  “There can be no place within the FreeBSD Community for discriminatory speech or action. We do not believe anyone should be treated any differently based on who they are, where they are from, where their ancestors were from, what they look like, what gender they identify as, who they choose to sleep with, how old they are, their physical capabilities or what sort of religious beliefs they may hold. What matters is the contribution they are able to make to the project, and only that.”

                                                  …is where the politically-motivated try to find a lot of wiggle room for censorship as beliefs vary. One reason I collect these is so we can look back at data in commits or on forums to see what impact they have. Note I said OS that was made with the activity enforced this way. Some could have it added as an evolution of moderation policies well after it’s a successful project that was built on a different philosophy. How long has that CoC been in FreeBSD?

                                                  1. 4

                                                    How long has that CoC been in FreeBSD?

                                                    It’s relatively new - it was announced in July 2015. Even before the CoC was added a few developers were ejected for abusive behaviour (I’m not going to dig those out, but you can find references online).

                                                    1. 2

                                                      Ok, so it’s not an example of an OS developed under the CoC. It was a highly-mature OS that probably started with really different kinds of people just because they were the norm for early days of BSD’s and Linux. With your comment, they were just using common sense of ejecting folks who were obviously abusive without anything more formal or constraining. That still leaves Redox as the only one I know that had the policy and supporters of it from the start.

                                                      The main way I think this can be tested is with frameworks or libraries that are in same language and crowd. Basically, keep the situation as close as possible so about the only strong variable is community style. Should be easier with libraries or frameworks since they’re more accessible to new contributors. People are always doing more of those.

                                          3. 6

                                            The fact is that Torvalds is correct. He ’s pushing back against the kind of stupid bureaucratic messing about that passes for engineering in many large organizations.

                                            1. [Comment from banned user removed]

                                              1. 2

                                                This reads like copypasta…