Not that he or anyone would or should particularly care, but Torvalds lost me absolutely and forever with this screed: I realized with this that, however similar our interests in system software might be, our values clash and that our differences are irreconcilable. On the one hand, this was a long time ago (though I remember vividly where I was when I first read it!) and I am forgiving of mistakes in youth (having made a few myself!), but I’m really not sure how one apologizes one’s way out of this one: even if there is regret for the tone (which there certainly should be!), the deeper message – that software should not, in fact, be debuggable – is deeply revealing. This also gives the lie to the notion that licensing has prevented DTrace from being integrated into Linux: licensing is a convenient excuse to cover for the larger sentiment that tooling for understanding the system is second class to the system itself.
I manage a few old Illumos boxes, (though I’m far from an expert). And it really does feel like the debugging tools are on a completely different level of integration to Linux.
A bunch of our internal documentation is mdb one liners. Eg Disk died? Query the SAS card with mdb
This also gives the lie to the notion that licensing has prevented DTrace from being integrated into Linux
I’m sorry. Are you actually stating (and able to state from a legal perspective) that DTrace and contributions to DTrace may be redistributed under the terms of the GPL?
Or are you referring to something else?
the deeper message – that software should not, in fact, be debuggable
I don’t read it that way. The point is that some kinds of debugging infrastructure have real costs, and he doesn’t think the juice is worth the squeeze.
The point is that some kinds of debugging infrastructure have real costs, and he doesn’t think the juice is worth the squeeze.
He is not speaking at all to the “real costs” of debugging infrastructure (something that I would also take issue with, as I believe that there is a false dichotomy between a production system and a debuggable one), but rather to the idea of debugging in the abstract. That he is (or certainly at least was) opposed to debuggers on principle – not as an engineering trade-off but rather the mere idea of understanding the system! – is (and was!) abhorrent to me.
I must misunderstand you: You seem to be suggesting debugging is impossible without a debugger. That’s bananas. What are you saying exactly?
He also, often talks about debugging with print statements, so I can’t understand why you would think he is opposed to “debugging” in the abstract. It just sounds like he doesn’t like debuggers — which are a specific and expensive kind of debugging infrastructure, but it sounds like you are talking about something else altogether…
I read this almost 20 years ago and it was a fantastic reason to work on FreeBSD or OpenSolaris. I think most Linux folks now use KVM’s gdb integration to debug the kernel in a VM.
Well worth reading from an anthropological point of view. It’s a great example of toxic gatekeeping! Of a a very specific kind with a very deliberate logic behind it, that is described explicitly:
I’m a bastard. I have absolutely no clue why people can ever think otherwise. Yet they do. People think I’m a nice guy, and the fact is that I’m a scheming, conniving bastard who doesn’t care for any hurt feelings or lost hours of work if it just results in what I consider to be a better system. […]
And I’m not just saying that. I’m really not a very nice person.
Kinda amazing that Linux ever got popular tbh. On the flip side, this kind of attitude is great at clamping down on bikesheds, useless design debates and other social antipatterns that make project management more difficult. If you take this attitude and someone is still willing to tell you you’re wrong about some technical thing, then a) they better be very sure and b) they better really know what they’re talking about. Ruthlessly weeding out the weak and ineffective (along with anyone unwilling or unable to argue with you) does get you a community with a certain kind of acumen and low overhead, as long as you have a steady stream of candidates lining up for the firing squad. It just, you know, does lots of other things too, like destroy people with high potential but low confidence, or normalize abuse based on certain criteria into abuse on whatever criteria one might feel like.
And if you take this kind of management approach, you better really know what you’re talking about as well, and you really, really better be right as often as you think you are. Otherwise you’re nothing but a demagogue who talks the talk but has no idea how to walk the walk, as is the case of plenty of recent popular figures I won’t bother naming.
(I am reminded of stories of Admiral King, the head of the US Navy for most of World War 2. My favorite comment about his personality was that he metaphorically “shaved with a blowtorch”. Incredibly competent and effective guy, largely because he generally had at least three other incredibly competent and effective people doing diplomacy and spin-control for him at any given time to keep up with all the people he pissed off along the way.)
Linus has since publicly acknowledged that his online communication skills were often more harmful than helpful. He has softened his tone tremendously and while he is still blunt about things he doesn’t like, he doesn’t really insult contributors anymore. This could partially be because he’s mostly a release manager these days.
The reason Linus kept a following at all was that even though he was a jerk in a lot of ways, he was almost always right. And people like a confident leader. I’m sure he (and other contributors at the time) justified the somewhat toxic culture by telling themselves that the quality of the kernel code was improved if contributors had (or quickly learned to develop) a thick skin against critical code reviews. Was it justified? IMO, probably not, since they could always refuse to merge a patch they think is dogshit without spewing insults. “The overall quality of this code is currently below our minimum standards, please feel free to resubmit after improvements are made.” Not sure this has the same ring to it.
People who know or talked to him in real life always say he is actually quite soft-spoken and nice to talk to, it was just his online presence that was abrasive.
From my recollection of that era, a lot of the credit goes to folks like Alan Cox. He has a high tolerance for Linus’ nonsense, but is a genuinely helpful and positive person. A lot of contributions were merged via his tree and folks working with him almost certainly had a much more positive experience than anyone working directly with Linus. As I recall, he was already at Red Hat then, which is probably a part of the reason they were able to have so much control over the young Linux ecosystem.
One aspect of this is that Linus has always meant for the group of people he directly interacts with to be small. There was a long period of development, contemporaneous with this mailing list post, where most users of linux were not using the mainline kernel. Distros all kept their own forks and those were what most people used. Many of the people who maintained those forks were also subsystem maintainers and they would funnel things up to the mainline. I’m not a Linus biographer but I’ve never read anything from him expressing a desire for that to end. I think that was a big part of why Linux got popular.
And if you take this kind of management approach, you better really know what you’re talking about as well, and you really, really better be right as often as you think you are.
I’d argue that this would be generally preferable, but I of course also know that this is not how it works and certainly not how people end up in management (or other) positions a lot of the time.
It’s a great example of toxic gatekeeping!
Out of curiosity. How would you define that? Do you differentiate between gatekeeping and toxic gatekeeping? I don’t know enough about it, so I am curious about “good” or “non-toxic” gatekeeping and whether that exists.
Well worth reading from an anthropological point of view.
I have another take then. To me the topic of “I am a bastard” and “I’m really not a very nice person.” feel horrendously unrelated to the topic, so not really like an argument, because it doesn’t seem to relate to the topic at all. The part about the debuggers being about details and not the overall picture, as well as learning to be careful kind of makes sense to me. I don’t know if I agree or disagree (I’d assume the best would be to be careful and be able to use a debugger), but it does relate to the topic, wheres Linus talking about his character traits feels out of place.
How would you define that? Do you differentiate between gatekeeping and toxic gatekeeping?
Good question! I hadn’t really thought about it in those terms. I suppose gatekeeping could be charitably defined as “preventing useless people from wasting my time” (and likely theirs as well), while toxic gatekeeping would be “gatekeeping by abusing the people I want to avoid until they go away”.
To me the topic of “I am a bastard” and “I’m really not a very nice person.” feel horrendously unrelated to the topic…
There are people out there that consider any form of gatekeeping toxic (I’m not one of them). In my view, “gatekeeping” is something like “That idea won’t work because …” or “That’s an interesting idea, do you have a proof-of-concept we can look at?”. Borderline “toxic gatekeeping” would be “That’s a stupid idea, because … “ (you’re not calling the person stupid, just the idea, but some people can’t or won’t see the distinction). “Toxic gatekeeping” would be “What are you? Stupid? That’s a horrible idea!” and end the discussion.
I wouldn’t call those first examples gatekeeping at all. When people talk about gatekeeping, they are usually talking about the exclusion of people from a community whose ideas or demographic don’t align with those of the group.
Bringing it back on-topic, the LKML used to be a somewhat toxic place but there was never any gatekeeping. Anyone was welcome to contribute improvements to the kernel no matter who they were, the only qualification being that you understand the kernel and the code you are writing, because if either fell short, you would certainly hear about it. That’s still somewhat true, but it’s a nicer place now.
To each their own, but a debugger makes my life as a developer 100x easier, and without any evidence the whole idea that a debugger somehow makes a person “less careful” or more prone to errors is a straw man.
A straw-man example of a debugger making a person “less careful” would be: “oh, there’s a NULL pointer dereference. Let me put a NULL check and return an error there” without investigating further as to why a NULL pointer is being passed in (in my opinion, that’s a bug—such checks tend to hide them).
Personally, I don’t use debuggers that often. In my career, I’ve been in too many environments (like DEBUG.EXE when trying to debug an MS-DOS program) or used languages that lack decent debuggers that I’m comfortable using printf() debugging. At my last job, I rarely used a debugger. My coworker was constantly in a debugger. Neither one of us was better than the other, just different.
My thought exactly. If you ever used IntelliJ (or another jetbrains tool interface to a debugger) you will wonder how you could ever live without it. I use it all the time.
A straw-man example of a debugger making a person “less careful” would be: “oh, there’s a NULL pointer dereference. Let me put a NULL check and return an error there” without investigating further as to why a NULL pointer is being passed in (in my opinion, that’s a bug—such checks tend to hide them).
I’m not sure of the argument being made here. A debugger doesn’t force that solution in any way. For me, a debugger makes it easier to fix that type of bug correctly because I can move up and down the stack to understand exactly how the null reference came about at runtime. printf has its place, but it generally there’s too much recompile/recreate overhead for my liking.
Not that he or anyone would or should particularly care, but Torvalds lost me absolutely and forever with this screed: I realized with this that, however similar our interests in system software might be, our values clash and that our differences are irreconcilable. On the one hand, this was a long time ago (though I remember vividly where I was when I first read it!) and I am forgiving of mistakes in youth (having made a few myself!), but I’m really not sure how one apologizes one’s way out of this one: even if there is regret for the tone (which there certainly should be!), the deeper message – that software should not, in fact, be debuggable – is deeply revealing. This also gives the lie to the notion that licensing has prevented DTrace from being integrated into Linux: licensing is a convenient excuse to cover for the larger sentiment that tooling for understanding the system is second class to the system itself.
I manage a few old Illumos boxes, (though I’m far from an expert). And it really does feel like the debugging tools are on a completely different level of integration to Linux.
A bunch of our internal documentation is mdb one liners. Eg Disk died? Query the SAS card with mdb
I’m sorry. Are you actually stating (and able to state from a legal perspective) that DTrace and contributions to DTrace may be redistributed under the terms of the GPL?
Or are you referring to something else?
I don’t read it that way. The point is that some kinds of debugging infrastructure have real costs, and he doesn’t think the juice is worth the squeeze.
Gnu confirms that upl is compatible https://www.gnu.org/licenses/license-list.en.html#UPL
He is not speaking at all to the “real costs” of debugging infrastructure (something that I would also take issue with, as I believe that there is a false dichotomy between a production system and a debuggable one), but rather to the idea of debugging in the abstract. That he is (or certainly at least was) opposed to debuggers on principle – not as an engineering trade-off but rather the mere idea of understanding the system! – is (and was!) abhorrent to me.
I must misunderstand you: You seem to be suggesting debugging is impossible without a debugger. That’s bananas. What are you saying exactly?
He also, often talks about debugging with print statements, so I can’t understand why you would think he is opposed to “debugging” in the abstract. It just sounds like he doesn’t like debuggers — which are a specific and expensive kind of debugging infrastructure, but it sounds like you are talking about something else altogether…
Modern Linux has a whole lot of tracing options though. perf, eBPF, ftrace, just to name a few.
For what should he be apologizing? and to whom?
Of what is the deeper message deeply revealing?
I read this almost 20 years ago and it was a fantastic reason to work on FreeBSD or OpenSolaris. I think most Linux folks now use KVM’s gdb integration to debug the kernel in a VM.
Well worth reading from an anthropological point of view. It’s a great example of toxic gatekeeping! Of a a very specific kind with a very deliberate logic behind it, that is described explicitly:
Kinda amazing that Linux ever got popular tbh. On the flip side, this kind of attitude is great at clamping down on bikesheds, useless design debates and other social antipatterns that make project management more difficult. If you take this attitude and someone is still willing to tell you you’re wrong about some technical thing, then a) they better be very sure and b) they better really know what they’re talking about. Ruthlessly weeding out the weak and ineffective (along with anyone unwilling or unable to argue with you) does get you a community with a certain kind of acumen and low overhead, as long as you have a steady stream of candidates lining up for the firing squad. It just, you know, does lots of other things too, like destroy people with high potential but low confidence, or normalize abuse based on certain criteria into abuse on whatever criteria one might feel like.
And if you take this kind of management approach, you better really know what you’re talking about as well, and you really, really better be right as often as you think you are. Otherwise you’re nothing but a demagogue who talks the talk but has no idea how to walk the walk, as is the case of plenty of recent popular figures I won’t bother naming.
(I am reminded of stories of Admiral King, the head of the US Navy for most of World War 2. My favorite comment about his personality was that he metaphorically “shaved with a blowtorch”. Incredibly competent and effective guy, largely because he generally had at least three other incredibly competent and effective people doing diplomacy and spin-control for him at any given time to keep up with all the people he pissed off along the way.)
It’s also worth noting a few other things…
Linus has since publicly acknowledged that his online communication skills were often more harmful than helpful. He has softened his tone tremendously and while he is still blunt about things he doesn’t like, he doesn’t really insult contributors anymore. This could partially be because he’s mostly a release manager these days.
The reason Linus kept a following at all was that even though he was a jerk in a lot of ways, he was almost always right. And people like a confident leader. I’m sure he (and other contributors at the time) justified the somewhat toxic culture by telling themselves that the quality of the kernel code was improved if contributors had (or quickly learned to develop) a thick skin against critical code reviews. Was it justified? IMO, probably not, since they could always refuse to merge a patch they think is dogshit without spewing insults. “The overall quality of this code is currently below our minimum standards, please feel free to resubmit after improvements are made.” Not sure this has the same ring to it.
People who know or talked to him in real life always say he is actually quite soft-spoken and nice to talk to, it was just his online presence that was abrasive.
From my recollection of that era, a lot of the credit goes to folks like Alan Cox. He has a high tolerance for Linus’ nonsense, but is a genuinely helpful and positive person. A lot of contributions were merged via his tree and folks working with him almost certainly had a much more positive experience than anyone working directly with Linus. As I recall, he was already at Red Hat then, which is probably a part of the reason they were able to have so much control over the young Linux ecosystem.
One aspect of this is that Linus has always meant for the group of people he directly interacts with to be small. There was a long period of development, contemporaneous with this mailing list post, where most users of linux were not using the mainline kernel. Distros all kept their own forks and those were what most people used. Many of the people who maintained those forks were also subsystem maintainers and they would funnel things up to the mainline. I’m not a Linus biographer but I’ve never read anything from him expressing a desire for that to end. I think that was a big part of why Linux got popular.
I’d argue that this would be generally preferable, but I of course also know that this is not how it works and certainly not how people end up in management (or other) positions a lot of the time.
Out of curiosity. How would you define that? Do you differentiate between gatekeeping and toxic gatekeeping? I don’t know enough about it, so I am curious about “good” or “non-toxic” gatekeeping and whether that exists.
I have another take then. To me the topic of “I am a bastard” and “I’m really not a very nice person.” feel horrendously unrelated to the topic, so not really like an argument, because it doesn’t seem to relate to the topic at all. The part about the debuggers being about details and not the overall picture, as well as learning to be careful kind of makes sense to me. I don’t know if I agree or disagree (I’d assume the best would be to be careful and be able to use a debugger), but it does relate to the topic, wheres Linus talking about his character traits feels out of place.
Good question! I hadn’t really thought about it in those terms. I suppose gatekeeping could be charitably defined as “preventing useless people from wasting my time” (and likely theirs as well), while toxic gatekeeping would be “gatekeeping by abusing the people I want to avoid until they go away”.
If you don’t look at the context, sure.
There are people out there that consider any form of gatekeeping toxic (I’m not one of them). In my view, “gatekeeping” is something like “That idea won’t work because …” or “That’s an interesting idea, do you have a proof-of-concept we can look at?”. Borderline “toxic gatekeeping” would be “That’s a stupid idea, because … “ (you’re not calling the person stupid, just the idea, but some people can’t or won’t see the distinction). “Toxic gatekeeping” would be “What are you? Stupid? That’s a horrible idea!” and end the discussion.
I wouldn’t call those first examples gatekeeping at all. When people talk about gatekeeping, they are usually talking about the exclusion of people from a community whose ideas or demographic don’t align with those of the group.
Bringing it back on-topic, the LKML used to be a somewhat toxic place but there was never any gatekeeping. Anyone was welcome to contribute improvements to the kernel no matter who they were, the only qualification being that you understand the kernel and the code you are writing, because if either fell short, you would certainly hear about it. That’s still somewhat true, but it’s a nicer place now.
I wonder if he’s changed his mind since then?
To each their own, but a debugger makes my life as a developer 100x easier, and without any evidence the whole idea that a debugger somehow makes a person “less careful” or more prone to errors is a straw man.
You could try asking.
A straw-man example of a debugger making a person “less careful” would be: “oh, there’s a
NULLpointer dereference. Let me put a NULL check and return an error there” without investigating further as to why aNULLpointer is being passed in (in my opinion, that’s a bug—such checks tend to hide them).Personally, I don’t use debuggers that often. In my career, I’ve been in too many environments (like
DEBUG.EXEwhen trying to debug an MS-DOS program) or used languages that lack decent debuggers that I’m comfortable usingprintf()debugging. At my last job, I rarely used a debugger. My coworker was constantly in a debugger. Neither one of us was better than the other, just different.If all you had was gdb, you’d think debuggers are useless too.
My thought exactly. If you ever used IntelliJ (or another jetbrains tool interface to a debugger) you will wonder how you could ever live without it. I use it all the time.
I’m not sure of the argument being made here. A debugger doesn’t force that solution in any way. For me, a debugger makes it easier to fix that type of bug correctly because I can move up and down the stack to understand exactly how the null reference came about at runtime. printf has its place, but it generally there’s too much recompile/recreate overhead for my liking.