1. 28

      I actually super appreciate your work on engaging with people friendlier than you have in the past. I agree with a lot of your points, but have been usually put off from how you frame them. I’m glad to see this change and it sounds like it’s worked out favorably (sans the downvotes).

      1. 22

        Is anybody else getting tired of this guy? It seems like he gets a pass for being an asshole because he’s been an asshole for longer than people have cared about being an asshole. This behaviour would violate the code of conduct of any sensible project.

        1. 20

          Yeah you can be direct without being a dick. “I won’t merge something that breaks the kernel, please find some other way.” would have worked just fine.

          1. 18

            And in fact, that’s how it works most of the time.

            Linus’ reputation as an asshole is due, in part, to selection bias, and the high profile of Linux. Thousands and thousands of merges go into the kernel all the time without a problem, and without Linus going off on a rant.

            I don’t work on the kernel, but my observation has been that the big blow ups seem to only come after people repeatedly break the rules. I won’t say Linus handles it well, but I don’t think he’s as bad as some maintainers in some smaller open source communities.

            1. 6

              It’s survivor bias, not selection bias. He also owes a lot of it to businesses that got his kernel out there plus make up a lot of contributions. It’s not as if him being an asshole combined with some FOSS contributors that loved that asshole equals success of Linux.

              1. 6

                Not that it makes a difference, but I believe I was correct in calling it selection bias. Nobody will post to Lobste.rs or write an article when Linus is being nice, so in general people only see the bitchy posts, hence the bad reputation.

                1. 7

                  I don’t think that’s strictly true.

                  I think there are a few salient points here:

                  • If you just go by his posts that make it to lobste.rs/hacker news/reddit, you’ll get an extremely skewed view of Linus’s attitude. The vast majority of his communications are somewhere between polite and blunt. (Remember, his job basically entails reading and writing emails all day every day, and he writes something social-media worthy at most monthly.) To the best of my knowledge, he’s never exploded at a kernel newbie, only at long-time kernel hackers.
                  • That said, his attitude is still incredibly problematic. It’s been demonstrated to drive away talented developers. It drives away new developers, even if they are not themselves directly getting yelled at by Linus.
                  • Linux’s success is a complicated beast dependent on a whole host of factors, including to varying extents all of good timing (a few years later and BSD would have made it through its legal troubles), technical talent, corporate support, sheer dumb luck. Linus’s attitude certainly had an impact, but where it slots in that long list is impossible to say; I think it was a negative factor and thus, based on Linux’s evident success, had a relatively low impact, but obviously that’s pure speculation.
                  1. 2

                    Even adding in that first bullet from you and jlarocco, I think I still agree with about everything you said. It’s consistent with my position that he goes too far with the bad stuff.

                2. 5

                  I have never ever behaved this way to my colleagues, and I suspect you haven’t either. So to call it selection bias is to ignore that he’s doing something that the vast majority of us would be fired for. It’s not okay to rarely shout down your coworkers. Sure it’s better to do it rarely than every single day, but the fact that we keep examples of this is a clear example that he has no checks and balances.

                  1. 1

                    And generally these are people who have a corporate position that makes them believe they are entitled to break the rules.

                3. 45

                  The only thing I’m getting tired of is people pulling the odd email out of thousands and wringing hands over how mean Old Man Linus is.

                  Maybe folks should reflect on how, after 25 years of loud and blatant protestations by Linus, fucking morons keep trying to merge the same types of userspace breaking bugs.

                  Maybe, sometimes, a broader more accepting tent isn’t the answer.

                  1. 27

                    If Linus being famously mean for 25 years hasn’t produced a productive culture, perhaps it’s time to try a new approach.

                    1. 26

                      But it has produced a plenty productive culture - a culture that produces a better end product than many more professional environments, in fact.

                      1. 5

                        Professionally “rewarding”, still toxic at the personal end. It’s mentioned in this article mentioned at the main link.

                        1. 3

                          Professionally “rewarding”, still toxic at the personal end. It’s mentioned in this article mentioned at the main link.

                          And little of value was lost. This is how Sarah Sharp tried to publicly humiliate the guy with a wife and daughter - https://lwn.net/Articles/559077/ :

                          *Snort*. Perhaps we haven’t interacted very often, but I have never seen you be nice in person at KS. Well, there was that one time you came to me and very quietly explained you had a problem with your USB 3.0 ports, but you came off as “scared to talk to a girl kernel developer” more than “I’m trying to be polite”.

                          I disagree with labelling things and people as “toxic” in general, but I’ll choose Linus over Sarah any day: https://linux.slashdot.org/story/15/10/05/2031247/linux-kernel-dev-sarah-sharp-quits-citing-brutal-communications-style

                          1. 12

                            Did we read the same mail? Did you read any of the quoted parts from Linus? A guy that refuses to even consider treating people with respect is a clear-cut asshole. I’d much rather work with someone that talks about treating people with dignity than someone that refuses to consider the concept seriously.

                            1. [Comment from banned user removed]

                              1. 16

                                You got it backward. Linus is the special snowflake here if he can continue to be that unnecessarily-abusive publicly with no consequences just because his work just happened to get popular in that way. Expecting people to deliver constructive criticism or not chase away good talent is the default for those managing good teams in most places. A manager/leaser simply getting off on abusing those doing work is adding nothing of value to the project in doing so.

                                Instead of a snowflake, people just expect to be treated with decency by default with shitflakes like Linus able to get away with being exceptional jerks.

                                1. [Comment from banned user removed]

                                  1. 2

                                    That would be a good trait if he had it. Instead, he’s still pushing monoliths in unsafe languages with limited metaprogramming. Took forever to get it reliable versus Minix 3’s a few developers in a few years. So much for his decisions being merit-based. ;)

                                    1. 3

                                      he’s still pushing monoliths in unsafe languages with limited metaprogramming

                                      Linux is modular.

                                      There was no serious alternative to C back in 1991 and, as much as I love metaprogramming, it increases the amount of surprises for the programmer.

                                      Took forever to get it reliable versus Minix 3’s a few developers in a few years.

                                      It’s easy to be reliable when your biggest deployment is on Intel’s spy chip.

                                      Minix was little more than an emulator pet for a few CS students, before that. Low on drivers, low on performance, low on functionality. You might as well compare Linux with L4…

                                      1. 4

                                        It’s modular in kernel mode for full compromise and crash potential. There were a bunch of memory-safe languages used in other OS’s before 1991, esp from Wirth, whose safety could be selectively disabled. Worst case compile them to C to leverage compilers while dodging programmer-related problems like some projects did.

                                        “It’s easy to be reliable when your biggest deployment is on Intel’s spy chip.”

                                        DOD is one of Red Hat’s biggest customers and sources of funding for contributions to Linux. Lots of kernel bugs were also found by analysis and testing tools from CompSci similarly funded by US-government. I agree that helps but a company just freeloaded off Minix 3. Should’ve went with GPL.

                                        “Minix was little more than an emulator pet for a few CS students, before that. Low on drivers, low on performance, low on functionality. “

                                        You should’ve seen the first Linux. It was similar but crashed more. Meanwhile, several years earlier than 1991, QNX folks were building a microkernel-based UNIX that became reliable as hell, fast, and deterministic. The Playbook versus iPad comparisons were the first I got to see with multimedia after BeOS. In both, the multithreading without stalling abilities were mindboggling versus the better-funded, older competition. My Linux systems can still come to a crawl over misbehaved applications to this day. Things that the others made highly unlikely with better architecture.

                                        You’re arguments were who used it and features that came with labor put in. Either one of those put into better architecture would’ve made an even better Linux. So, they’re neutral points. Mine was Linus wouldn’t listen anyway. If you believed him in Linus vs Tannenbaum, things like the Playbook w/ QNX and BeOS would’ve been impossible to program easily or perform well. Way wrong cuz he’s about politics and arbitrary preferences as much as merit. Like most developers.

                      2. 18

                        It has, though?

                        What I meant was that newcomers seem to be ignoring 25 years of norms and others being surprised when those newcomers–who are doing dumb things–are told to knock it off.

                        1. 6

                          Yeah, With “productive”, which seems to have been a really poor word choice, I meant one that didn’t have to teach the same thing over and over in the way you described. Sorry to you and the other responders for the confusion.

                          1. 2

                            Thanks for the clarification, and agreed.

                        2. 13

                          Linux is the most successful, widespread operating system kernel of all time. You can say the man’s rude, but you can’t say the results demonstrate unproductivity.

                          1. 2

                            The others from Microsoft, Apple, and IBM also were driven by assholes who were greedy on top of it. Just throwing that in there even though Im anti-Linus in this debate.

                        3. 21

                          There’s honestly no good reason to be hostile. It doesn’t actually help reduce the problem, evidenced by the fact that what he has done hasn’t worked. Instead they need processes for check in, code reviews, and linters. Linus should be delegating more as well if this is bothering him so much.

                          1. 4

                            That’s not a theory supported by the evidence.

                            1. 3

                              What he’s done hasn’t worked. Most contributions are from businesses. Many good talent say they avoid it. That seems to be evidence of something. Meanwhile, the Rust crowd managed to get piles of people early on for one of the hardest-to-learn languages I’ve seen in a while. They used the opposite approach. Now, two projects or even ten aren’t a lot of datapoints for an empirical assessment of which method is working. Oh, what can we do to see how much or how little damage Linus is doing to kernel in terms of lost contributions?

                              Oh wait, it turns out researchers in universities have been doing both observational studies and surveys on large numbers of organizations and people for decades covering this very thing. A key question was which management styles have most positive impact. One thing that’s pretty consistent in the research is that people working for assholes were much more likely to half-ass their work on purpose, dodge doing work, or even sabotage that person where possible. People working for those that treated them with respect or constructive criticism did better work. That kept being a result of most studies. Crazy to ignore decades of consistency in human behavior when trying to decide how best to treat them in a FOSS project for achieving goals such as more contributors, higher-quality contributions, and so on.

                              The theory supported by the evidence is that Linus’ style when doing what’s in the OP is unnecessarily rude and destructive. The evidence says he’ll loose a lot of talent since that talent just needs a worthwhile project to work on rather than his project. Just like he feels he doesn’t need them. Objectively, such a result is bad for the project if one wants it to improve. He might be willing to sacrifice features, QA, and so on for the personal enjoyment of those insults. That is what he’s doing. Anyone defending him shouldn’t pretend otherwise. Instead, they should shift to their actual argument of “I know we’re losing contributors that could’ve made the Linux kernel even better. The main reason is Linus’s personal preference. We think that’s a good status quo to maintain because…” That does look to be a harder position to defend, though, on either technical or moral grounds.

                              1. 1

                                Just to say, would be nice if you posted source of the research you’re referencing.

                                1. 3

                                  I’m too much of an overloaded procrastinator to give it to you. I’d have to resurvey it as I bet the Web 1.0 sites are gone, new ones have formed, and I’ll have to dig through tons of noise. I do plan to either find or do another meta study on that in future since it’s so critical. For IT, I always told people to read the PeopleWare book and Dale Carnegie’s How to Win Friends and Influence People. Lots of managers hand out the latter believing it’s great advice. Implies they think blunt assholes are non-ideal. The No Asshole Rule book also cited a bunch of studies on effects of people being assholes downward or upward in an organizations recommending against it.

                                  I do need to recollect the studies, though. Plus do a new bookmarking solution I’ve been procrastinating on since Firefox is full to point it constantly looses bookmarks lol.

                        4. 8

                          Linux would not be what it is today if they would be “merge-first-fix-later” type code-conducted safe place for noobs to mess around in.

                          1. 16

                            If you’re going to be derogatory, safe space is properly mocking.

                            There is a near infinite gap between “let the noods do whatever they want to the codebase” and “don’t degrade people’s character because they submitted a PR you dislike”.

                            I guess some people are just more tolerant of a project leader taking their anger and frustration out on people trying to get involved?

                            1. 20

                              The problem isn’t that he wouldn’t merge the person’s code. The problem is the unprofessional way that he treats other people. The fact that you think the problem is that he wouldn’t merge the code is either deeply concerning or purposefully avoiding the issue.

                              1. 7

                                If you actually read the damn thread, you see that Linus actually explained this pretty clearly: http://lkml.iu.edu/hypermail/linux/kernel/1711.2/01357.html

                                The person decides to ignore Linus and Linus gets angry, I really don’t see a problem here.

                                1. 2

                                  Ok, I read the full thread. It’s more reasonable in the other parts. Kees seems to have put some work into making it acceptable. Later on, I see someone do what Linus should’ve done in the first place in giving specific details about where he’s coming from in a way that wouldn’t have bothered me as a contributor:


                                  After seeing that, I’m more annoyed by whoever was half-assing security contributions to the kernel so much that it will be hard for worthwhile contributions to get in.

                                  1. 1

                                    Yeah, same here - I think there are just special snowflakes who think that human psychology has anything to do with whether or not the kernel is going to continue running reliably for me, the kernel user. Guess what snowflakes, nobody cares about the feelings if the product doesn’t work.

                                    Not to mention, this is only the squeaky wheel - Linus has been nice and professional and accommodating many, many times over. Many more times over, in fact. It just never makes the news ..

                                2. [Comment removed by author]

                                  1. 3

                                    I’m not used to navigating the CVE database, is there an easy way to restrict issues to just the Linux kernel?

                                3. 6

                                  Nope. I think he’s great. And I’m very glad that he is stewarding the Linux project to this day. Whether you think its ‘nice’ or not, his management of the Linux kernel has produced superlative results - and sometimes, in the thick of the mob, you have to be an asshole to get people to work the way they need to work to continue producing quality results.

                                  What I am sick of, is petulant snowflakes who think they know better than Linus how to manage the 1000’s of developers that want to have their fingers in the pie. The kernel doesn’t care about your feelings, and neither do 99.9999% of the kernels really important people: its users.

                                  1. 4

                                    Since when did asking to be treated with the bare minimum of basic human decency become a “special snowflake” thing? Nobody wants Linus to write “You’re so wonderful, and special, and beautiful, but I cannot accept this patch because, despite how wonderful and unique it and you are, it just won’t work with Linux’s performance requirements.”

                                    NOBODY is asking for that. So I don’t get why I keep seeing “special snowflake” thrown around. I guess it’s just a strawman? (OH WAIT I GET IT NOW!)

                                    Notice how your comment is verging on “nobody can critique the way Linus runs the project (that we all rely on in myriad ways)”. Aren’t snowflakes the ones who want to shut people down and stop discussion? Isn’t it the “snowflakes” that want to prevent people from having to hear mean things? (Like, stop taking your anger out on contributors because you’re not 7 anymore).

                                    Doesn’t it kind of seem like–and bear with me here, I know it hurts–that you’ve become the special snowflake? Stifling discussion, needing a space where someone you look up to is immune to criticism, insulting people who are just trying to have a conversation?

                                    Isn’t it your post that seems to be the petulant one here?

                                    1. 2

                                      Since when did asking to be treated with the bare minimum of basic human decency become a “special snowflake” thing?

                                      Precisely at the point where well-established ground rules, respected by the rest of us, were continually broken with no regard for the work load incurred, nor the hassle of having to deal with all the noise. Or did you miss the part where known, functional, productive policies were repeatedly ignored in the rush to get this patch included in the next release?

                                      Its one thing for a contributor to feel like they should be treated with respect as a special snowflake whose feelings are more important than the work, or in this case non-work, that they are contributing to the lives of others; its another thing to respect the very foundations of the activity from which one is attempting to derive that respect in ones own life.

                                      Perhaps you missed the part where this could have been a disaster for the Linux kernel, and a lot of time was wasted having to deal with it, since the original developer decided to ignore the policies, well-since established as being necessary to the task of managing the Kernel patch integration process?

                                      “nobody can critique the way Linus runs the project (that we all rely on in myriad ways)”

                                      Well, whether you like it or not, its the truth: Linus has guided the way through decades of these kinds of events, and we have an extraordinarily powerful tool that has revolutionised computers as a result. Perhaps you ought to consider whether the quality of your own work and contributions might improve if you harden up a little and don’t take offence so easily. Time and again, this proves to be true - in the real world and in this fantasy land we’re currently sharing as participants in this thread.

                                      The poster involved in this incident seems to have accepted that they were, in fact, violating a fundamental policy of the Linux kernel developer group, and has addressed the issue in a way that moves things forward - how, exactly, would Linux kernel development be pushed forward by your insistence at being treated like a snowflake?

                                      A mistake was made - the policy was not followed - and Linus jumped on the guy. He’ll never do it again, many many others have also learned the importance of the check-in policy (Rule #1: Don’t Break The Kernel.) and he doesn’t seem at all worse for the wear, personally, as a consequence; its really only folks such as yourself who are getting so easily upset about this, because Linus somehow doesn’t conform to your particular cultural ideal.

                                      Perhaps you haven’t been following Linux kernel development for long, or with much attention - there are many, many counter-cases of Linus having great relations with the developer group, which don’t seem to figure into your equation that “Linus is rude”. He’s precisely rude when he needs to be, and an awesome, polite, respectful individual, all the while. Please try to avail yourself of that truth before you continue ad-hoc insults and insinuations against random Internet strangers. It hurts my feelings to be challenged by an ignoramus.

                                      Doesn’t it kind of seem like–and bear with me here, I know it hurts–that you’ve become the special snowflake?

                                      Are you assuming that I wouldn’t want to be called a snowflake when appropriate? Because, I’m quite a snowflake, and often, when its appropriate or otherwise. Absolutely nothing with being called one, when you are one. Or, is there some other kind of kettle we should be boiling for tea?

                                  2. 2

                                    If a security vulnerability is introduced by design it’s still a bug. It just means the mistake was made at design time as opposed to implementation time.

                                    1. 2

                                      In all sincerity here, what would it mean for a person to say, “I’m not going to tolerate this behavior?”

                                      Linus would still own the Linux trademark. He’d still control the mainline kernel repo. The “lieutenants” that manage various areas of the kernel would still control those areas and report to him. It seems very unlikely that they would support a coup. (Anyone who had a major problem with Linus’ behavior wouldn’t have lasted long enough to get one of the top positions.)

                                      As a user, you can choose not to use or support Linux. But as a user, you don’t get to change the way the project runs.

                                      I think the most extreme option you’d have would be to fork the source code and try to attract both a large developer community and a large user base on the basis of running a more inclusive community. But there’s a chicken-and-egg problem to that approach.

                                      There’s an implicit hypothesis that says, “A more inclusive community will produce a better kernel.” Let’s assume that proves to be true. Some users would switch on that basis alone, but most will wait to see practical benefits. Since it would still take time for a fork to produce tangible benefits, you’d have to attract developers and users with the promise alone. We have a small set of cases to examine, where a major open source project was forked with the intention of creating a better community. It appears that the majority of users will hang back with a “wait and see” approach.

                                      I really don’t know what kind of negative feedback anyone could apply to Linus that would have an effect.

                                      1. 1

                                        Working code doesn’t care about your feelings. Working code is completely orthogonal to human emotions. My computer runs whether I’m crying or not.

                                      2. 0

                                        This behaviour would violate the code of conduct of any sensible project.

                                        Maybe you should run a kernel made by the CoC crowd. I’ll stick with the foul-mouthed guy.

                                        1. 5

                                          The only one I know off top of head is Redox OS since it used Rust CoC. It’s got potential but is alpha software. All the rest that are good seem to be made with different philosophies with a range of civility.

                                          I am interested if anyone knows of another usable OS made with all activity enforced with a CoC a la Rust/Redox. At least the basic console or GUI apps so it’s usable for some day to day stuff.

                                            1. 1

                                              Good catch. This one…

                                              “There can be no place within the FreeBSD Community for discriminatory speech or action. We do not believe anyone should be treated any differently based on who they are, where they are from, where their ancestors were from, what they look like, what gender they identify as, who they choose to sleep with, how old they are, their physical capabilities or what sort of religious beliefs they may hold. What matters is the contribution they are able to make to the project, and only that.”

                                              …is where the politically-motivated try to find a lot of wiggle room for censorship as beliefs vary. One reason I collect these is so we can look back at data in commits or on forums to see what impact they have. Note I said OS that was made with the activity enforced this way. Some could have it added as an evolution of moderation policies well after it’s a successful project that was built on a different philosophy. How long has that CoC been in FreeBSD?

                                              1. 4

                                                How long has that CoC been in FreeBSD?

                                                It’s relatively new - it was announced in July 2015. Even before the CoC was added a few developers were ejected for abusive behaviour (I’m not going to dig those out, but you can find references online).

                                                1. 2

                                                  Ok, so it’s not an example of an OS developed under the CoC. It was a highly-mature OS that probably started with really different kinds of people just because they were the norm for early days of BSD’s and Linux. With your comment, they were just using common sense of ejecting folks who were obviously abusive without anything more formal or constraining. That still leaves Redox as the only one I know that had the policy and supporters of it from the start.

                                                  The main way I think this can be tested is with frameworks or libraries that are in same language and crowd. Basically, keep the situation as close as possible so about the only strong variable is community style. Should be easier with libraries or frameworks since they’re more accessible to new contributors. People are always doing more of those.

                                      1. 11


                                        I’m a new manager and I will not be afraid to tell my people to leave when I think they’ve been working too much.

                                        1. 3

                                          This is super important! Please keep doing this!!

                                        1. 7

                                          I’m lucky to have Exceptional work life balance. I work at a fairly large company in the SF bay area on a very supportive team. It’s probably worth noting that I’m a staff-level engineer and a tech lead.


                                          • We implemented unlimited PTO recently, but as part of that, our company gave us an additional week of vacation off in July for people with children and to offset people that don’t take time off since there’s no pressure of accrued PTO expiring.
                                          • Management in my org actively tells employees to take more than a week off every quarter and are almost always supportive of employees taking PTO (the only exception is in the case where an entire team is out at once - that isn’t feasible)
                                          • Except when oncall, work pays for a separate phone for me, and that gets switched off precisely when I leave for the day. My coworkers and management know that I’m unreachable except in cases of emergency. What’s an emergency? Things like the leap second bug a few years ago - I was called in then to help, and that was the last time I was reached out to as an emergency.
                                          • I have a 1+ hour commute each way, but I can and do work from home as much as I want. This an be as few as one day a week to the whole week. As long as I make myself available for video calls when I’m working, it’s absolutely not a problem.
                                          • My director and I have both stood at people’s desks and told them to go home if we were worried that they were staying too late or working to long.
                                          • On days I commute, I usually work from 10:30am to 5:30pm. This is all super flexible, and I have rolled in at 11am and left at 4pm. Again, it’s never a problem since I get my work done and meet my commitments.
                                          • My management makes a conscious effort to look at optics of their involvement in post-work activities. They make sure that while they’re supportive of people going to meetups or doing technical stuff after work, that it’s never mandatory, that they’re never involved, and people with any role power are not involved.
                                          • Time is dedicated to allowing employees read and learn while they’re at work. If an IC wants to work towards a personal goal, sometimes they’ll work with the manager to make that personal goal, even if not work-related, an OKR.
                                          1. 1

                                            I’m super interested in this, except for the part where it seems to have an explicit dependency on keybase’s infrastructure? I don’t have any qualms with keybase as a company, but it’d be nice to be able to use this in restricted environments where I may not have access to keybase, or if I want data that isn’t subject to keybases availability constraints.

                                            1. 3

                                              Interesting article, but it’d be super interesting to know how dropping rtti and exception support led to the hang (or if those weren’t the culprit and it was some other compiler flags omitted from that snippet).

                                              1. 3


                                                I had tried something like this a year or so ago, but ended up dropping it.


                                                1. 15

                                                  Just use runit. It’s dead simple, actually documented, actually used in production, BSD licensed, and so on. I use it on my work computer with no problems. It’s no bullshit, no bloat, you don’t have to “learn runit” to use it and get exactly what you want.

                                                  1. 5

                                                    I’m aware of runit. It does seem pretty nice, but there are a few things about it that bother me. I don’t want to get into specifics here since it can so easily become a matter of one opinion vs the other, but I’ll try to write about some general issues which Dinit should handle well (and which I don’t think runit does) at some point in the near future.

                                                    1. 3

                                                      Well one things that comes to mind is that runit doesn’t deal well (or at all) with (double-)forking services. Those are unfortunate by themselves — I mean, let the OS do its job please! — but still exist.

                                                    2. 3

                                                      I have run into some odd behavior with runit a time or two, somehow managing to get something into a weird wedged state. I could never figure out what the exact problem was (maybe it is fixed by now?). Oddly enough, I never had the same issue with ye olde daemontools.

                                                      Aside from that, I do also like runit – as a non pid 1 process supervisor.

                                                      1. 2

                                                        We use runit heavily at my job. It’s a massive pain to deal with, and we have to use a lot of automation to deal with the incredibly frequent issues we have with it. I would never recommend it to anyone, honestly.

                                                        1. 2


                                                          1. 4

                                                            I’ve mentioned this here: https://lobste.rs/s/2qjf4o/problems_with_systemd_why_i_like_bsd_init#c_8qtwla

                                                            Also, since then, we’ve had problems with svlogd losing track of the process that it’s logging for. Also it should be noted that you absolutely don’t get logging for free, and it requires additional management.

                                                            1. 2

                                                              Runit does have support for dependencies in a way, you put the service start command in the run file and it starts the other service first, or blocks until it finishes starting. Right?

                                                              How does it lose track of its controlled processes? Like do you know what causes it? For example I know runit doesn’t manage double forking daemons.

                                                              What kind of scaffolding have you set up to ensure logging? What breakages do you have to guard against? How do you guard against them?

                                                              Do you know why svlogd loses the process? As I understand, it’s just hooked to stdout/stderr, so how could it lose track? What specific situations does that happen in? How did you resolve?

                                                              I know it’s a lot of questions, but I’m genuinely curious and would love to learn more.

                                                              1. 5

                                                                How does it lose track of its controlled processes? Like do you know what causes it? For example I know runit doesn’t manage double forking daemons.

                                                                The reason runit, daemontools classic, and most other non-pid-1 supervisors “lose track” of supervised processes comes down to the lack of setsid(2). If you have a multiprocess service, in 99% of cases you should create a new process group for it and use process group signaling rather than single process signaling. If you don’t signal the entire process group when you sv down foo, you’re only killing the parent, and any children will become orphans inherited by pid 1, or an “orphaned process group” that might keep running.

                                                                A few years back I got a patch into daemontools-encore to do all of this, although we screwed up the default behavior (more on that in a second). You can read more about the hows and whys of multiprocess supervision in that daemontools-encore PR.

                                                                If you’re using a pid-1 supervisor like BSD init, upstart, systemd, etc it can do something more intelligent with these orphans, since it’s the one that inherits them. Also, pid-1 supervisors usually run services in a new process group by default.

                                                                Now, the screw-up: when we added multiprocess service support to daemontools-encore, I initially made setsid an opt-in feature. So by default, services wouldn’t run in a new process group, which is the “classic” behavior of daemontools, runit, et al. There are a few popular services like nginx that actually rely on this behavior for hot upgrades, or for more control over child processes during shutdown. Unfortunately I let myself get talked out of that, and we made setsid opt-out. That broke some of these existing services for people, and the maintainer did the worst thing possible, and half-backed out multiprocess service support.

                                                                At this point bruceg/daemontools-encore is pretty broken wrt multiprocess services, and I wouldn’t recommend using it. I don’t have the heart to go back and argue with the maintainer that we fix it by introducing breaking behavior again. Instead I re-forked it, fixed multiprocess support, and have been happily and quietly managing multiprocess services on production systems for several years now. It all just works. If you’re interested, here’s my fork: https://github.com/acg/daemontools-encore/tree/ubuntu-package-1.13

                                                                I guess I’ll end with a request for advice. How should I handle this situation fellow lobsters? Suck it up and get the maintainer to fix daemontools-encore? Make my fork a real fork? Give up and add proper setsid support to another daemontools derivative like runit?

                                                                1. 1

                                                                  Thank you for all your answers! Can you comment on the -P flag in runsvdir? Does that not do what you want?

                                                                  1. 4

                                                                    There are several problems with multiprocess services in runit.

                                                                    1. As mentioned above, some services should not use setsid, although most properly written services should. But runsvdir -P is global.

                                                                    2. If you use runsvdir -P, then sv down foo should use process group signalling instead of parent process signalling, or you can still create orphans. As another example, sv stop foo should send SIGSTOP to all processes in the process group, but since it doesn’t, everyone but the parent process continues to run (ouch!). Unfortunately runit entirely lacks facilities for process group signalling.

                                                                    In my patched daemontools-encore:

                                                                    • svc -=X foo signals the parent process only
                                                                    • svc -+X foo signals the entire process group
                                                                    • svc -X foo does one or the other depending on whether you’ve marked the service as multiprocess with a ./setsid file

                                                                    But generally you just use the standard svc -X foo, because it does the right thing.

                                                                    Besides the things mentioned above, runsvdir -P introduces some fresh havoc in other settings. Try this in a foreground terminal:

                                                                    mkdir -p ./service/foo
                                                                    printf '#!/bin/sh\nfind / | wc -l\n' > ./service/foo/run
                                                                    chmod +x ./service/foo/run
                                                                    runsvdir -P ./service
                                                                    ps ax | grep find
                                                                    ps ax | grep wc

                                                                    The find / | wc -l is still running, even though you ^C’ed the whole thing! What happened? Well, things like ^C and ^Z result in signals being sent to the terminal’s foreground process group. Your service is running in a new, separate process group, so it gets spun off as an orphan. The only good way to handle this is for the supervisor to trap and relay SIGINT and SIGHUP to the process groups underneath it.

                                                                    To those wondering who runs a supervisor in a foreground terminal as non-root…me! All the time. The fact that daemontools derivatives let you supervise processes directly, without all that running-as-root action-at-a-distance system machinery, is one of their huge selling points.

                                                                    1. 2

                                                                      Dinit already used setsid, today I made it signal service process groups instead of just the main process. However when run as a foreground process - which btw Dinit fully supports, that’s how I test it usually - you can specify that individual services run “on console” and they won’t get setsid()‘d in this case. I’m curious though as to how running anything in a new session (with setsid) actually causes anything to break? Unless a service somehow tries to do something to the console directly it shouldn’t matter at all, right?

                                                                      1. 2

                                                                        I’m curious though as to how running anything in a new session (with setsid) actually causes anything to break? Unless a service somehow tries to do something to the console directly it shouldn’t matter at all, right?

                                                                        The problems are outlined above. ^C, ^Z etc get sent to the tty’s foreground process group. If the supervisor is running foreground with services under it in separate process groups, they will continue running on ^C and ^Z. In order to get the behavior users expect – total exit and total stop of everything in the process tree, respectively – you need to catch SIGINT, SIGTSTP, and SIGCONT in the foreground process group and relay them to the service process groups. Here’s what the patch to add that behavior to daemontools-encore looked like.

                                                                      2. 2

                                                                        Thanks for the info! I still have a few questions, correlated to your numbered points:

                                                                        1. Like nginx yes? How does a pid 1 handle nginx differently / what makes a pid 1 different? If 99% of stuff needs the process group signaled, but nginx works with pid 1 supervisors, do they not signal the process group? How does all that work? And how does all of this tie in to using runit as a pid 1? Would the problems you have with it not exist for people using it as a pid 1? Because the original discussion was about alternate init systems, which is how I use it.

                                                                        2. This would only create orphans if the child process ignores sighup right? Obviously that’s still a big problem, but am I correctly understanding that? And when runsvdir gets sighup it then correctly forwards sigterm to its children yes? Not as easy as ^C but still possible. Would any of this behavior be different if you were running as root, but still not as pid 1?

                                                        1. 2

                                                          I love this, and can’t wait to join in once I have more time!

                                                          1. 4

                                                            If you have python installed, I like glances for a system monitoring program. I leave a copy running all the time, even on my X system at home.

                                                            1. 3

                                                              I could never get into glances, just because it seems like such a resource hog compared to top.

                                                            1. 1

                                                              FILT/ESFP (that said, I really hate MBTI ;) )

                                                              1. 2

                                                                I really wish this supported UTF-8, because it’d be great to type out ?‍?‍?‍? in binary

                                                                1. 1

                                                                  Soooo, h2n but for the cloud?

                                                                  1. 5

                                                                    I love this. I’ve heard a ton of startup bros talk about containers as if they’re some brand new panacea that Linux crafted from the ether, without actually knowing any of the history behind them. This was a very satisfying read.

                                                                    1. 6

                                                                      I’ve spent some time elucidating on the history to those smitten with the container “gold rush” and it’s made exactly zero impact – not positive or negative – just zero change. it has not helped those I’ve spoken with understand anything more clearly, or approach their usage (planned or existing) of containers differently…literally no impact.

                                                                      the hype is very real. very strong. and very, very vague.

                                                                      and it’s here to stay, unfortunately.

                                                                    1. 11

                                                                      The downside to this is that the reason emacs is awesome it because it’s essentially a lisp VM. In vim land you have vimscript. They don’t even compare

                                                                      1. 6

                                                                        And emacs can show images inline. This is e.g. used to preview LaTeX equations in org-mode, AUCTeX, etc.

                                                                        You can even use emacs + org-mode as a Jupyter-like notebook, with executable code blocks where matplotlib outputs are shown inline, etc. As a bonus, you have git manageable plaintext rather than a big JSON blob (as in Jupyter).

                                                                        1. 1

                                                                          Agreed. It is worth point out that they are using some neovim-specific plugins, which should help things since they can be written in python, lua, etc.

                                                                        1. 20

                                                                          Nice cheat sheet (albeit from 2011) - some of those I wasn’t even aware of, like ss replacing netstat and arp being deprecated.

                                                                          I do find things like this annoying though - I am all for progress, but if net-tools is unmaintained, why create a new tool with a completely different CLI? Rewrite ifconfig if you must, but do it in a way that maintains compatibility. The OpenBSD ifconfig is often getting tweaked, but it’s still ifconfig!

                                                                          Or am I just getting old? (PS: get off my lawn!).

                                                                          1. 9

                                                                            The iproute (and subsequently iproute2) design goal was just to expose the Linux kernel networking facilities through a CLI with as close to a 1-to-1 mapping as possible, so sysadmins could use all the available features from scripts.

                                                                            It’s harder to find clear documentation on this part, but it doesn’t seem to have originally been intended as a replacement for the traditional tools. The expectation seems to have been that, more like you suggest, they would keep being maintained too, providing a more familiar, non-1-to-1 mapping to kernel facilities. But maintenance fell off as most people interested in keeping up with kernel networking just used and contributed to iproute/iproute2 (which was also easier to do, since it was just a thin wrapper around kernel functionality), so in 2009 the net-tools maintainers declared bankruptcy, so to speak.

                                                                            1. 6

                                                                              as close to a 1-to-1 mapping as possible, so sysadmins could use all the available features from scripts.

                                                                              What gets me is that at every stage (ifconfig, ip, etc) wifi has always been in a different tool..

                                                                              1. 3

                                                                                There’s a nice summary of the deprecation over on ServerFault. Of course, some of the comments are just plain wrong, but the references make for interesting reading.

                                                                              2. 1

                                                                                tbh this is a good example of why I tend to prefer BSDs, as well as just why I prefer cathedral-style development over the mess of the bazaar. I don’t like the pattern (specifically in Linux) of constantly replacing old tools with entirely new tools that cover slightly different use-cases of the old ones, while usually missing some functionality.

                                                                              1. 3

                                                                                This looks pretty interesting! I’ve been considering switching to Evil Mode, but maybe some of this material will convince me to stay for a bit longer :P

                                                                                I do wonder about the usefulness of these sorts of projects overall: I’d argue that most developers would prefer to build their dev environments from the ground up, rather than inheriting a whole slew of keycommands they haven’t defined themselves (also, I like to avoid language-specific plugins for languages I do not use :P)

                                                                                That being said, this looks very well thought out :)

                                                                                1. 2

                                                                                  Lazy loading gets rid of most of the issues around language specific plugins, but I guess what will make or break this is how well integrated the plugins are and how heavy this all ends up being. I like spacemacs, but it’s a little clunky. I’ll try this out when I have the time.

                                                                                  1. 1

                                                                                    A lot of this went away for me when I started using emacs in daemon mode. Granted, it’s a pretty silly suggestion to run your text editor in a daemon. :)

                                                                                  2. 2

                                                                                    I don’t think of spacemacs as an evironment as muich as I do a separate app that runs on the emacs vm

                                                                                    1. 5

                                                                                      Did you ever play a “total conversion” mod for a game like Quake or Unreal or Half-Life? Usually the experience you get is mostly seamless and all new but then occasionally a little bit of the underlying game that it was based on pokes through and is visible - say a stray texture or something that was reused from the base game.

                                                                                      That’s kind of how using Spacemacs feels relative to Emacs. :)

                                                                                    2. 1

                                                                                      I’ve been using vimrc for a while (because I lost my original setup) and I keep forgetting combos for things like opening NERDTree, etc… I think because I didn’t set them up myself, it didn’t sort of burn the keystrokes into memory. vimrc has too many things in it for me. This looks a bit more lean by comparison so I might actually try this out to put off setting up all my own again.

                                                                                      Edited: meant vimrc not vimawesome

                                                                                    1. 24

                                                                                      I’ll be honest, I hate the prevalence of Electron apps now. Bundling a stripped down chromium + a webapp is way overkill for a lot of the apps that take advantage of this approach, and it also ends up bringing a lot of the garbage tech and web bugs along with it.

                                                                                      I understand that it’s great for web developers to reuse their existing skills on desktop apps, but at the same time, it hurts users.

                                                                                      1. 2

                                                                                        Where do you encounter such apps? I have no such app installed and the only one I know about is atom/vs code. What else is there?

                                                                                        1. 11

                                                                                          Another popular Electron app is the Slack desktop client.

                                                                                          I can only agree with @worr’s sentiment - I’ve not used VS Code but I tried Atom for a few weeks last year and even on a fast 4C/8T i7 desktop with 16GB RAM and SSD it was horrible. It ate RAM like there was no tomorrow and the whole thing was laggy and sluggish. The Slack desktop client is much the same - it can quite easily eat 20% CPU doing absolutely nothing.

                                                                                          A common Emacs joke used to be that the name stood for “Eight Megs and Constantly Swapping” (referring to its high resource requirements). How long until some wag comes up with something similar for Electron…

                                                                                          1. 3

                                                                                            A Terabyte Of Memory?

                                                                                            1. 2

                                                                                              ah yes, Slack. They force me to use that as well. Totally forgot about that beast.

                                                                                              1. 1

                                                                                                VSCode runs fine on my ~2 year old laptop. Is currently consuming ~617mb of ram in total between all the helpers. VSCode is much better about resources than atom.

                                                                                              2. 3

                                                                                                Full(ish) list of apps here: https://electron.atom.io/apps/

                                                                                                There is potential for Signal Desktop to join the ranks as well.. Hopefully they come up with a better solution (I am not holding my breath).

                                                                                                1. 1

                                                                                                  I wish this was satire, but it isn’t https://github.com/Meadowcottage/gitmoji

                                                                                                2. 2

                                                                                                  For me, it was Slack, Brave and Discord. I had heard similar issues from friends that used VS Code and Atom, and that’s when the pattern emerged for me

                                                                                              1. 6

                                                                                                imo, this is not as cool as it sounds. This was added by an external contributor, the Go team likely doesn’t care nor will use it in any of the official go tools. I’m guessing that someone added it for their own project, which, while super cool for their project, isn’t the same as Golang signaling support for this mechanism.

                                                                                                That said, Go’s typical concurrency model does not work well with pledge. All of the concurrency primitives that are typically used to separate concerns where you’d want to drop privileges with pledge operate on goroutines, not processes. It’s definitely possible to write a privsepped + pledge daemon in Golang, but it’s not easy nor is it likely to become a common practice.

                                                                                                1. 20

                                                                                                  This was added by an external contributor, the Go team likely doesn’t care nor will use it in any of the official go tools.

                                                                                                  It’s a native interface to a system call, how did you interpret that to mean that the Go team is (or isn’t) going to switch everything to OpenBSD just to be able to use pledge?

                                                                                                  This was added by an external contributor, the Go team likely doesn’t care nor will use it in any of the official go tools. I’m guessing that someone added it for their own project, which, while super cool for their project, isn’t the same as Golang signaling support for this mechanism.

                                                                                                  Someone created a 3rd party package that only did pledge, Brad Fitzpatrick (a core Go team member) requested that it be upstreamed and so it was submitted as a formal addition to the core unix package. It stalled out, I asked what the status was, and Brad did the work to clean it up and then commit it.

                                                                                                  1. 5

                                                                                                    My assertion is that this change isn’t indicative of any intent for any type of wider adoption of privsep/privdrop practices by any of the golang core. I guess I don’t really understand the excitement behind this merge.

                                                                                                    1. 7

                                                                                                      You are right. This change is not indicative of anything. In particular, it’s not in the standard library, so the Go project can’t use it. It’s in the x/sys/unix repository.

                                                                                                      x/sys/unix is just a dumping ground for all possible system calls so that Go users can use it, if they need it. If the standard library needs it, it goes into syscall not x/sys/unix.

                                                                                                    2. 1

                                                                                                      If you have ideas for places this would be useful in libraries, running binaries or in the standard library, I can try to submit some OpenBSD-specific patches to run pledge before making a system call.

                                                                                                      1. 5

                                                                                                        It’s not possible to use it in the standard library because it’s not in the standard library. It’s in a second-class repository maintained by the Go project.

                                                                                                        Of course we could add it to the standard library if we had a specific need for it, but it’s a hard sell.

                                                                                                    3. 1

                                                                                                      Its mostly as easy as C, especially given there is a built in rpc package that works decently.