1. 65
  1.  

  2. 27

    And yes, a large part of this may be that I no longer feel like I can trust “init” to do the sane thing. You all presumably know why.

    No, I don’t. Please explain what you mean.

    Such appeal to everyone “knowing what you mean” and the implication that everyone supports your standpoint are toxic. They are a good way of making a personal opinion look like a group opinion. Combined with a fuzzy notion of “sane”, this is basically just spreading bile.

    And it works. If this mail were more complex or that sentence would be missing, it probably wouldn’t be here on lobsters. It’s certainly not posted for the review above it.

    1. 85

      If you’re reading the kernel mailing list, it can be assumed that you have some familiarity with the subject matter, and if not, you’re not missing anything crucial to the discussion here. Torvalds has decided not to point and name the implied party, probably to avoid another heated flame war on the mailing list.

      Some context to get you up to speed:

      He is referring to what is currently the most popular init system on Linux, systemd. systemd is a relatively recent development of Red Hat, and has been adopted by all major distributions. Prior to systemd gaining popularity, the init system was a hodgepodge of shell scripts, which clearly had its share of problems.

      However, systemd has been adding more features to its resume. Besides just being an init system, it has also absorbed the hardware abstraction layer udev, it implements its own dbus daemon (a popular Linux message bus used to communicate between different services and programs), it has taken control of some power management features such as suspend on lid close for laptops, it implements login and virtual terminal handling, it contains a dhcp client and server, and it provides its own system logger using a binary format that can in practice only be used through the tools provided by systemd.

      This attitude of trying to do everything from a single piece of software has proven to be somewhat controversial among Linux users, because the old UNIX mantra for software was “do one thing and do it well”. This is especially controversial for something as central as the init system, because it is always running and runs with elevated rights.

      The lead developer of systemd has also responded to a few issues with some unpopular comments, and has in the past been in conflict with the Linux kernel developers by refusing to cooperate on certain issues caused by Linux and systemd interaction. systemd has also, despite its widespread use, been hit by a number of fairly serious bugs, some of which had significant security impacts. The simplicity and potential impact of some of these bugs has left many people in doubt over the general quality of systemd and related projects.

      1. 18

        In particular Linus has had specific issues in the past - there was a problem a while back where kernel developers would boot with the “debug” flag and systemd would start spamming the console with messages and drown out the kernel information said developers needed. See https://lkml.org/lkml/2014/4/2/580 where someone proposed a patch that would remove “debug” from /process/cmdline so that the presence of that flag was completely unavailable to userspace (including systemd), thus literally preventing the problem from happening. Real icky situation.

        1. 9

          Ha, while this isn’t quite what I called for, it is a great explanation of the state of things. “Controversial”, I certainly agree with (although, coming to systemd, I’m on the “It fixes a lot of things for me” side of things).

          Thanks for that.

          1. 14

            There seems to be two main camps of users (3 if you include distro maintainers as a separate camp):

            1. People who maintain a few systems, perhaps use Linux on the desktop.

            2. People who maintain many systems.

            Camp 1 people don’t mind when Systemd does something arbitrary, unexpected or indeterminate. Camp 2 people hate Systemd’s indeterminism.

            Personally I hate magic. Systemd is magic. I can’t trust it to do what I want to do, only what it wants to do.

            1. 9

              As a person who maintains many systems professionally, i have to interject here since I always see it stated as fact that professional operators dislike systemd. I like systemd a lot because it gives badly needed structure to Linux service management. Most colleagues who worked with systemd feel the same. (This doesn’t mean it’s perfect or bug-free)

              1. 4

                Is there a reason you didn’t deploy daemontools or runit or some such to give badly needed structure to Linux service management before systemd forced it on you (however willingly)?

                1. 5

                  I did use those at various times but that’s not the same as being the default that manages all services on the system. Systemd also has a powerful declarative configuration the other options did not.

              2. 2

                There seems to be two main camps of users (3 if you include distro maintainers as a separate camp):

                1. People who maintain a few systems, perhaps use Linux on the desktop. Camp 1 people don’t mind when Systemd does something arbitrary, unexpected or indeterminate.

                I’m a group 1 member but I absolutely hate when systemd doesn’t operate as I would expect a init daemon.

            2. 4

              This attitude of trying to do everything from a single piece of software has proven to be somewhat controversial among Linux users, because the old UNIX mantra for software was “do one thing and do it well”.

              I’ve found this mantra to be only applicable in certain situations, usually when it comes to applications that users directly interact with. Things like email clients, text editors, and IRC clients (web browsers could spawn an entire discussion on this all their own). I’m not an expert on init systems, and your previous paragraph on systemd clearly shows its feature creep. But when it comes to an init system, I’ve always seen that as a complex process where it’s necessary for it to do more than one thing. This can be especially true with modern systems where everything you’ve mentioned (HAL, dbus, power management, login, networking, etc) being (arguably) necessary for the system to run correctly and in a useful way.

              So, I wonder, is it possible to have an init system that is:

              • fast
              • effective
              • correct

              that still abides by “doing one thing, and doing it well”?

              1. 8

                The mantra predates text editors (well aside of ‘ed’), email and irc clients. Any user interfaces in the modern sense really. It meant using a bunch of small, single purpose programs (like cat, troff, tail, ps…) which could be combined by user to the desired effect with standard system mechanisms like redirection, pipes and shell scripts.

                We can argue the practical merits of systemd forever but it’s fairly clear it goes against the tradition of UNIX systems development. It’s a huge, opaque, uncooperative beast that makes turtles cry. I hope Linus is close to the point where he’ll just come up with something more digestible.

                1. 7

                  The original idea behind that mantra was to make tools that were:

                  1. simple (Note: I’m using the Rich Hickey definition here)
                  2. composable. This composability requirement is why all the command line tools ingest and output strings.

                  As systemd takes over more of a Linux machine, they destroy their own simplicity, requiring someone to keep a massive amount of state in their head to modify the code or even work on units as an administrator. However, they also destroy the composiblity of the system’s tools. Things like binary logs and internal-to-systemd protocols can’t be parsed by standard command line tools, and thus users lose this ability to compose different parts of the system. This has been my biggest issue with systemd, that it violates not only “do one thing and do it well”, but also the composiblity that makes that possible.

                  A side note on GUI’s: The GUI design model is specifically the opposite of “do one thing and do it well”. GUI’s are not designed for composibility, they are designed to take the user from one end of a specific process to the other. They trade off the ability to compose with other programs, for a more robust control of the user experience.

                  1. 1

                    What’s wrong with piping journalctl to things? Legitimately curious.

                2. 1

                  The context of this discussion is around trying to bring sanity to rlimits for setuid processes…

                  In an attempt to provide sensible rlimit defaults for setuid execs, this inherits the namespace’s init rlimits:

                  $ ulimit -s 8192 $ ulimit -s unlimited $ /bin/sh -c ‘ulimit -s’ unlimited $ sudo /bin/sh -c ‘ulimit -s’ 8192

                  This is modified from Brad Spengler/PaX Team’s hard-coded setuid exec stack rlimit (8MB) in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don’t reflect the original grsecurity/PaX code.

                  Certainly traditionally it has been trivially easy for a rogue daemon to bring a system to it’s knees…. since traditionally, out of the box, there are no rlimits imposed.

                  It is the init systems job to start daemons… it would be really nice if it imposes sane rlimits on anything it starts.

                  Systemd does that, and attempts to do it in sanish ways by imposing the limits on process groups. (ie. A rogue daemon cannot escape it’s constraints by spawning a legion).

                  I’m would be easily convinced that systemd’s approach is not the best and/or not correct.

                  However I’m certain that the linux ecosystem needs work in this area and systemd is at least undertaking that work.

                  1. 2

                    You seem to be mixing up system daemons and setuid utilities. This patch has nothing do with the limits systemd imposes on the processes it starts, so whatever systemd does, or other systems did not do, in this area is irrelevant.

                    There’s no question that systemd is capable of setting rlimits for child processes. The question is whether the limits systemd sets for itself are a good default template for setuid processes run by users.

                  2. 1

                    I figured it was systemd, and I know there’s a TON of systemd hate floating around, but I didn’t realize just how rampant the freeping creaturism had become.

                    That’s really unfortunate, thanks for the clarifying comment.

                    1. 4
                  3. [Comment removed by author]

                    1. 8

                      What qualifies as “insane”?

                      This might sound trollish, but I seriously don’t find “sane/insane” distinctions very useful (beyond their connotations). In my experience, these words get used when you can’t construct a better argument for why the other side is doing something wrong, so you call yours “sane” and the others “insane”.

                      1. [Comment removed by author]

                        1. 8

                          That sounds like a nice definition, but “without reason or logic” is just as fuzzy, “counter productive, destructive, harmful” are also easily stated, but must be followed by hard facts to be held up.

                          Also, regarding the principle of least surprise: Matz (who popularised it) also famously said that it applies to his surprise.

                          Your definition just moves the playing field.

                          Also, I would argue that anyone implementing a piece of software so central to the Linux world is a “domain expert”. This is boundary play at its finest.

                          1. 19

                            For a specific example: How would you classify the change to kill tmux servers after a user logged out? Lots of people found that surprising. And in the larger space of existing init systems, quite unprecedented. I think “insane” is lacking precision, but adequately captures many people’s sentiment.

                            1. 10

                              and to add to this response, I find it insane that the response was asking tmux to include a change for the new behavior systemd enforced.

                              1. 2

                                Especially since it changes how every unix has behaved for almost 30-40 years regarding HUP.

                                Or a more recent one where it parses a username of “0haha” as being an invalid username and runs the unit as root. And now perfectly valid usernames starting with a number won’t work in systemd unit files as they get interpreted as being invalid because systemd can’t seemingly parse numbers in a config file sanely to distinguish a user name versus a user id.

                                This all might sound like splitting hairs, but breaking userspace (HUP behavior shouldn’t need a patch for your init in tools), and not parsing a username sanely are pretty basic things I would expect a first year undergrad to be able to do.

                                So yes, I agree insane is a good word to use for things. I could come up with hard facts, but systemd really feels like one step forward and two steps back for a lot of things. I don’t really feel like its a very good example of good engineering practices, aka binary logs that can be corrupted forcing you to do insane things to get a system online due to a short write to the filesystem is… also insane, we have decades of knowledge of how to do this that has been ignored.

                                If the corners aren’t rounded on this desk, why should I feel safe about the rest of the desk?

                            2. 2

                              Boundary play? The guy who wrote Linux made an opinion about it being so.

                          2. 13

                            In my experience, these words get used when you can’t construct a better argument for why the other side is doing something wrong, so you call yours “sane” and the others “insane”

                            “In your experience”, huh? So you’re just extrapolating your own personal experiences to all of mankind, then? “Citation needed!”

                            You see, it’s easy to filibuster any conversation by calling for better argumentation, proof, evidence, studies to back claims up and so on.

                            You know perfectly well why someone might call systemd “insane”. What’s your actual contribution to the conversation, besides signalling to everyone what a rational and sophisticated person you are?

                            1. 10

                              “In your experience”, huh? So you’re just extrapolating your own personal experiences to all of mankind, then? “Citation needed!”

                              No, I don’t. That’s why I wrote: In my experience.

                              You know perfectly well why someone might call systemd “insane”.

                              No, I don’t. I use systemd every day and I’m very fine with how it works and how it behaves.

                              It has, as all implementations of a thing, issues and flaws, but that’s all. I’d be happy to try an alternative, which would for sure improve in a lot of areas (and may be worse in others), but that’s a trade-off, nothing more.

                              What’s your actual contribution to the conversation, besides signalling to everyone what a rational and sophisticated person you are?

                              I’m highlighting a conversational pattern that is all too often used to create unity were there is none. I’m neither rational or sophisticated.

                              1. -1

                                Not to put too much effort into pointless bickering, but:

                                No, I don’t. That’s why I wrote: In my experience.

                                Here’s what you said:

                                In my experience, these words get used when you can’t construct a better argument for why the other side is doing something wrong, so you call yours “sane” and the others “insane”

                                You’re generalizing based on your experience. In other words, you’re not just talking about your personal experiences themselves - you’re suggesting that you’ve observed a general trend, and that it applies to the guy you responded to.

                                For whatever it’s worth, I suppose you’re technically correct about what you said. Pointing it out is just not particularly valuable, because discussions don’t go anywhere if people insist on being super rigorous about them.

                                In fact, much like you pointed out that “insane” is often used as a substitute for an argument, evidence is usually demanded as a way of seemingly proving someone wrong, or perhaps more accurately, just as a way of silencing someone.

                                It’s like everyone assumes that if no academic institution has produced a paper that says X, then X is not true.

                                We can all generally see what someone is saying, even if his post isn’t 100% logically sound in every conceivable way, and accompanied by scientific research to back up his claims.

                      2. 3

                        Break up the banks systemd!