1. 32
  1.  

  2. 47

    As with many systemd bugs, its some combination of silly and stupid and amusing, but the real story, as ever, is king Lennart’s decree that these previously valid usernames are now invalid. And thus it shall be!

    1. 34

      This really upsets me. Red Hat, through Poettering, is pushing all kinds of pretty invasive infrastructural changes in the Linux ecosystem. Some of these things are good (like declarative service definitions).

      But the quality of these services that end up in supposedly robust “enterprise” Linux distributions is just astonishingly bad. I work in a team that develops a software package for CentOS / RedHat. During the lifetime of RedHat 7.x, we’ve run into:

      1. A systemd bug that allows any local user to take down the box (https://github.com/systemd/systemd/issues/4234)
      2. A firewalld race condition that causes the network to be completely locked down at startup or whenever the firewall service is restarted. This means that I need to find a KVM connection to all the rack units that ended up being unreachable. I am responsible for the firewalld service definitions for our team and I was unable to test this because I would invariably end up with an unreachable machine when the regression tests ran. This was reported in 7.0 and not fixed until 7.3.
      3. A systemd bug that leaks sessions, causing any interaction with the systemd dbus service (used for starting and stopping services as well as checking service status) to take minutes (https://github.com/systemd/systemd/issues/1961)
      4. A bug in abrtd (Red Hat’s core dump processing tool) that fails to configure the kernel’s core_pattern when (re)starting, resulting in missing core dumps.
      5. systemd hijacking kernel command line arguments making it impossible to debug the Linux kernel (https://bugs.freedesktop.org/show_bug.cgi?id=76935). We ship modified version of specific third-party proprietary drivers with our product, so this kind of stuff also affects us.

      This, combined with Poettering’s general attitude of “it’s not me, it’s you” whenever a bug hits (the world should adapt to systemd, because systemd will not adapt to the world) is driving me insane. On top of this toxic attitude, systemd also keeps on tacking on other features to what should just have been an init daemon (init system, system logger, dbus server, power manager, login manager, VT handler, dhcp client/server, hardware abstraction & initialization layer …) meaning that these symptoms are spreading wider and wider within the Linux ecosystem. I don’t understand why such anti-social developers were able to push such low-quality software through a conservative enterprise distribution such as Red Hat, and why other distributions went along with this.

      It’s really a stark contrast with the much more thought out development of something like OpenBSD, where most of the bigger changes strike me as non-invasive good ideas. It’s a shame that a lot of things don’t exist on OpenBSD (like, in my case, OpenCL drivers for Intel graphics), or I would probably just switch my development environment over and be rid of the madness.

      1. 14

        RedHat employs most of the developers for various projects e.g. Gnome. So those projects hard-depend on systemd and this forces adoption in other distributions.

        I switched to FreeBSD a few years ago and am very glad to have done so.

        1. 5

          This is a valid point. Same with all this udev & systemd integration forcing people using udev to adopt systemd.

          1. 7

            udev, d-bus, libinput, logind, polkit, gtk/gnome all work together to slowly force you into accepting the times and participating in this incestuous interdependency orgy :(

        2. 9

          I don’t think these things are simply pushed on to the ecosystem. There might be some push, but ultimately the community is accepting these ideas. The problem lies in the “democratic” decision making process where people argue over technical merit.

          It is easy to point at something old, some concrete problem with it, and then present something new and show that it solves that problem (and maybe another problem). It is a very concrete argument to make.

          Some interesting counters would be that the new thing is too big, too complex, or poorly designed. Size is a concrete thing, but it sounds hand-wavy and it is easy to brush off with another equally hand-wavy: the software is solving a complex problem and therefore needs to be complex, and therefore it also becomes big. It doesn’t help that concern about size can be seen more as an expression of personal belief (“I prefer minimalism”) that has little weight in face of concrete features. And just so, arguing about the bugs that size & complexity inherently bring along can be brushed and hand-waved away. Someone wants or needs these features and developers will provide; and bugs will be eventually fixed.

          If you have a hunch that a simpler, smaller, cleaner solution could be made, the onus is on you to provide it. Otherwise you’re not contributing, just complaining. If you propose to hold on to the old thing to wait and see if another viable solution would emerge with time, you’re holding back progress. You’re a caveman, and again you’re not contributing.

          Now arguing that some software is poorly designed is even harder. It is an abstract argument to make, and unless you can articulate the design flaws to make the argument concrete, you’re just hand-waving. But making the argument concrete is very difficult when there is no universal image of “good design”. It changes with time and place, and it seems to be as much religion as your choice of text editor or programming paradigm is. By contrast, it is trivial to point out a feature that software A has and B doesn’t have.

          Then there’s the fact that most of these decision makers are typical software developers. Software exists to solve problems. Features solve problems, and you solve more problems by adding more features – adding more code. More features, more code is good. Complexity is inherent, and something you’ll have to live with. That is the natural state of being for the typical software developer. The people at large want features, and if some thing already provides these features, nobody has the right to stop and wonder if we couldn’t solve problems differently.

          I did not pay attention to what happened on e.g. Debian’s mailing lists when they decided to adopt systemd. But I doubt the most pressing reason was Gnome, or a conspiracy of Gnome developers pushing it. No, it is most likely that they evaluated their choices, and systemd won on “technical merit.” They chose the best solution.

          In such a “merit” focused democratic process, making the arguments about size and complexity is an uphill battle. All the odds are against you, unless you’re operating in some fringe self-selecting community where personal values coincide in favour of what you’re arguing for (suckless people, various minimalistic linux distros, etc).

          By contrast, when you have a powerful leader, he does not have to make that losing argument. So Theo can say “I don’t like it.” And that might as well be the end of discussion until someone comes up with another solution. There is no need for him to spend the next three weeks defending these arguments against “merit” in a shitstorm tier nerd fight. He also doesn’t need to offer a “technically superior” solution to whatever problem happens to be troubling one of these contestants. No action is valid action.

          If you want good software, you need a strong leader with good taste. Or a community with clear values. Unfortunately values have the tendancy to erode over time as the community grows, and trying to enforce them turns into politics and drama, with arguments that will look very much like the arguments over technical merit. A self-selecting community that naturally stays small might work.

          1. 1

            If you propose to hold on to the old thing to wait and see if another viable solution would emerge with time, you’re holding back progress.

            SysVinit was siting there for years waiting to be disrupted at least since I started to use Linux in 2001. How long should we wait for this new system that is all better. Is 15 years not enough? Can we expect a decent replacement before 2038? Maybe the generation of our grandchildren can come up with something.

            1. 10

              There were a bunch of other init systems that offered all the advantages of systemd with less of the downsides. Unfortunately since they were good citizens that used standard mechanisms and didn’t get applications hard-depending on them, distributions weren’t forced to adopt them and largely didn’t.

              1. 5

                Are these rhethorical questions? If not, they are misdirected. Whatever OS you’re using, I’m not the leader of that project. Alternatives to sysvinit have been around forever, so when “we” (that does not include me) decide to do something about sysvinit is up to each project that still lives with it. Of course you can look for another project if you’re not happy with the decisions they make.

                If that’s a hypothetical question that puts me in the dictator’s role, I would say that you wait as long as it takes. If you have a pressing need, you can help yourself. But you won’t drag my project into accepting a solution I find repulsive. Not in a year, not in 20 years.

                Of course, my project today would not be using sysvinit, just as I am not using it today.

                1. 4

                  SysVinit was siting there for years waiting to be disrupted at least since I started to use Linux in 2001.

                  OpenRC has been available since 2007: https://en.wikipedia.org/wiki/OpenRC

                  1. 2

                    Reading the linked article:

                    Since 0.25 OpenRC includes openrc-init, which can replace /sbin/init

                    Which was according to the 0.25 tag 17th of April 2017. And the only distribution I know using it was Gentoo which is not surprising, since they invented it.

                    1. 5

                      OpenRC used the regular SysV init as its /sbin/init. Gentoo’s been using it for many years (at least since 2012, but I can’t find documentation of its initial inclusion), and Debian includes it.

              2. 3

                It’s a shame that a lot of things don’t exist on OpenBSD (like, in my case, OpenCL drivers for Intel graphics), or I would probably just switch my development environment over and be rid of the madness.

                No need to go that far. Use Gentoo with OpenRC and stay Poettering-free and sane.

                1. 2

                  What would’ve made this thing more palatable to me at least (from a perspective of not having written an init system in decades) would’ve been a split into:

                  1. a deterministic even if slow, init(1) then script runner - even ‘make’ would work :-)
                  2. a service manager independent dependency/config format, unit files as they are would suffice I guess
                  3. systemd as a service manager, akin to good ol’ daemontools. Since it’s already “linux is the only thing that exists” they can get away with the prctl+subreaper system to catch services that tries to escape via double-fork

                  but that would also remove much of the leverage they are using to push inane shit and rediscover old vulnerabilities.

                  1. 3

                    a deterministic even if slow, init(1) then script runner - even ‘make’ would work :-)

                    That’s…actually a fascinating idea.

                    Could you perhaps use a script to create an artifact in a, say, /var/startup_status directory that make checks for to resolve boot dependencies?

                    EDIT: In #lobsters we came up with maybe using shell scripts w/ traps to handle signals for init. This might be doable. I’ll upvote any submissions that tries. :)

                    1. 2

                      It does fit with the OpenBSD narrative of relinking kernel at boot for ASLR as well :D the more interesting part about the ‘make’ bit is the choice between parallelism and repeatable evaluation order: just -j it. If it works with -j0 but not without, you likely have a race.

                  2. 2

                    firewalld isn’t part of systemd

                    1. 2

                      -1 incorrect

                      really?

                      1. 3

                        I got one of those too, although maybe I deserved it. It seems to defeat the whole purpose of explaining downvotes: a Dalek battle cry is no improvements on mystery downvotes.

                        1. 0

                          Sigh, an objective, technical community, free from fanboysm, my ass.

                2. 27

                  the problem about this bug is not that it exists, but what it says about the creators of systemd and their behaviour when bugs are found:

                  a) why gets this parsed as number in the first place? is really only the first byte checked and if it looks numeric then its a number? are there no sane .ini style parsers which can do that, or did the systemd folks just want to write their own?

                  b) the handling of this bug is an example of lennarts social skills and technical expertise: “works for me, you are dumb”.

                  and that’s the perspective imho.

                  1. 11

                    It also speaks volumes about the complexity in systemd as shit like this creeps in more often than in certain competition.

                    1. 1

                      It also speaks volumes about the complexity in systemd as shit like this creeps in more often than in certain competition.

                      It could also just be speaking volumes about the sizes of the install bases for systemd and competing “modern” init systems.

                      1. 1

                        so windows bugs also speak primarily of the size of the install base than of the quality of code?

                  2. 15

                    Here’s the deal folks: systemd is software and it has bugs

                    Sure, bugs happen, but I think the beat up is more because of:

                    poettering closed this
                    poettering added the not-a-bug label

                    It’s not the crime, it’s the cover-up.

                    1. [Comment removed by author]

                      1. 10

                        It’s a response to some of the wilder comments in https://github.com/systemd/systemd/issues/6237

                        Pottering is never wondrously conciliatory and diplomatic, and his responses tend to wonder off the direct topic and case in point….

                        poettering commented 4 days ago

                        Yes, as you found out “0day” is not a valid username. I wonder which tool permitted you to create it in the first place. Note that not permitting numeric first characters is done on purpose: to avoid ambiguities between numeric UID and textual user names.

                        systemd will validate all configuration data you drop at it, making it hard to generate invalid configuration. Hence, yes, it’s a feature that we don’t permit invalid user names, and I’d consider it a limitation of xinetd that it doesn’t refuse an invalid username.

                        So, yeah, I don’t think there’s anything to fix in systemd here. I understand this is annoying, but still: the username is clearly not valid.

                        I hope that makes sense?

                        but then various commentators hype the issue beyond reason….

                        julian-klode commented a day ago

                        This is a severe security issue that should be fixed, and assigned a CVE, and not not-a-bug.

                        Of course the diversity of the linux ecosystem didn’t help Pottering here.. it is indeed not a valid username on the tool he used and tested, and cannot even be created……

                        …however using other distros and tools you can.

                        And then, sadly, things went rapidly downhill from there….

                        mbiebl commented a day ago

                        Sigh, since this now attracting the trolls, I’m locking the conversation.

                        mbiebl locked and limited conversation to collaborators a day ago

                        poettering deleted a comment from jrmithdobbs 18 hours ago

                        poettering deleted a comment from sergeyfrolov 18 hours ago

                        Being a startup service requires a certain measure of flexibility…. what do would you want it to do? Lock up and die? Or log, and take best guess that won’t stop things from working?

                        In this case, I would only fault it for quashing the User= parameter to root instead of failing that particular service.

                        It’s a bug, not a particularly problematic bug.

                        1. 9

                          Hence, yes, it’s a feature that we don’t permit invalid user names,

                          This is the part I take issue with. According to the bug report, systemd was permitting invalid usernames in the config, interpreting ones beginning with numbers as root.

                          If the intent was to disallow those names, the config that specifies them needs to be rejected.

                          1. 4

                            True enough that people jump on any evidence that Systemd is less than perfect and make more of it than it ever really was. I think this is a result of frustration: people don’t like/want Systemd, but it’s forced on them (to some extent - because eg their favourite distribution uses it) and it’s difficult to critique properly because, essentially, a lot of what it does “differently” it does so for a reason. I’m no fan myself but I can why Systemd and why in various ways it’s better than what was available before.

                            In this case, though, the developer attitude to a bug (regardless of how serious it is) is disturbing:

                            systemd will validate all configuration data you drop at it, making it hard to generate invalid configuration. Hence, yes, it’s a feature that we don’t permit invalid user names

                            The first sentence is clearly wrong; it didn’t validate the “0day” username, or it would have rejected it. The second sentence implies that “0day” is an invalid username, but I’m not aware of any official specification of what constitutes a valid username, and clearly some people (and some tools) consider it valid. And the response is not “oh, there’s a case we didn’t think of, we should fix that, thanks”, it’s “you’ve done something wrong, of course it doesn’t work”.

                            1. 3

                              The second sentence implies that “0day” is an invalid username, but I’m not aware of any official specification of what constitutes a valid username, and clearly some people (and some tools) consider it valid.

                              That’s exactly what intrigues me. What are the rules, anyway? A powerpoint [0] says usernames are checked against NAME_REGEX = ^[a-z][-a-z0-9]*$, for some good reasons, but those checks can be overridden.

                              If we look at FreeBSD [1], the manual states The user name is restricted to whatever pw(8) will accept. Generally this means it may contain only lowercase characters or digits but cannot begin with the - character. .

                              So what have I learned? Usernames on Linux can probably be made to start with 0. It’s not a great idea, as the powerpoint explains, but it can be done. You can put vodka in your lawnmower, so to speak. What’s frustrating to me is when I see this from a Make illegal states unrepresentable viewpoint, systemd will consider it illegal to have digit-starting-usernames, but it’s everyone else’s job to make that happen. When the reality of digit-starting-usernames meets the idealism of systemd, the user is blamed for not appreciating the feature. There is much room for improvement here.

                              [0]. http://paulgorman.org/technical/presentations/linux_username_conventions.pdf

                              [1]. https://www.freebsd.org/cgi/man.cgi?query=adduser&sektion=8&apropos=0&manpath=FreeBSD+11.0-RELEASE+and+Ports

                        2. 10

                          I really can’t stand systemd. Absolutely hate it.

                          • I hate the complexity of the thing (what’s it doing at any given time? I don’t know. Systemd is magic)
                          • I hate the sheer scope of the thing (why does pid 1 need a DNS server?)
                          • I hate the toolset needed to manage basic stuff (journalctl and co can go bite me)

                          Most people wouldn’t accept Emacs with a ton of plugins as the default editor on a Linux system, yet we accept a huge, monolithic, scope and feature creeping sprawling mess as our init in most of the major distros.

                          Even worse, this is being led by someone who ignores and tries to cover up attempts to improve the spaghetti.

                          1. 9

                            I hate the sheer scope of the thing (why does pid 1 need a DNS server?)

                            PID1 does not include a dns server, it’s a separate binary.

                            1. 2

                              You are of course, correct. Thanks for pointing that out. I should’ve said, (why does systemd need a DNS server?)

                              1. 2

                                It doesn’t. To the best of my knowledge, systemd works just fine without systemd-resolved. I also wouldn’t characterize systemd-resolved as a “DNS server”, though, so maybe you’re talking about some other component I don’t know of.

                                1. 1

                                  I’d characterise systemd-resolved as a DNS server. You can split hairs over whether or not it needs to serve it’s own records to constitute a full DNS server or not, but it listens on port 53 and serves responses to DNS queries.