Hmm. I have just spent a week or two getting my mind around systemd, so I will add a few comments….
Systemd is a Big step forward on sysv init and even a good step forward on upstart. Please don’t throw the baby out with the bathwater in trying achieve what seems to be mostly political rather than technical aims. ie.
** The degree of parallelism achieved by systemd does very good things to start up times. (Yes, that is a critical parameter, especially in the embedded world)
** Socket activation is very nifty / useful.
** There are a lot of learning that has gone into things like dbus https://lwn.net/Articles/641277/ While there are things I really don’t like about dbus (cough, xml, cough)…. I respect the hard earned experience encoded into it)
** Systemd’s use of cgroups is actually a very very nifty feature in creating rock solid systems, systems that don’t go sluggish because a subsystem is rogue or leaky. (But I think we are all just learning to use it properly)
** The thought and effort around “playing nice” with distro packaging systems via “drop in” directories is valuable. Yup, it adds complication, but packaging is real and you need a solution.
** The thought and complication around generators to aid the transition from sysv to systemd is also vital. Nobody can upgrade tens of thousands of packages in one go.
TL;DR; Systemd actually gives us a lot of very very useful and important stuff. Any competing system with the faintest hope of wide adoption has a pretty high bar to meet.
The biggest sort of “WAT!?” moments for me around systemd is that it creates it’s own entirely new language… that is remarkably weaker even than shell. And occasionally you find yourself explicitly invoking, yuck, shell, to get stuff done.
Personally I would have preferred it to be something like guile with some addons / helper macros.
I actually agree with most of what you’ve said here, Systemd is definitely trying to solve some real problems and I fully acknowledge that. The main problem I have with Systemd is the way it just subsumes so much and it’s pretty much all-or-nothing; combined with that, people do experience real problems with it and I personally believe its design is too complicated, especially for such an essential part of the system. I’ll talk about it a bit more in my blog (along with lots of other things) at some stage, but in general the features you list are good features and I hope to have Dinit support eg socket activation and cgroups (though as an optional rather than mandatory feature). On the other hand I am dead-set that there will never be a dbus-connection in the PID 1 process nor any XML-based protocol, and I’m already thinking about separating the PID 1 process from the service manager, etc.
Please don’t. It is a lot easier to turn machine-readable / binary logs to human-readable than the other way around, and machines will be processing and reading logs a lot more than humans.
Human-readable doesn’t mean freeform. It can be machine-readable too. At my last company, we logged everything as date, KV pairs, and only then freeform text. It had a natural mapping to JSON and protocol buffers after that.
Yeah, you can do that. But then it becomes quite a bit harder to sign, encrypt, or index logs. I still maintain that going binary->human readable is more efficient, and practical, as long as computers do more processing on the logs than humans do.
Mind you, I’m talking about storage. The logs should be reasonably easy for a human to process when emitted, and a mapping to a human-readable format is desirable. When stored, human-readability is, in my opinion, a mistake.
You make good points. It’s funny, because I advocated hard for binary logs (and indeed stored many logs as protocol buffers on Kafka; only on the filesystem was it text) from systems at $dayjob-1, but when it comes to my own Linux system it’s a little harder for me to swallow. I suppose I’m looking at it from the perspective of an interactive user and not a fleet of Linux machines; on my own computer I like to be able to open my logs as standard text without needing to pipe it through a utility.
I’ll concede the point though: binary logs do make a lot more sense as building blocks if they’re done right and have sufficient metadata to be better than the machine-readable text format. If it’s a binary log of just date + facility + level + text description, it may as well have been a formatted text log.
I’m sort of curious about which “subsume everything” bits are hurting you in particular.
For example, subsuming the business of mounting is fairly necessary since these days the order in which things get mount relative to the order in which various services are run is pretty inexorable.
I have doubts about how much of the networkd / resolved should be part of systemd…. except something that collaborates with the startup infrastructure is required. ie. I suspect your choices in dinit will be slightly harsh…. modding dinit to play nice with existing network managers or modding existing network managers to play nice with dinit or subsuming the function of network management or leaving fairly vital chunks of functionality undone and undoable.
Especially in the world of hot plug devices and mobile data….. things get really really hairy.
I am dead-set that there will never be a dbus-connection in the PID 1
You still need a secure way of communicating with pid 1….
That said, systemd process itself could perhaps be decomposed into more processes than it currently is.
However as I hinted…. there are things that dbus gives you, like bounded trusted between untrusted and untrusting and untrustworthy programs that is hard to achieve without reimplementing large chunks of dbus….
…and then going through the long and painful process of learning from your mistakes that dbus has already gone through.
Yes, I truly hate xml in there…. but you still need some security sensitive serialization mechanism in there.
ie. Whatever framework you choose will still need to enforce the syntactic contract of the interface so that a untrusted and untrustworthy program cannot achieve a denial of service or escalation of privilege through abuse of a serialized interface.
There are other things out there that do that (eg. protobuffers, cap’n’proto, …), but then you still in a world where desktops and bluetooth and network managers and …….. need to be rewritten to use the new mechanism.
For example, subsuming the business of mounting is fairly necessary since these days the order in which things get mount relative to the order in which various services are run is pretty inexorable.
systemd’s handling of mounting is beyond broken. It’s impossible to get bind mounts to work successfully on boot, nfs mounts don’t work on boot unless you make systemd handle it with autofs and sacrifice a goat, and last week
I had a broken mount that couldn’t be fixed. umount said there were open files, lsof said none were open. Had to reboot because killing systemd would kill the box anyway.
It doesn’t even start MySQL reliably on boot either. Systemd is broken. Stop defending it.
For example, subsuming the business of mounting is fairly necessary since these days the order in which things get mount relative to the order in which various services are run is pretty inexorable.
There are a growing number of virtual filesystems that Linux systems expect or need to be mounted for full operation - /proc, /dev, /sys and cgroups all have their own - but these can all be mounted in the traditional way: by running ‘/bin/mount’ from a service. And because it’s a service, dependencies on it can be expressed. What Systemd does is understand the natural ordering imposed by mount paths as implicit dependencies between mount units, which is all well and good but which could also be expressed explicitly in service descriptions, either manually (how often do you really change your mount hierarchies…) or via an external tool. It doesn’t need to be part of the init system directly.
(Is it bad that systemd can do this? Not really; it is a feature. On the other hand, systemd’s complexity has I feel already gotten out of hand. Also, is this particular feature really giving that much real-world benefit? I’m not convinced).
I suspect your choices in dinit will be slightly harsh…. modding dinit to play nice with existing network managers or modding existing network managers to play nice with dinit
At this stage I want to believe there is another option: delegating Systemd API implementation to another daemon (which communicates with Dinit if and as it needs to). Of course such a daemon could be considered as part of Dinit anyway, so it’s a fine distinction - but I want to keep the lines between the components much clearer (than I feel they are in Systemd).
I believe in many cases the services provided by parts of Systemd don’t actually need to be tied to the init system. Case in point, elogind has extraced the logind functionality from systemd and made it systemd-independent. Similarly there’s eudev, the Gentoo fork of the udev device node management daemon which extracts it from systemd.
You still need a secure way of communicating with pid 1…
Right now, that’s via root-only unix socket, and I’d like to keep it that way. The moment unprivileged processes can talk to a privileged process, you have to worry about protocol flaws a lot more. The current protocol is compact and simple. More complicated behavior could be wrapped in another daemon with a more complex API, if necessary, but again, the boundary lines (is this init? is this service management? or is this something else?) can be kept clearer, I feel.
Putting it another way, a lot of the parts of Systemd that required a user-accessible API just won’t be part of Dinit itself: they’ll be part of an optional package that communicates the Dinit only if it needs to, and only by a simple internal protocol. That way, boundaries between components are more clear, and problems (whether bugs or configuration issues) are easier to localise and resolve.
Systemd solves (or attempts to) some actually existing problems, yes. It solves them from a purely Dev(Ops) perspective while completely ignoring that we use Linux-based systems in big part for how flexible they are. Systemd is a very big step towards making systems we use less transparent and simple in design. Thus, less flexible.
And if you say that’s the point: systems need to get more uniform and less unique!.. then sure. I very decidedly don’t want to work in an industry that cripples itself like that.
As a simple example, in sysv your only “targets” were the 7 runlevels. Pretty crude.
Alas the sysv simplicity came at a huge cost. Slow boots since it was hard to parallelize, and Moore’s law has stopped giving us more clock cycles… it only gives us more cores these days.
On my ubuntu xenial box I get…
locate target | grep -E ‘^/(run|etc|lib)/.*.target$’ | grep -v wants | wc
61 61 2249
(Including the 7 runlevels for backwards compatibility)
ie. Much more flexibility.
ie. You have much more flexibility than you ever had in sysv…. and if you need to drop into a whole of shell (or whatever) flexibility…. nothing is stopping you.
It’s actually very transparent…. the documentation is actually a darn sight better that sysv init ever was and the source code is pretty readable. (Although at the user level I find I can get by mostly by looking at the .service files and guessing, it’s a lot easy to read than a sysv init script.)
So my actual experience of wrangling systemd on a daily basis is it is more transparent and flexible than what we had before…..
A bunch of the complexity is due to the need to transition from sysv/upstart to systemd.
I can see on my box a huge amount of crud that can just be deleted once everything is converted.
All the serious “Huh!? WTF!?” moments in the last few weeks have been around the mishmash of old and new.
Seriously. It is simpler.
That said, could dinit be even simpler?
I don’t know.
As I say, systemd has invented it’s own quarter arsed language for the .unit files. Maybe if dinit uses a real language…. (I call shell a half arsed language)
You are comparing systemd to “sysv”. That’s a false dichotomy that was very agressively pushed into every conversation about systemd. No. Those are not the only two choices.
BTW, sysvinit is a dumb-ish init that can spawn processes and watch over them. We’ve been using it as more or less just a dumb init for the last decade or so. What you’re comparing systemd to is an amorphous, distro-specific blob of scripts, wrappers and helpers that actually did the work. Initscripts != sysvinit. Insserv != sysvinit.
I was using sysv as a hand waving reference to the various flavours of init /etc/init.d scripts, including upstart that Debian / Ubuntu have been using prior to systemd.
My point is not to say systemd is the greatest and end point of creation… my point is it’s a substantial advance on what went before (in yocto / ubuntu / debian land) (other distros may have something better than that I haven’t experienced.)
And I wasn’t seeing anything in the dinit aims and goals list yet that was making me saying, at the purely technical level, that the next step is on it’s way.
Ah, no, you probably meant just the language within systemd. But adding systemd-like functionality to The Shepherd would do that. I think running things in containers is in, or will be, but maybe The Shepherd is too tangled up in GuixSD for many people’s use cases.
Thanks for the encouragement. I hope to keep posting followups every 1-2 weeks. I know what you mean about the “will-be”s - I didn’t want it to sound like it was “nearly finished” - but some of those “will-be”s are in fact already done. The current code base seems to build correctly on Linux, OpenBSD and Mac OS (and was ok on FreeBSD too last time I tried); it can already act as a service manager/process supervisor on these systems, though I wouldn’t be willing to say it’s bug-free yet.
Just use runit. It’s dead simple, actually documented, actually used in production, BSD licensed, and so on. I use it on my work computer with no problems. It’s no bullshit, no bloat, you don’t have to “learn runit” to use it and get exactly what you want.
I’m aware of runit. It does seem pretty nice, but there are a few things about it that bother me. I don’t want to get into specifics here since it can so easily become a matter of one opinion vs the other, but I’ll try to write about some general issues which Dinit should handle well (and which I don’t think runit does) at some point in the near future.
Well one things that comes to mind is that runit doesn’t deal well (or at all) with (double-)forking services. Those are unfortunate by themselves — I mean, let the OS do its job please! — but still exist.
I have run into some odd behavior with runit a time or two, somehow managing to get something into a weird wedged state. I could never figure out what the exact problem was (maybe it is fixed by now?). Oddly enough, I never had the same issue with ye olde daemontools.
Aside from that, I do also like runit – as a non pid 1 process supervisor.
We use runit heavily at my job. It’s a massive pain to deal with, and we have to use a lot of automation to deal with the incredibly frequent issues we have with it. I would never recommend it to anyone, honestly.
Also, since then, we’ve had problems with svlogd losing track of the process that it’s logging for. Also it should be noted that you absolutely don’t get logging for free, and it requires additional management.
Runit does have support for dependencies in a way, you put the service start command in the run file and it starts the other service first, or blocks until it finishes starting. Right?
How does it lose track of its controlled processes? Like do you know what causes it? For example I know runit doesn’t manage double forking daemons.
What kind of scaffolding have you set up to ensure logging? What breakages do you have to guard against? How do you guard against them?
Do you know why svlogd loses the process? As I understand, it’s just hooked to stdout/stderr, so how could it lose track? What specific situations does that happen in? How did you resolve?
I know it’s a lot of questions, but I’m genuinely curious and would love to learn more.
How does it lose track of its controlled processes? Like do you know what causes it? For example I know runit doesn’t manage double forking daemons.
The reason runit, daemontools classic, and most other non-pid-1 supervisors “lose track” of supervised processes comes down to the lack of setsid(2). If you have a multiprocess service, in 99% of cases you should create a new process group for it and use process group signaling rather than single process signaling. If you don’t signal the entire process group when you sv down foo, you’re only killing the parent, and any children will become orphans inherited by pid 1, or an “orphaned process group” that might keep running.
A few years back I got a patch into daemontools-encore to do all of this, although we screwed up the default behavior (more on that in a second). You can read more about the hows and whys of multiprocess supervision in that daemontools-encore PR.
If you’re using a pid-1 supervisor like BSD init, upstart, systemd, etc it can do something more intelligent with these orphans, since it’s the one that inherits them. Also, pid-1 supervisors usually run services in a new process group by default.
Now, the screw-up: when we added multiprocess service support to daemontools-encore, I initially made setsid an opt-in feature. So by default, services wouldn’t run in a new process group, which is the “classic” behavior of daemontools, runit, et al. There are a few popular services like nginx that actually rely on this behavior for hot upgrades, or for more control over child processes during shutdown. Unfortunately I let myself get talked out of that, and we made setsid opt-out. That broke some of these existing services for people, and the maintainer did the worst thing possible, and half-backed out multiprocess service support.
At this point bruceg/daemontools-encore is pretty broken wrt multiprocess services, and I wouldn’t recommend using it. I don’t have the heart to go back and argue with the maintainer that we fix it by introducing breaking behavior again. Instead I re-forked it, fixed multiprocess support, and have been happily and quietly managing multiprocess services on production systems for several years now. It all just works. If you’re interested, here’s my fork: https://github.com/acg/daemontools-encore/tree/ubuntu-package-1.13
I guess I’ll end with a request for advice. How should I handle this situation fellow lobsters? Suck it up and get the maintainer to fix daemontools-encore? Make my fork a real fork? Give up and add proper setsid support to another daemontools derivative like runit?
There are several problems with multiprocess services in runit.
As mentioned above, some services should not use setsid, although most properly written services should. But runsvdir -P is global.
If you use runsvdir -P, then sv down foo should use process group signalling instead of parent process signalling, or you can still create orphans. As another example, sv stop foo should send SIGSTOP to all processes in the process group, but since it doesn’t, everyone but the parent process continues to run (ouch!). Unfortunately runit entirely lacks facilities for process group signalling.
In my patched daemontools-encore:
svc -=X foo signals the parent process only
svc -+X foo signals the entire process group
svc -X foo does one or the other depending on whether you’ve marked the service as multiprocess with a ./setsid file
But generally you just use the standard svc -X foo, because it does the right thing.
Besides the things mentioned above, runsvdir -P introduces some fresh havoc in other settings. Try this in a foreground terminal:
The find / | wc -l is still running, even though you ^C’ed the whole thing! What happened? Well, things like ^C and ^Z result in signals being sent to the terminal’s foreground process group. Your service is running in a new, separate process group, so it gets spun off as an orphan. The only good way to handle this is for the supervisor to trap and relay SIGINT and SIGHUP to the process groups underneath it.
To those wondering who runs a supervisor in a foreground terminal as non-root…me! All the time. The fact that daemontools derivatives let you supervise processes directly, without all that running-as-root action-at-a-distance system machinery, is one of their huge selling points.
Dinit already used setsid, today I made it signal service process groups instead of just the main process. However when run as a foreground process - which btw Dinit fully supports, that’s how I test it usually - you can specify that individual services run “on console” and they won’t get setsid()‘d in this case. I’m curious though as to how running anything in a new session (with setsid) actually causes anything to break? Unless a service somehow tries to do something to the console directly it shouldn’t matter at all, right?
I’m curious though as to how running anything in a new session (with setsid) actually causes anything to break? Unless a service somehow tries to do something to the console directly it shouldn’t matter at all, right?
The problems are outlined above. ^C, ^Z etc get sent to the tty’s foreground process group. If the supervisor is running foreground with services under it in separate process groups, they will continue running on ^C and ^Z. In order to get the behavior users expect – total exit and total stop of everything in the process tree, respectively – you need to catch SIGINT, SIGTSTP, and SIGCONT in the foreground process group and relay them to the service process groups. Here’s what the patch to add that behavior to daemontools-encore looked like.
Thanks for the info! I still have a few questions, correlated to your numbered points:
Like nginx yes? How does a pid 1 handle nginx differently / what makes a pid 1 different? If 99% of stuff needs the process group signaled, but nginx works with pid 1 supervisors, do they not signal the process group? How does all that work? And how does all of this tie in to using runit as a pid 1? Would the problems you have with it not exist for people using it as a pid 1? Because the original discussion was about alternate init systems, which is how I use it.
This would only create orphans if the child process ignores sighup right? Obviously that’s still a big problem, but am I correctly understanding that? And when runsvdir gets sighup it then correctly forwards sigterm to its children yes? Not as easy as ^C but still possible. Would any of this behavior be different if you were running as root, but still not as pid 1?
We’ve already developed sinit, so what it really boils down to is a good service manager. We have svc, but fresh ideas are always welcome.
If I may give advice, it really makes sense to keep PID1 as dumb as possible (i.e. using sinit). For everything else built on top of it: Keep it simple! :)
Also, there is the possibility of having a small, simple PID-1 init which sends terminated process IDs over a pipe to Dinit.
:)
The transmission of process IDs is neccessary for supervising double-forked processes (although of course ideally those don’t exist), and there also needs to be some co-operation from pid 1 if Dinit is going to properly handle cgroups (essentially: to kill all processes in a cgroup, you need to iterate through the process IDs and send them a signal. That’s racy, and if processes can die and be reaped during the process, the process ID can theoretically be recycled, which could lead to the wrong process receiving the signal. Systemd suffered from this problem at one stage, I’m not sure if it handles it correctly now).
Thanks. C++11 seemed like a fairly safe bet in terms of compiler support on a good range of platforms, was the main reason. I’d consider upping that to ‘14 if I found a compelling need, but I haven’t (so far).
The apache-2.0 license was a bit unexpected for an intended-to-be cross platform init though, especially given the explicit mention of OpenBSD in the source README, as apache-2.0 works are explicitly not shippable in OpenBSD base.
Thanks. I guess as I’m at this stage the only contributor, I could change the license - and I would do so happily enough, if OpenBSD (or any other of the BSDs) said they’d use it. But I’m pretty sure Theo de Raadt is also violently allergic to C++, so I’m fairly dubious it will ever make it into OpenBSD base.
Please let me know though if you chose Apache for a specific reason
Yes and no. I generally like the “you can’t bung some patented code in and later use this to prevent use/distribution of the software” clause though I don’t see it as much of a practical concern. On the other hand I didn’t choose GPL because I’d rather have people (and OSes) who would reject GPL software consider using it, than have the vague comfort that no corporation can simply take my code and slap a commercial license over it.
One minor niggle with the ISC license - and it’s really just a niggle, not a major concern - is the “provided that the above copyright notice [is included]” clause - because I honestly don’t care if my name is one of potentially hundreds listed in some later code that incorporates this particular work.
Finally, I’ve incorporated code from an asynchronous event processing library, Dasynq, which is also my own project and which is also under the Apache license. Of course, since it’s my own code and there are no other contributors I could easily re-license it either for this case or generally.
Are there any particular problems with the Apache license I should be aware of (other than that it’s a tad longer than probably necessary)?
That surprised me, I thought it was GPL compatible. But on checking I see you’re right, it’s GPLv3 compatible but not GPLv2 compatible (at least according to GNU). I’m now considering either dual licensing or switching to ISC, but I’ll probably give it some time before I make a decision.
I know you said you’d cover this in a future blog post, but can you give a high-level list of bullet points? I’m sure the Rust devs would love to hear your thoughts.
I’m sure I’ll get a “well actually” for this but did you considered Go? If so, what turned you off to the idea.
Sorry, I meant to respond to this earlier. There’s not really anything I can say that’s likely to be of much use to the Rust developers, I’m afraid; I feel like the metaprogramming facilities in Rust aren’t yet as powerful as they should be (remember that C++ is my model here), and the language is harder to use than perhaps it could be (though I’m speaking mainly from secondhand experience here - I have a friend who’s using it relatively heavily and have seen him struggle with how to implement certain patterns). I know it’s also fiendishly tricky getting the Rust compiler itself to compile on OpenBSD (though of course the stable compiler is packaged). Probably the single biggest issue is that I couldn’t find an event loop library for Rust that looked like it fit the bill (must be able to handle timers - ideally both monotonic and realtime - signals, and process watch, as well as file descriptor readiness events, with ahead-of-time allocation where possible, and cross-platform of course).
As for Go, I don’t think garbage collection belongs in an init process, and in particular I want to be certain that I can control allocation if not deallocation. I will talk a bit more about this in my blog.
But ultimately, I like C++ and am familiar with it. That’s probably the main reason for the language choice, truth be told.
Hmm. I have just spent a week or two getting my mind around systemd, so I will add a few comments….
** The degree of parallelism achieved by systemd does very good things to start up times. (Yes, that is a critical parameter, especially in the embedded world)
** Socket activation is very nifty / useful.
** There are a lot of learning that has gone into things like dbus https://lwn.net/Articles/641277/ While there are things I really don’t like about dbus (cough, xml, cough)…. I respect the hard earned experience encoded into it)
** Systemd’s use of cgroups is actually a very very nifty feature in creating rock solid systems, systems that don’t go sluggish because a subsystem is rogue or leaky. (But I think we are all just learning to use it properly)
** The thought and effort around “playing nice” with distro packaging systems via “drop in” directories is valuable. Yup, it adds complication, but packaging is real and you need a solution.
** The thought and complication around generators to aid the transition from sysv to systemd is also vital. Nobody can upgrade tens of thousands of packages in one go.
TL;DR; Systemd actually gives us a lot of very very useful and important stuff. Any competing system with the faintest hope of wide adoption has a pretty high bar to meet.
The biggest sort of “WAT!?” moments for me around systemd is that it creates it’s own entirely new language… that is remarkably weaker even than shell. And occasionally you find yourself explicitly invoking, yuck, shell, to get stuff done.
Personally I would have preferred it to be something like guile with some addons / helper macros.
I actually agree with most of what you’ve said here, Systemd is definitely trying to solve some real problems and I fully acknowledge that. The main problem I have with Systemd is the way it just subsumes so much and it’s pretty much all-or-nothing; combined with that, people do experience real problems with it and I personally believe its design is too complicated, especially for such an essential part of the system. I’ll talk about it a bit more in my blog (along with lots of other things) at some stage, but in general the features you list are good features and I hope to have Dinit support eg socket activation and cgroups (though as an optional rather than mandatory feature). On the other hand I am dead-set that there will never be a dbus-connection in the PID 1 process nor any XML-based protocol, and I’m already thinking about separating the PID 1 process from the service manager, etc.
Please stick with human-readable logs too. :)
Please don’t. It is a lot easier to turn machine-readable / binary logs to human-readable than the other way around, and machines will be processing and reading logs a lot more than humans.
Human-readable doesn’t mean freeform. It can be machine-readable too. At my last company, we logged everything as date, KV pairs, and only then freeform text. It had a natural mapping to JSON and protocol buffers after that.
https://github.com/uber-go/zap This isn’t what we used, but the general idea.
Yeah, you can do that. But then it becomes quite a bit harder to sign, encrypt, or index logs. I still maintain that going binary->human readable is more efficient, and practical, as long as computers do more processing on the logs than humans do.
Mind you, I’m talking about storage. The logs should be reasonably easy for a human to process when emitted, and a mapping to a human-readable format is desirable. When stored, human-readability is, in my opinion, a mistake.
You make good points. It’s funny, because I advocated hard for binary logs (and indeed stored many logs as protocol buffers on Kafka; only on the filesystem was it text) from systems at $dayjob-1, but when it comes to my own Linux system it’s a little harder for me to swallow. I suppose I’m looking at it from the perspective of an interactive user and not a fleet of Linux machines; on my own computer I like to be able to open my logs as standard text without needing to pipe it through a utility.
I’ll concede the point though: binary logs do make a lot more sense as building blocks if they’re done right and have sufficient metadata to be better than the machine-readable text format. If it’s a binary log of just date + facility + level + text description, it may as well have been a formatted text log.
So long as they accumulate the same amount of useful info…. and is machine parsable, sure.
journalctl spits out human readable or json or whatever.
I suspect to achieve near the same information density / speed as journalctl with plain old ascii will be a hard ask.
In my view I want both. Human and machine readable… how that is done is an implementation detail.
I’m sort of curious about which “subsume everything” bits are hurting you in particular.
For example, subsuming the business of mounting is fairly necessary since these days the order in which things get mount relative to the order in which various services are run is pretty inexorable.
I have doubts about how much of the networkd / resolved should be part of systemd…. except something that collaborates with the startup infrastructure is required. ie. I suspect your choices in dinit will be slightly harsh…. modding dinit to play nice with existing network managers or modding existing network managers to play nice with dinit or subsuming the function of network management or leaving fairly vital chunks of functionality undone and undoable.
Especially in the world of hot plug devices and mobile data….. things get really really hairy.
You still need a secure way of communicating with pid 1….
That said, systemd process itself could perhaps be decomposed into more processes than it currently is.
However as I hinted…. there are things that dbus gives you, like bounded trusted between untrusted and untrusting and untrustworthy programs that is hard to achieve without reimplementing large chunks of dbus….
…and then going through the long and painful process of learning from your mistakes that dbus has already gone through.
Yes, I truly hate xml in there…. but you still need some security sensitive serialization mechanism in there.
ie. Whatever framework you choose will still need to enforce the syntactic contract of the interface so that a untrusted and untrustworthy program cannot achieve a denial of service or escalation of privilege through abuse of a serialized interface.
There are other things out there that do that (eg. protobuffers, cap’n’proto, …), but then you still in a world where desktops and bluetooth and network managers and …….. need to be rewritten to use the new mechanism.
systemd’s handling of mounting is beyond broken. It’s impossible to get bind mounts to work successfully on boot, nfs mounts don’t work on boot unless you make systemd handle it with autofs and sacrifice a goat, and last week I had a broken mount that couldn’t be fixed. umount said there were open files, lsof said none were open. Had to reboot because killing systemd would kill the box anyway.
It doesn’t even start MySQL reliably on boot either. Systemd is broken. Stop defending it.
There are a growing number of virtual filesystems that Linux systems expect or need to be mounted for full operation - /proc, /dev, /sys and cgroups all have their own - but these can all be mounted in the traditional way: by running ‘/bin/mount’ from a service. And because it’s a service, dependencies on it can be expressed. What Systemd does is understand the natural ordering imposed by mount paths as implicit dependencies between mount units, which is all well and good but which could also be expressed explicitly in service descriptions, either manually (how often do you really change your mount hierarchies…) or via an external tool. It doesn’t need to be part of the init system directly.
(Is it bad that systemd can do this? Not really; it is a feature. On the other hand, systemd’s complexity has I feel already gotten out of hand. Also, is this particular feature really giving that much real-world benefit? I’m not convinced).
At this stage I want to believe there is another option: delegating Systemd API implementation to another daemon (which communicates with Dinit if and as it needs to). Of course such a daemon could be considered as part of Dinit anyway, so it’s a fine distinction - but I want to keep the lines between the components much clearer (than I feel they are in Systemd).
I believe in many cases the services provided by parts of Systemd don’t actually need to be tied to the init system. Case in point, elogind has extraced the logind functionality from systemd and made it systemd-independent. Similarly there’s eudev, the Gentoo fork of the udev device node management daemon which extracts it from systemd.
Right now, that’s via root-only unix socket, and I’d like to keep it that way. The moment unprivileged processes can talk to a privileged process, you have to worry about protocol flaws a lot more. The current protocol is compact and simple. More complicated behavior could be wrapped in another daemon with a more complex API, if necessary, but again, the boundary lines (is this init? is this service management? or is this something else?) can be kept clearer, I feel.
Putting it another way, a lot of the parts of Systemd that required a user-accessible API just won’t be part of Dinit itself: they’ll be part of an optional package that communicates the Dinit only if it needs to, and only by a simple internal protocol. That way, boundaries between components are more clear, and problems (whether bugs or configuration issues) are easier to localise and resolve.
Comments like this makes me wonder what you actually know about D-Bus and what you think it uses XML for.
I suppose you are hinting that I’ve somehow claimed D-Bus is/uses an XML-based protocol? Read the statement again…
It certainly sounded like it anyway.
Systemd solves (or attempts to) some actually existing problems, yes. It solves them from a purely Dev(Ops) perspective while completely ignoring that we use Linux-based systems in big part for how flexible they are. Systemd is a very big step towards making systems we use less transparent and simple in design. Thus, less flexible.
And if you say that’s the point: systems need to get more uniform and less unique!.. then sure. I very decidedly don’t want to work in an industry that cripples itself like that.
Hmm. I strongly disagree with that.
As a simple example, in sysv your only “targets” were the 7 runlevels. Pretty crude.
Alas the sysv simplicity came at a huge cost. Slow boots since it was hard to parallelize, and Moore’s law has stopped giving us more clock cycles… it only gives us more cores these days.
On my ubuntu xenial box I get… locate target | grep -E ‘^/(run|etc|lib)/.*.target$’ | grep -v wants | wc 61 61 2249
(Including the 7 runlevels for backwards compatibility)
ie. Much more flexibility.
ie. You have much more flexibility than you ever had in sysv…. and if you need to drop into a whole of shell (or whatever) flexibility…. nothing is stopping you.
It’s actually very transparent…. the documentation is actually a darn sight better that sysv init ever was and the source code is pretty readable. (Although at the user level I find I can get by mostly by looking at the .service files and guessing, it’s a lot easy to read than a sysv init script.)
So my actual experience of wrangling systemd on a daily basis is it is more transparent and flexible than what we had before…..
A bunch of the complexity is due to the need to transition from sysv/upstart to systemd.
I can see on my box a huge amount of crud that can just be deleted once everything is converted.
All the serious “Huh!? WTF!?” moments in the last few weeks have been around the mishmash of old and new.
Seriously. It is simpler.
That said, could dinit be even simpler?
I don’t know.
As I say, systemd has invented it’s own quarter arsed language for the .unit files. Maybe if dinit uses a real language…. (I call shell a half arsed language)
You are comparing systemd to “sysv”. That’s a false dichotomy that was very agressively pushed into every conversation about systemd. No. Those are not the only two choices.
BTW, sysvinit is a dumb-ish init that can spawn processes and watch over them. We’ve been using it as more or less just a dumb init for the last decade or so. What you’re comparing systemd to is an amorphous, distro-specific blob of scripts, wrappers and helpers that actually did the work. Initscripts != sysvinit. Insserv != sysvinit.
Ok, fair cop.
I was using sysv as a hand waving reference to the various flavours of init /etc/init.d scripts, including upstart that Debian / Ubuntu have been using prior to systemd.
My point is not to say systemd is the greatest and end point of creation… my point is it’s a substantial advance on what went before (in yocto / ubuntu / debian land) (other distros may have something better than that I haven’t experienced.)
And I wasn’t seeing anything in the dinit aims and goals list yet that was making me saying, at the purely technical level, that the next step is on it’s way.
So, https://www.gnu.org/software/shepherd/ ?
Ah, no, you probably meant just the language within systemd. But adding systemd-like functionality to The Shepherd would do that. I think running things in containers is in, or will be, but maybe The Shepherd is too tangled up in GuixSD for many people’s use cases.
The blog post is pretty heavy on the “will-be”s, but I do see lots of progress in the git repo. Looking forward to a followup post!
Thanks for the encouragement. I hope to keep posting followups every 1-2 weeks. I know what you mean about the “will-be”s - I didn’t want it to sound like it was “nearly finished” - but some of those “will-be”s are in fact already done. The current code base seems to build correctly on Linux, OpenBSD and Mac OS (and was ok on FreeBSD too last time I tried); it can already act as a service manager/process supervisor on these systems, though I wouldn’t be willing to say it’s bug-free yet.
Just use
runit. It’s dead simple, actually documented, actually used in production, BSD licensed, and so on. I use it on my work computer with no problems. It’s no bullshit, no bloat, you don’t have to “learn runit” to use it and get exactly what you want.I’m aware of runit. It does seem pretty nice, but there are a few things about it that bother me. I don’t want to get into specifics here since it can so easily become a matter of one opinion vs the other, but I’ll try to write about some general issues which Dinit should handle well (and which I don’t think runit does) at some point in the near future.
Well one things that comes to mind is that runit doesn’t deal well (or at all) with (double-)forking services. Those are unfortunate by themselves — I mean, let the OS do its job please! — but still exist.
I have run into some odd behavior with runit a time or two, somehow managing to get something into a weird wedged state. I could never figure out what the exact problem was (maybe it is fixed by now?). Oddly enough, I never had the same issue with ye olde daemontools.
Aside from that, I do also like runit – as a non pid 1 process supervisor.
We use runit heavily at my job. It’s a massive pain to deal with, and we have to use a lot of automation to deal with the incredibly frequent issues we have with it. I would never recommend it to anyone, honestly.
Examples?
I’ve mentioned this here: https://lobste.rs/s/2qjf4o/problems_with_systemd_why_i_like_bsd_init#c_8qtwla
Also, since then, we’ve had problems with svlogd losing track of the process that it’s logging for. Also it should be noted that you absolutely don’t get logging for free, and it requires additional management.
Runit does have support for dependencies in a way, you put the service start command in the run file and it starts the other service first, or blocks until it finishes starting. Right?
How does it lose track of its controlled processes? Like do you know what causes it? For example I know runit doesn’t manage double forking daemons.
What kind of scaffolding have you set up to ensure logging? What breakages do you have to guard against? How do you guard against them?
Do you know why svlogd loses the process? As I understand, it’s just hooked to stdout/stderr, so how could it lose track? What specific situations does that happen in? How did you resolve?
I know it’s a lot of questions, but I’m genuinely curious and would love to learn more.
The reason
runit,daemontoolsclassic, and most other non-pid-1 supervisors “lose track” of supervised processes comes down to the lack ofsetsid(2). If you have a multiprocess service, in 99% of cases you should create a new process group for it and use process group signaling rather than single process signaling. If you don’t signal the entire process group when yousv down foo, you’re only killing the parent, and any children will become orphans inherited by pid 1, or an “orphaned process group” that might keep running.A few years back I got a patch into
daemontools-encoreto do all of this, although we screwed up the default behavior (more on that in a second). You can read more about the hows and whys of multiprocess supervision in that daemontools-encore PR.If you’re using a pid-1 supervisor like BSD init, upstart, systemd, etc it can do something more intelligent with these orphans, since it’s the one that inherits them. Also, pid-1 supervisors usually run services in a new process group by default.
Now, the screw-up: when we added multiprocess service support to
daemontools-encore, I initially madesetsidan opt-in feature. So by default, services wouldn’t run in a new process group, which is the “classic” behavior ofdaemontools,runit, et al. There are a few popular services likenginxthat actually rely on this behavior for hot upgrades, or for more control over child processes during shutdown. Unfortunately I let myself get talked out of that, and we madesetsidopt-out. That broke some of these existing services for people, and the maintainer did the worst thing possible, and half-backed out multiprocess service support.At this point
bruceg/daemontools-encoreis pretty broken wrt multiprocess services, and I wouldn’t recommend using it. I don’t have the heart to go back and argue with the maintainer that we fix it by introducing breaking behavior again. Instead I re-forked it, fixed multiprocess support, and have been happily and quietly managing multiprocess services on production systems for several years now. It all just works. If you’re interested, here’s my fork: https://github.com/acg/daemontools-encore/tree/ubuntu-package-1.13I guess I’ll end with a request for advice. How should I handle this situation fellow lobsters? Suck it up and get the maintainer to fix
daemontools-encore? Make my fork a real fork? Give up and add propersetsidsupport to another daemontools derivative likerunit?Thank you for all your answers! Can you comment on the -P flag in runsvdir? Does that not do what you want?
There are several problems with multiprocess services in
runit.As mentioned above, some services should not use
setsid, although most properly written services should. Butrunsvdir -Pis global.If you use
runsvdir -P, thensv down fooshould use process group signalling instead of parent process signalling, or you can still create orphans. As another example,sv stop fooshould sendSIGSTOPto all processes in the process group, but since it doesn’t, everyone but the parent process continues to run (ouch!). Unfortunatelyrunitentirely lacks facilities for process group signalling.In my patched daemontools-encore:
svc -=X foosignals the parent process onlysvc -+X foosignals the entire process groupsvc -X foodoes one or the other depending on whether you’ve marked the service as multiprocess with a./setsidfileBut generally you just use the standard
svc -X foo, because it does the right thing.Besides the things mentioned above,
runsvdir -Pintroduces some fresh havoc in other settings. Try this in a foreground terminal:The
find / | wc -lis still running, even though you ^C’ed the whole thing! What happened? Well, things like ^C and ^Z result in signals being sent to the terminal’s foreground process group. Your service is running in a new, separate process group, so it gets spun off as an orphan. The only good way to handle this is for the supervisor to trap and relaySIGINTandSIGHUPto the process groups underneath it.To those wondering who runs a supervisor in a foreground terminal as non-root…me! All the time. The fact that daemontools derivatives let you supervise processes directly, without all that running-as-root action-at-a-distance system machinery, is one of their huge selling points.
Dinit already used setsid, today I made it signal service process groups instead of just the main process. However when run as a foreground process - which btw Dinit fully supports, that’s how I test it usually - you can specify that individual services run “on console” and they won’t get setsid()‘d in this case. I’m curious though as to how running anything in a new session (with setsid) actually causes anything to break? Unless a service somehow tries to do something to the console directly it shouldn’t matter at all, right?
The problems are outlined above. ^C, ^Z etc get sent to the tty’s foreground process group. If the supervisor is running foreground with services under it in separate process groups, they will continue running on ^C and ^Z. In order to get the behavior users expect – total exit and total stop of everything in the process tree, respectively – you need to catch
SIGINT,SIGTSTP, andSIGCONTin the foreground process group and relay them to the service process groups. Here’s what the patch to add that behavior todaemontools-encorelooked like.Thanks for the info! I still have a few questions, correlated to your numbered points:
Like nginx yes? How does a pid 1 handle nginx differently / what makes a pid 1 different? If 99% of stuff needs the process group signaled, but nginx works with pid 1 supervisors, do they not signal the process group? How does all that work? And how does all of this tie in to using runit as a pid 1? Would the problems you have with it not exist for people using it as a pid 1? Because the original discussion was about alternate init systems, which is how I use it.
This would only create orphans if the child process ignores sighup right? Obviously that’s still a big problem, but am I correctly understanding that? And when runsvdir gets sighup it then correctly forwards sigterm to its children yes? Not as easy as ^C but still possible. Would any of this behavior be different if you were running as root, but still not as pid 1?
We’ve already developed sinit, so what it really boils down to is a good service manager. We have svc, but fresh ideas are always welcome.
If I may give advice, it really makes sense to keep PID1 as dumb as possible (i.e. using sinit). For everything else built on top of it: Keep it simple! :)
From the current TODO:
:)
The transmission of process IDs is neccessary for supervising double-forked processes (although of course ideally those don’t exist), and there also needs to be some co-operation from pid 1 if Dinit is going to properly handle cgroups (essentially: to kill all processes in a cgroup, you need to iterate through the process IDs and send them a signal. That’s racy, and if processes can die and be reaped during the process, the process ID can theoretically be recycled, which could lead to the wrong process receiving the signal. Systemd suffered from this problem at one stage, I’m not sure if it handles it correctly now).
Good luck with your project.
After building, I noticed that you’re using C++11, why not 14 or 17?
Thanks. C++11 seemed like a fairly safe bet in terms of compiler support on a good range of platforms, was the main reason. I’d consider upping that to ‘14 if I found a compelling need, but I haven’t (so far).
C++11 + std::experimental::filesystem from 17 is a good alternative until we get good 14/17 support.
Or you could use OpenRC. It even has a (primitive) supervisor: https://github.com/OpenRC/openrc/blob/master/supervise-daemon-guide.md
Seems interesting.
The apache-2.0 license was a bit unexpected for an intended-to-be cross platform init though, especially given the explicit mention of OpenBSD in the source README, as apache-2.0 works are explicitly not shippable in OpenBSD base.
Thanks. I guess as I’m at this stage the only contributor, I could change the license - and I would do so happily enough, if OpenBSD (or any other of the BSDs) said they’d use it. But I’m pretty sure Theo de Raadt is also violently allergic to C++, so I’m fairly dubious it will ever make it into OpenBSD base.
Just use the ISC license. Please let me know though if you chose Apache for a specific reason.
Yes and no. I generally like the “you can’t bung some patented code in and later use this to prevent use/distribution of the software” clause though I don’t see it as much of a practical concern. On the other hand I didn’t choose GPL because I’d rather have people (and OSes) who would reject GPL software consider using it, than have the vague comfort that no corporation can simply take my code and slap a commercial license over it.
One minor niggle with the ISC license - and it’s really just a niggle, not a major concern - is the “provided that the above copyright notice [is included]” clause - because I honestly don’t care if my name is one of potentially hundreds listed in some later code that incorporates this particular work.
Finally, I’ve incorporated code from an asynchronous event processing library, Dasynq, which is also my own project and which is also under the Apache license. Of course, since it’s my own code and there are no other contributors I could easily re-license it either for this case or generally.
Are there any particular problems with the Apache license I should be aware of (other than that it’s a tad longer than probably necessary)?
We use MIT/Apache2.0 for Rust because Apache2.0 isn’t GPL compatible, but MIT is, so you get to choose which aspect you care about more.
That surprised me, I thought it was GPL compatible. But on checking I see you’re right, it’s GPLv3 compatible but not GPLv2 compatible (at least according to GNU). I’m now considering either dual licensing or switching to ISC, but I’ll probably give it some time before I make a decision.
Hah! I didn’t even think twice about the use of C++, but you might be right.
I know you said you’d cover this in a future blog post, but can you give a high-level list of bullet points? I’m sure the Rust devs would love to hear your thoughts.
I’m sure I’ll get a “well actually” for this but did you considered Go? If so, what turned you off to the idea.
Sorry, I meant to respond to this earlier. There’s not really anything I can say that’s likely to be of much use to the Rust developers, I’m afraid; I feel like the metaprogramming facilities in Rust aren’t yet as powerful as they should be (remember that C++ is my model here), and the language is harder to use than perhaps it could be (though I’m speaking mainly from secondhand experience here - I have a friend who’s using it relatively heavily and have seen him struggle with how to implement certain patterns). I know it’s also fiendishly tricky getting the Rust compiler itself to compile on OpenBSD (though of course the stable compiler is packaged). Probably the single biggest issue is that I couldn’t find an event loop library for Rust that looked like it fit the bill (must be able to handle timers - ideally both monotonic and realtime - signals, and process watch, as well as file descriptor readiness events, with ahead-of-time allocation where possible, and cross-platform of course).
As for Go, I don’t think garbage collection belongs in an init process, and in particular I want to be certain that I can control allocation if not deallocation. I will talk a bit more about this in my blog.
But ultimately, I like C++ and am familiar with it. That’s probably the main reason for the language choice, truth be told.