There is nothing really desirable about openrc and it simply does not support the required features like supervision.
Sometimes its better to start fresh, or in this case with the already existing s6/s6-rc which is build on a better design.
But why compare the “simplicity” to systemd instead of something actually simple, openrcs design choices with its shell wrapping instead of a simple supervision design and a way to express dependencies outside of the shell script is a lot simpler.
The daemontool like supervision systems simply have no boilerplate in shell scripts and provide good features like tracking pids without pid files and therefor reliably signaling the right processes, they are able to restart services if they get down and they provide a nice and reliable way to collect stdout/stderr logs of those services.
Edit: this is really what the post is about, taking the better design and making it more user friendly and implementing the missing parts.
This work will also build on the work we’ve done with ifupdown-ng, as ifupdown-ng will be able to reflect its own state into the service manager allowing it to start services or stop them as the network state changes. OpenRC does not support reacting to arbitrary events, which is why this functionality is not yet available.
also, the second to last graf
Alpine has gotten a lot of mileage out of OpenRC, and we are open to contributing to its future maintenance while Alpine releases still include it as part of the base system, but our long-term goal is to adopt the s6-based solution.
so, they are continuing to maintain OpenRC while alpine still requires it, but it doesn’t meet their needs, hence they are designing something new
To my knowledge, there’s serious discussion in the FreeBSD community about replacing their init system (for example, see this talk from FreeBSD contributor and previous Core Team member Benno Rice: The Tragedy of systemd).
And then there’s the FreeBSD-based Darwin, whose launchd is much more similar to systemd than to either BSD init or SysVinit to my knowledge.
I believe the last major change on FreeBSD was adding the rc-order stuff (from NetBSD?) that allowed expressing dependencies between services and sorting their launch order so that dependencies were fulfilled.
That said, writing a replacement for the FreeBSD service manager infrastructure is something I’d really, really like to do. Currently devd, inetd, and cron are completely separate things and so you have different (but similar) infrastructure for running a service:
At system start / shutdown
At a specific time
In response to a kernel-generated event
In response to a network connection
I really like the way that Launchd unifies these (though I hate the fact that it uses XML property lists, which are fine as a human-readable serialisation of a machine format, but are not very human-writeable). I’d love to have something that uses libucl to provide a nice composable configuration for all of these. I’d also like an init system that plays nicely with the sandboxing infrastructure on FreeBSD. In particular, I’d like to be able to manage services that run inside a jail, without needing to run a service manager inside the jail. I’d also like something that can set up services in Capsicum sandboxes with libpreopen-style behaviour.
I believe the last major change on FreeBSD was adding the rc-order stuff (from NetBSD?) that allowed expressing dependencies between services and sorting their launch order so that dependencies were fulfilled.
I’ve been able to manage standalone services to run inside a jail, but it’s more than a little hacky. For fun a while back, I wrote a finger daemon in Go, so I could keep my PGP keys available without needing to run something written in C. This runs inside a bare-jail with a RO mount of the homedirs and not much else and lots of FS restrictions. So jail.conf ended up with this in the stanza:
finger {
# ip4.addr, ip6.addr go here; also mount and allow overrides
exec.start = "";
exec.stop = "";
persist;
exec.poststart = "service fingerd start";
exec.prestop = "service fingerd stop";
}
and then the service file does daemon -c jexec -u ${runtime_user_nonjail} ${jail_name} ${jail_fingerd} ...; the tricky bit was messing inside the internals of rc.subr to make sure that pidfile management worked correctly, with the process finding handling that the jail is not “our” jail:
It works, but I suspect that stuff like $JID can change without notice to me as an implementation detail of rc.subr. Something properly supported would be nice.
I think the core issue is that desktops have very different requirements than servers. Servers generally have fixed hardware, and thus a hard-coded boot order can be sufficient.
Modern desktops have to deal with many changes like: USB disks being plugged in (mounting and unmounting), Wi-Fi going in and out, changing networks, multiple networks, Bluetooth audio, etc. It’s a very different problem
I do think there should be some “server only” init systems, and I think there are a few meant for containers but I haven’t looked into them. If anyone has pointers I’d be interested. Desktop is a complex space but I don’t think that it needs to infect the design for servers (or maybe I’m wrong).
Alpine has a mix of requirements I imagine. I would only use it for servers, and its original use case was routers, but I’m guessing the core devs also use it as their desktops.
I’m not a huge systemd opponent, but the reason I dislike it enough to have an opinion are the unit files. The language is neither flexible nor intuitive enough to be guessed, nor do I use it often enough to memorize it. That means whenever I want to create a new service or timer or whatever, I have to deal with onscure documentation or copy-paste-tweak some code from arch wiki or stack overflow. A system like runit is attractive (though not technically superior) because all I need is to write my command in a file named “run”.
I hope that Alpine’s alternative will make improvments on this front, be it by providing better tooling or by improving the language. Ideally it should be able to parse systemd unit files but allow these to be degenerated as much as possible, allowing lazy users to only write as much as neccesary.
I find the systemd documentation pretty straight-forward and very well written, so I’m surprised that you find it obscure :)
The man pages are my go to for anything related to systemd, and examples of all kinds of systemd units, including timers, are plentiful and already installed on my laptop:
$ systemctl list-units --type timer
UNIT LOAD ACTIVE SUB DESCRIPTION
anacron.timer loaded active waiting Trigger anacron every hour
apt-daily-upgrade.timer loaded active waiting Daily apt upgrade and clean activities
apt-daily.timer loaded active waiting Daily apt download activities
e2scrub_all.timer loaded active waiting Periodic ext4 Online Metadata Check for All Filesystems
fstrim.timer loaded active waiting Discard unused blocks once a week
fwupd-refresh.timer loaded active waiting Refresh fwupd metadata regularly
logrotate.timer loaded active waiting Daily rotation of log files
man-db.timer loaded active waiting Daily man-db regeneration
motd-news.timer loaded active waiting Message of the Day
systemd-tmpfiles-clean.timer loaded active waiting Daily Cleanup of Temporary Directories
update-notifier-download.timer loaded active waiting Download data for packages that failed at package install time
update-notifier-motd.timer loaded active waiting Check to see whether there is a new version of Ubuntu available
zsys-gc.timer loaded active waiting Clean up old snapshots to free space
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
13 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
$ systemctl cat apt-daily.timer
# /lib/systemd/system/apt-daily.timer
[Unit]
Description=Daily apt download activities
[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
[Install]
WantedBy=timers.target
Once in a while I find a systemd keyword used in units that I can’t find the documentation for, but luckily man can be used for this purpose too:
Maybe obscure it the wrong word, but as I say, as someone who only deals with systemd directly once or twice a year,
$ man -k systemd | wc -l
205
is a first step I have to make every time.
Just a few months ago I wanted to create a timer, to replace a user cron job, but the number of concepts I have to read up on before I write anything seems like it’s not worth it.
Maybe this is just a mindset thing, *nix is certainly not obvious when you first start, but you gain an intuition over time and can start guessing how to do things. Systemd doesn’t play into this intuition.
Completely understand what you mean – the increasing complexity of computers and the layers upon layers of abstraction is definitely an issue. I deal with systemd unit files fairly often, so I’ve internalized much of the knowledge and feel that systemd units behave pretty much like I expect them to.
Much of it probably comes down to systemd simply being different from what we’ve used in the past – sysv style init, cron daemons etc. However even though I’ve used Linux for almost two decades and I’ve written plenty of cron jobs I still find myself looking at the man page to decipher the cron timer values. In that regard I do find systemd a lot more sane – the unit files themselves are much more readable than both cron jobs and sysv style init scripts, not at least because they’re more verbose, and the documentation is really good.
However even though I’ve used Linux for almost two decades and I’ve written plenty of cron jobs I still find myself looking at the man page to decipher the cron timer values. In that regard I do find systemd a lot more sane
Yes, but the difference I was implying is that (at least on my systems), crontabs start with a comment reminding you of the syntax with a few examples. That kind of approach is missing in systemd. Maybe a simple interactive create-service shell script could fix this issue.
Agree that the unit file language is not terribly good. In terms of strings and paths and stuff it’s just enough like bash to trip you up, and then you say “fuckit” and write a bash script that gets called by the unit file and now it’s no longer declarative. Its reading of environment variable files is similarly not-quite-what-you-expect. The combinations of options, dependencies and pre/post/during choices for units is not exactly the most elegant or intuitive, by a long shot. It works, but only once you understand every single option in isolation and can pick and choose the right ones. It is the exact opposite of the delight you feel when you say “okay, option X must exist, and it’s probably called X…” and discover you are correct.
I’m sort of neutral about systemd, but I do like the unit files. I have some of the same problems as you do when writing a unit file from scratch. Systemd is well-documented, but there are so many knobs that it can be hard to find exactly what you want. I think it is a tooling problem.
If I were working with systemd on a daily basis, I’d probably try my hand at writing a small unit file generator. I’ve used ansible to generate .service files from a Jinja template. That takes care of a lot of rote, boiler-plate stuff. My point is there are 90% solutions that would make things more pleasant.
Taking the same language and providing better infrastructure around creating them would suffice in 99% of cases. After all, runit would be enough for most people most of the time anyway.
I use runit on my Alpine machines; I had to write some custom services and starting runit with OpenRC is easier than dealing with those stupid scripts. It’s a bit bespoke and non-standard, but it works quite well.
This is, I guess, a personal plug, but I want to point out that my own init/service manager, “dinit” (https://davmac.org/projects/dinit/), builds and runs fine on Alpine and I even have a VM image that boots Alpine with dinit. It doesn’t yet support external control of service state as this post discusses (to allow integration with network state or device presence for example) but that has been on the TODO list for a while and is high priority (when I do get time to work on it again - paid work is taking all my energy right now).
That said, S6(-RC) is a really well-engineered piece of software and I can see why Alpine are interested in using it.
I would love it if some of the concepts of erlang supervisor trees are applied. Things like recovering from partial failure of a subsystem by progressive rollback and restart.
I’m not a huge systemd opponent, but the reason I dislike it enough to have an opinion are the unit files. The language is neither flexible nor intuitive enough to be guessed, nor do I use it often enough to memorize it. That means whenever I want to create a new service or timer or whatever, I have to deal with onscure documentation or copy-paste-tweak some code from arch wiki or stack overflow. A system like runit is attractive (though not technically superior) because all I need is to write my command in a file named “run”.
I hope that Alpine’s alternative will make improvments on this front, be it by providing better tooling or by improving the language. Ideally it should be able to parse systemd unit files but allow these to be degenerated as much as possible, allowing lazy users to only write as much as neccesary.
As a systemd appreciater, its unit files are like 90% of the reason why I like it. If Alpine manages to improve on that somehow, I’ll be happy. It’s highly probable that they won’t, though.
Both seem fine to me. I genuinely don’t understand the animosity.
BTW, I used to use Arch when the systemd change happened in the Linux world, so I may be influenced by how well we implemented that change. Or by how mediocre the Arch situation was before systemd ;) I also used to be a serious distro hopper, so I may have greater appreciation towards how systemd managed to rally the wagons, so to speak, and unify large parts of the Linux ecosystem that IMHO had no good reason for being different.
The language is neither flexible nor intuitive enough to be guessed, nor do I use it often enough to memorize it. That >means whenever I want to create a new service or timer or whatever, I have to deal with onscure documentation or >copy-paste-tweak some code from arch wiki or stack overflow.
I agree, although the systemd documentation isn’t terrible, but* there’s so much in it that navigating the whole thing is cumbersome.
I don’t really see this as a bad thing.
Also, wouldn’t the obvious choice be to pick up maintenance of OpenRC rather than writing something brand new that will need to be maintained?
There is nothing really desirable about openrc and it simply does not support the required features like supervision. Sometimes its better to start fresh, or in this case with the already existing s6/s6-rc which is build on a better design.
I’d say this is a matter of opinion, because there’s inherent value in simplicity and systemd isn’t simple.
But why compare the “simplicity” to systemd instead of something actually simple, openrcs design choices with its shell wrapping instead of a simple supervision design and a way to express dependencies outside of the shell script is a lot simpler. The daemontool like supervision systems simply have no boilerplate in shell scripts and provide good features like tracking pids without pid files and therefor reliably signaling the right processes, they are able to restart services if they get down and they provide a nice and reliable way to collect stdout/stderr logs of those services.
Edit: this is really what the post is about, taking the better design and making it more user friendly and implementing the missing parts.
the 4th paragraph
also, the second to last graf
so, they are continuing to maintain OpenRC while alpine still requires it, but it doesn’t meet their needs, hence they are designing something new
I was thinking the same thing.
I have no sources, but when was the last time OpenBSD or FreeBSD had a substantial change to their init systems?
I don’t know enough to know why there’s a need to iterate so I won’t comment on the quality of the changes or existing system.
To my knowledge, there’s serious discussion in the FreeBSD community about replacing their init system (for example, see this talk from FreeBSD contributor and previous Core Team member Benno Rice: The Tragedy of systemd).
And then there’s the FreeBSD-based Darwin, whose launchd is much more similar to systemd than to either BSD init or SysVinit to my knowledge.
This was well worth the watch/listen. Thanks for the link.
I believe the last major change on FreeBSD was adding the rc-order stuff (from NetBSD?) that allowed expressing dependencies between services and sorting their launch order so that dependencies were fulfilled.
That said, writing a replacement for the FreeBSD service manager infrastructure is something I’d really, really like to do. Currently
devd,inetd, andcronare completely separate things and so you have different (but similar) infrastructure for running a service:I really like the way that Launchd unifies these (though I hate the fact that it uses XML property lists, which are fine as a human-readable serialisation of a machine format, but are not very human-writeable). I’d love to have something that uses libucl to provide a nice composable configuration for all of these. I’d also like an init system that plays nicely with the sandboxing infrastructure on FreeBSD. In particular, I’d like to be able to manage services that run inside a jail, without needing to run a service manager inside the jail. I’d also like something that can set up services in Capsicum sandboxes with libpreopen-style behaviour.
Yep, The Design and Implementation of the NetBSD rc.d system, Luke Mewburn, 2000. One of the earlier designs of a post-sysvinit dependency based init for Unix.
I’ve been able to manage standalone services to run inside a jail, but it’s more than a little hacky. For fun a while back, I wrote a finger daemon in Go, so I could keep my PGP keys available without needing to run something written in C. This runs inside a bare-jail with a RO mount of the homedirs and not much else and lots of FS restrictions. So
jail.confended up with this in the stanza:and then the service file does
daemon -c jexec -u ${runtime_user_nonjail} ${jail_name} ${jail_fingerd} ...; the tricky bit was messing inside the internals ofrc.subrto make sure that pidfile management worked correctly, with the process finding handling that the jail is not “our” jail:It works, but I suspect that stuff like
$JIDcan change without notice to me as an implementation detail ofrc.subr. Something properly supported would be nice.I think the core issue is that desktops have very different requirements than servers. Servers generally have fixed hardware, and thus a hard-coded boot order can be sufficient.
Modern desktops have to deal with many changes like: USB disks being plugged in (mounting and unmounting), Wi-Fi going in and out, changing networks, multiple networks, Bluetooth audio, etc. It’s a very different problem
I do think there should be some “server only” init systems, and I think there are a few meant for containers but I haven’t looked into them. If anyone has pointers I’d be interested. Desktop is a complex space but I don’t think that it needs to infect the design for servers (or maybe I’m wrong).
Alpine has a mix of requirements I imagine. I would only use it for servers, and its original use case was routers, but I’m guessing the core devs also use it as their desktops.
I’m not a huge systemd opponent, but the reason I dislike it enough to have an opinion are the unit files. The language is neither flexible nor intuitive enough to be guessed, nor do I use it often enough to memorize it. That means whenever I want to create a new service or timer or whatever, I have to deal with onscure documentation or copy-paste-tweak some code from arch wiki or stack overflow. A system like runit is attractive (though not technically superior) because all I need is to write my command in a file named “run”.
I hope that Alpine’s alternative will make improvments on this front, be it by providing better tooling or by improving the language. Ideally it should be able to parse systemd unit files but allow these to be degenerated as much as possible, allowing lazy users to only write as much as neccesary.
I find the systemd documentation pretty straight-forward and very well written, so I’m surprised that you find it obscure :)
The man pages are my go to for anything related to systemd, and examples of all kinds of systemd units, including timers, are plentiful and already installed on my laptop:
Once in a while I find a systemd keyword used in units that I can’t find the documentation for, but luckily
mancan be used for this purpose too:I remember the #systemd channel on Freenode as pretty active, so if you’re stuck consider asking there.
Maybe obscure it the wrong word, but as I say, as someone who only deals with systemd directly once or twice a year,
is a first step I have to make every time.
Just a few months ago I wanted to create a timer, to replace a user cron job, but the number of concepts I have to read up on before I write anything seems like it’s not worth it.
Maybe this is just a mindset thing, *nix is certainly not obvious when you first start, but you gain an intuition over time and can start guessing how to do things. Systemd doesn’t play into this intuition.
Completely understand what you mean – the increasing complexity of computers and the layers upon layers of abstraction is definitely an issue. I deal with systemd unit files fairly often, so I’ve internalized much of the knowledge and feel that systemd units behave pretty much like I expect them to.
Much of it probably comes down to systemd simply being different from what we’ve used in the past – sysv style init, cron daemons etc. However even though I’ve used Linux for almost two decades and I’ve written plenty of cron jobs I still find myself looking at the man page to decipher the cron timer values. In that regard I do find systemd a lot more sane – the unit files themselves are much more readable than both cron jobs and sysv style init scripts, not at least because they’re more verbose, and the documentation is really good.
Yes, but the difference I was implying is that (at least on my systems), crontabs start with a comment reminding you of the syntax with a few examples. That kind of approach is missing in systemd. Maybe a simple interactive
create-serviceshell script could fix this issue.Agree that the unit file language is not terribly good. In terms of strings and paths and stuff it’s just enough like bash to trip you up, and then you say “fuckit” and write a bash script that gets called by the unit file and now it’s no longer declarative. Its reading of environment variable files is similarly not-quite-what-you-expect. The combinations of options, dependencies and pre/post/during choices for units is not exactly the most elegant or intuitive, by a long shot. It works, but only once you understand every single option in isolation and can pick and choose the right ones. It is the exact opposite of the delight you feel when you say “okay, option X must exist, and it’s probably called X…” and discover you are correct.
I’m sort of neutral about systemd, but I do like the unit files. I have some of the same problems as you do when writing a unit file from scratch. Systemd is well-documented, but there are so many knobs that it can be hard to find exactly what you want. I think it is a tooling problem.
If I were working with systemd on a daily basis, I’d probably try my hand at writing a small unit file generator. I’ve used ansible to generate .service files from a Jinja template. That takes care of a lot of rote, boiler-plate stuff. My point is there are 90% solutions that would make things more pleasant.
Taking the same language and providing better infrastructure around creating them would suffice in 99% of cases. After all, runit would be enough for most people most of the time anyway.
I use runit on my Alpine machines; I had to write some custom services and starting runit with OpenRC is easier than dealing with those stupid scripts. It’s a bit bespoke and non-standard, but it works quite well.
This is, I guess, a personal plug, but I want to point out that my own init/service manager, “dinit” (https://davmac.org/projects/dinit/), builds and runs fine on Alpine and I even have a VM image that boots Alpine with dinit. It doesn’t yet support external control of service state as this post discusses (to allow integration with network state or device presence for example) but that has been on the TODO list for a while and is high priority (when I do get time to work on it again - paid work is taking all my energy right now).
That said, S6(-RC) is a really well-engineered piece of software and I can see why Alpine are interested in using it.
I love the idea of having a competitor for systemd. Would be very interesting to see.
I would love it if some of the concepts of erlang supervisor trees are applied. Things like recovering from partial failure of a subsystem by progressive rollback and restart.
How do you imagine that working?
I’m not a huge systemd opponent, but the reason I dislike it enough to have an opinion are the unit files. The language is neither flexible nor intuitive enough to be guessed, nor do I use it often enough to memorize it. That means whenever I want to create a new service or timer or whatever, I have to deal with onscure documentation or copy-paste-tweak some code from arch wiki or stack overflow. A system like runit is attractive (though not technically superior) because all I need is to write my command in a file named “run”.
I hope that Alpine’s alternative will make improvments on this front, be it by providing better tooling or by improving the language. Ideally it should be able to parse systemd unit files but allow these to be degenerated as much as possible, allowing lazy users to only write as much as neccesary.
As a systemd appreciater, its unit files are like 90% of the reason why I like it. If Alpine manages to improve on that somehow, I’ll be happy. It’s highly probable that they won’t, though.
But do you like the concept or the way it is implemented?
Both seem fine to me. I genuinely don’t understand the animosity.
BTW, I used to use Arch when the systemd change happened in the Linux world, so I may be influenced by how well we implemented that change. Or by how mediocre the Arch situation was before systemd ;) I also used to be a serious distro hopper, so I may have greater appreciation towards how systemd managed to rally the wagons, so to speak, and unify large parts of the Linux ecosystem that IMHO had no good reason for being different.
I agree, although the systemd documentation isn’t terrible, but* there’s so much in it that navigating the whole thing is cumbersome.
I’m glad to hear Laurent is on board with this. He’s a true expert in this field.
The email link at the end of that article is broken, it isn’t a
mailto:scheme. Though that might be on purpose…