This can’t be stated strong enough.
I recently had to fix a vendor provided (not from the particular software in question) init script that spawned a daemon that inherited file descriptors it shouldn’t have. There are simply too many ways to just pile on spaghetti that can have strange effects. There were other maintenance tasks that required rewrites of most of these scripts just because critical software changes: going from udevd to eudev.
I recently took a look at some software a customer wanted installed on all their Linux servers. It had an undocumented upgrade function that fetched and installed software without any checksums and without verifying HTTPS certificates, the upgrade function was not disabled even when explicitly using the somewhat documented flag to disallow upgrades. It dropped lines in /etc/rc.d/rc.local, it spawned itself during postinst scripts outside of the init system, and trying to upgrade the package resulted in the software deleting itself. The postinst script also looks for a line in /etc/rc.d/rc.local that suggests that they previously modified /etc/ssh/sshd_config on install…
Best part: that particular piece of software has some huge partners. To name a few: AWS, Capgemini, Microsoft Azure, Google Cloud, Infosys, Tata. Apparently none of the partners ever took a look at the package…
As a long time systemd fanboy and heavy user, I learned a lot of things! systemd-analyze security being at the top of the list.
One thing not mentioned are systemd timers which allows for very easy debuugging (when will the next cron run?) but also all other systemd goodies (= resource limits, privatetmp, etc.)
I like how systemd brings all these features, but I don’t like how this makes this not portable to other operating systems, as systemd only supports Linux. I know that not all operating systems support all the underlying features needed by systemd, but I believe it is a shame to be Linux-centric.
I am not a user of non Linux-based operating systems myself, but I prefer having common standards.
Personally, I’m completely fine that Systemd-the-init-system is Linux-only. It’s essentially built around cgroups, and I can imagine reimplementing everything cgroups-like on top of whatever FreeBSD offers would be extremely challenging if at all possible. FreeBSD can build its own init system.
…However, I would prefer if systemd didn’t work to get other software to depend on systemd. It definitely sucks that systemd has moved most desktop environments from being truly cross platform to being Linux-only with a hack to make them run on the BSDs. That’s not an issue with the init system being Linux-only though, it’s an issue with the scope and political power of the systemd project.
The issue is that it’s expensive to maintain things like login managers and device notification subsystems, so if the systemds of the world are doing it for free, that’s a huge argument to take advantage of it. No political power involved.
With politcal power I just meant that RedHat and Poettering have a lot of leverage. If I, for example, made a login manager that’s just as high quality as logind, I can’t imagine GNOME would switch to supporting my login manager, especially as the only login manager option. (I suppose we’ll get to test that hypothesis though by seeing whether GNOME will ever adopt seatd/libseat as an option.)
It’s great that systemd is providing a good login manager for free, but I can’t shake the feeling that, maybe, it would be possible to provide an equally high quality login daemon without a dependency on a particular Linux-only init system.
I don’t think the “political power” (call it leverage if you disagree with that term) of the systemd project is inherently an issue, but it becomes an issue when projects add a hard dependency on systemd tools which depend on the systemd init system where OS-agnostic alternatives exist and are possible.
Systemd is built on Linux’s capabilities, so this is really a question of–should people not try to take advantage of platform-specific capabilities? Should they always stay stuck on the lowest-common denominator? This attitude reminds me of people who insist on treating powerful relational databases like dumb key-value stores in the name of portability.
I believe the BSDs can do many of the things listed in the article, but also in their very own ways. A cross-platform system manager would be some sort of a miracle, I believe.
The big difference is that systemd (as well as runit, s6, etc.) stay attached to the process, whereas the BSD systems (and OpenRC, traditional Linux init scripts) expect the program to “daemonize”.
Aside from whatever problems systemd may or may not have, I feel this model is vastly superior in pretty much every single way. It simplifies almost everything, especially for application authors, but also for the init implementation and system as a whole.
A cross-platform system manager would be some sort of a miracle, I believe.
daemontools already ran on many different platforms around 2001. I believe many of its spiritual successors do too.
It’s not that hard; like many programs it’s essentially a glorified for loop:
for service in get_services()
start_process(service)
Of course, it’s much more involved with restarts, logging, etc. etc. but you can write a very simple cross-platform proof-of-concept service manager in a day.
Yes and no. Socket activation can be done with inetd(8), and on OpenBSD you can at least limit what filesystem paths are available with unveil(2), although that requires system-specific changes to your code. As far as dynamic users, I don’t think there’s a solution for that.
Edit: Also, there’s no real substitute for LoadCredentials, other than using privdropping and unveil(8). I guess you could use relayd(8) to do TLS termination and hand-off to inetd(8). If you’re doing strictly http, you could probably use a combo of httpd(8) and slowcgi(8) to accomplish similar.
Then I’m imagining a modular system with different features that can be plugged together, with specifications and different implementations depending to the OS. Somehow a way to go back to having a single piece of software for each feature, but at another level. The issue is how you write these specifications while having things implementable on any operating system it makes sense of.
Hell, a Docker API implementation for BSD would be a miracle. The last FreeBSD Docker attempt was ancient and has fallen way out of date. Have a daemon that could take OCI containers and run them with ZFS layers in a BSD jail with BSD virtual networks would be a huge advantage for BSD in production environments.
There is an exciting project for an OCI-compatible runtime for FreeBSD: https://github.com/samuelkarp/runj. containerd has burgeoning FreeBSD support as well.
In fact aside from the XML I’d say SMF is the kind of footprint I’d prefer systemd to have, it points to (and reads from) log files instead of subsuming that functionality, handles socket activation, supervises processes/services and drops privileges. (It can even run zones/jails/containers).
But to answer the question: yes any of the scripts can be used essentially* verbatim on any other platform.
(There might be differences in pathing, FreeBSD installs everything to /usr/local by default)
Absolutely not. Even though they’re just shell scripts, there are a ton of different concerns that make them non-portable.
I’m gonna ignore the typical non-portable problems with shell scripts (depending on system utils that function differently on different systems (yes, even within the BSDs), different shells) and just focus on the biggest problem: both are written depending on their own shell libraries.
If we look at a typical OpenBSD rc.d script, you’ll notice that all the heavy-lifting is done by /etc/rc.d/rc.subr. FreeBSD has an /etc/rc.subr that fulfills the same purpose.
It’s also important to note that trying to wholesale port rc.subr(8) into your init script to make it compatible across platforms will be quite the task, since they’re written for different shells (OpenBSD ksh vs whatever /bin/sh is on FreeBSD). Moreover, the rc.subr(8) use OS-specific features, so porting them wholesale will definitely not work (just eyeballing the OpenBSD /etc/rc.d/rc.subr, I see getcap(1) and some invocations of route(8) that only work on OpenBSD. FreeBSD’s /etc/rc.subr uses some FreeBSD-specific sysctl(8) MIBs.)
If you’re writing an rc script for a BSD, it’s best to just write them from scratch for each OS, since the respective rc.subr(8) framework gives you a lot of tools to make this easy.
This is notably way better than how I remember the situation on sysvinit Linux, since iirc there weren’t such complete helper libraries, and writing such a script could take a lot of time and be v error-prone.
Yeah, exactly. The rc scripts aren’t actually portable, so why do people (even in this very thread) expect the systemd scripts (which FWIW are easier to parse programmatically, see halting theory) to be?
I’m completely in agreement with you. I want rc scripts/unit files/SMF manifests to take advantage of system-specific features. It’s nice that an rc script in OpenBSD can allow me to take advantage of having different rtables or that it’s jail-aware in FreeBSD.
I think there are unfortunate parts of this, since I think it’d be non-trivial to adapt provided program in this example to socket activation in inetd(8) (tbh, maybe I should try when I get a chance). What would be nice is if there was a consistent set of expectations for daemons about socket-activation behavior/features, so it’d be easier to write portable programs, and then ship system-specific configs for the various management tools (systemd, SMF, rc/inetd). Wouldn’t be surprised if that ship has sailed though.
“Avoiding complexity with a large has-been-an-init system that absorbs roughly half of your daemons and prefers binary logs to actually greppable logs”.
The binary log thing is a fair point, but “absorbs roughly half of your daemons” implies that you then have roughly half as many standalone daemons interacting with each other. I think there’s a good case to be made that that’s a reduction in total system complexity, even if it means systemd, considered in isolation, is more complex.
But then again, if you have a lot of standalone daemons, it’s probably because they don’t depend on each other; they’re probably not as coupled as systemd daemons are. The coupling, I believe, is what leads to systemd having more complexity because decoupling makes the whole system more modular. [1]
While I am not a systemd fan, I have to say this was a very useful and well written article.
The
loadcredentialargument will be especially useful to me, thanks!I’m so glad we’ve moved past trying to do all of this with shell scripts.
This can’t be stated strong enough. I recently had to fix a vendor provided (not from the particular software in question) init script that spawned a daemon that inherited file descriptors it shouldn’t have. There are simply too many ways to just pile on spaghetti that can have strange effects. There were other maintenance tasks that required rewrites of most of these scripts just because critical software changes: going from udevd to eudev.
I cannot stand init scripts.
I recently took a look at some software a customer wanted installed on all their Linux servers. It had an undocumented upgrade function that fetched and installed software without any checksums and without verifying HTTPS certificates, the upgrade function was not disabled even when explicitly using the somewhat documented flag to disallow upgrades. It dropped lines in
/etc/rc.d/rc.local, it spawned itself duringpostinstscripts outside of the init system, and trying to upgrade the package resulted in the software deleting itself. Thepostinstscript also looks for a line in/etc/rc.d/rc.localthat suggests that they previously modified/etc/ssh/sshd_configon install…Best part: that particular piece of software has some huge partners. To name a few: AWS, Capgemini, Microsoft Azure, Google Cloud, Infosys, Tata. Apparently none of the partners ever took a look at the package…
I have a lot of problems with systemd, with many articles showing how to do simple things in a complex way.
This is the first article I’ve read showing actual examples of complex things made simple with systemd, and appreciate it.
As a long time
systemdfanboy and heavy user, I learned a lot of things!systemd-analyze securitybeing at the top of the list.One thing not mentioned are systemd timers which allows for very easy debuugging (when will the next cron run?) but also all other systemd goodies (= resource limits, privatetmp, etc.)
That was a great read. Thanks.
I like how systemd brings all these features, but I don’t like how this makes this not portable to other operating systems, as systemd only supports Linux. I know that not all operating systems support all the underlying features needed by systemd, but I believe it is a shame to be Linux-centric.
I am not a user of non Linux-based operating systems myself, but I prefer having common standards.
Personally, I’m completely fine that Systemd-the-init-system is Linux-only. It’s essentially built around cgroups, and I can imagine reimplementing everything cgroups-like on top of whatever FreeBSD offers would be extremely challenging if at all possible. FreeBSD can build its own init system.
…However, I would prefer if systemd didn’t work to get other software to depend on systemd. It definitely sucks that systemd has moved most desktop environments from being truly cross platform to being Linux-only with a hack to make them run on the BSDs. That’s not an issue with the init system being Linux-only though, it’s an issue with the scope and political power of the systemd project.
The issue is that it’s expensive to maintain things like login managers and device notification subsystems, so if the systemds of the world are doing it for free, that’s a huge argument to take advantage of it. No political power involved.
With politcal power I just meant that RedHat and Poettering have a lot of leverage. If I, for example, made a login manager that’s just as high quality as logind, I can’t imagine GNOME would switch to supporting my login manager, especially as the only login manager option. (I suppose we’ll get to test that hypothesis though by seeing whether GNOME will ever adopt seatd/libseat as an option.)
It’s great that systemd is providing a good login manager for free, but I can’t shake the feeling that, maybe, it would be possible to provide an equally high quality login daemon without a dependency on a particular Linux-only init system.
I don’t think the “political power” (call it leverage if you disagree with that term) of the systemd project is inherently an issue, but it becomes an issue when projects add a hard dependency on systemd tools which depend on the systemd init system where OS-agnostic alternatives exist and are possible.
Everybody loves code that hasn’t been written yet. I think we need to learn to looks realistically at what we have now (for free, btw) instead of insisting on the perfect, platform-agnostic software. https://lobste.rs/s/xxyjxl/avoiding_complexity_with_systemd#c_xviza7
Systemd is built on Linux’s capabilities, so this is really a question of–should people not try to take advantage of platform-specific capabilities? Should they always stay stuck on the lowest-common denominator? This attitude reminds me of people who insist on treating powerful relational databases like dumb key-value stores in the name of portability.
I believe the BSDs can do many of the things listed in the article, but also in their very own ways. A cross-platform system manager would be some sort of a miracle, I believe.
The big difference is that systemd (as well as runit, s6, etc.) stay attached to the process, whereas the BSD systems (and OpenRC, traditional Linux init scripts) expect the program to “daemonize”.
Aside from whatever problems systemd may or may not have, I feel this model is vastly superior in pretty much every single way. It simplifies almost everything, especially for application authors, but also for the init implementation and system as a whole.
daemontools already ran on many different platforms around 2001. I believe many of its spiritual successors do too.
It’s not that hard; like many programs it’s essentially a glorified for loop:
Of course, it’s much more involved with restarts, logging, etc. etc. but you can write a very simple cross-platform proof-of-concept service manager in a day.
Yes and no. Socket activation can be done with
inetd(8), and on OpenBSD you can at least limit what filesystem paths are available withunveil(2), although that requires system-specific changes to your code. As far as dynamic users, I don’t think there’s a solution for that.Edit: Also, there’s no real substitute for
LoadCredentials, other than using privdropping andunveil(8). I guess you could userelayd(8)to do TLS termination and hand-off toinetd(8). If you’re doing strictly http, you could probably use a combo ofhttpd(8)andslowcgi(8)to accomplish similar.Then I’m imagining a modular system with different features that can be plugged together, with specifications and different implementations depending to the OS. Somehow a way to go back to having a single piece of software for each feature, but at another level. The issue is how you write these specifications while having things implementable on any operating system it makes sense of.
Hell, a Docker API implementation for BSD would be a miracle. The last FreeBSD Docker attempt was ancient and has fallen way out of date. Have a daemon that could take OCI containers and run them with ZFS layers in a BSD jail with BSD virtual networks would be a huge advantage for BSD in production environments.
There is an exciting project for an OCI-compatible runtime for FreeBSD: https://github.com/samuelkarp/runj. containerd has burgeoning FreeBSD support as well.
But, are FreeBSD rc.d scripts usable verbatim on, say, OpenBSD or SMF?
SMF is a lot more like systemd than the others.
In fact aside from the XML I’d say SMF is the kind of footprint I’d prefer systemd to have, it points to (and reads from) log files instead of subsuming that functionality, handles socket activation, supervises processes/services and drops privileges. (It can even run zones/jails/containers).
But to answer the question: yes any of the scripts can be used essentially* verbatim on any other platform.
(There might be differences in pathing, FreeBSD installs everything to /usr/local by default)
I wish SMF was more portable. I actually like it a lot.
Absolutely not. Even though they’re just shell scripts, there are a ton of different concerns that make them non-portable.
I’m gonna ignore the typical non-portable problems with shell scripts (depending on system utils that function differently on different systems (yes, even within the BSDs), different shells) and just focus on the biggest problem: both are written depending on their own shell libraries.
If we look at a typical OpenBSD rc.d script, you’ll notice that all the heavy-lifting is done by
/etc/rc.d/rc.subr. FreeBSD has an/etc/rc.subrthat fulfills the same purpose.These have incredibly different interfaces for configuration, you can just take a look at the manpages: OpenBSD rc.subr(8), FreeBSD rc.subr(8). I don’t have personal experience here, but NetBSD appears to have a differing rc.subr(8) as well.
It’s also important to note that trying to wholesale port
rc.subr(8)into your init script to make it compatible across platforms will be quite the task, since they’re written for different shells (OpenBSD ksh vs whatever/bin/shis on FreeBSD). Moreover, therc.subr(8)use OS-specific features, so porting them wholesale will definitely not work (just eyeballing the OpenBSD/etc/rc.d/rc.subr, I seegetcap(1)and some invocations ofroute(8)that only work on OpenBSD. FreeBSD’s/etc/rc.subruses some FreeBSD-specificsysctl(8)MIBs.)If you’re writing an rc script for a BSD, it’s best to just write them from scratch for each OS, since the respective
rc.subr(8)framework gives you a lot of tools to make this easy.This is notably way better than how I remember the situation on sysvinit Linux, since iirc there weren’t such complete helper libraries, and writing such a script could take a lot of time and be v error-prone.
Yeah, exactly. The rc scripts aren’t actually portable, so why do people (even in this very thread) expect the systemd scripts (which FWIW are easier to parse programmatically, see halting theory) to be?
Also, thank you for the detailed reply.
I’m completely in agreement with you. I want rc scripts/unit files/SMF manifests to take advantage of system-specific features. It’s nice that an rc script in OpenBSD can allow me to take advantage of having different rtables or that it’s jail-aware in FreeBSD.
I think there are unfortunate parts of this, since I think it’d be non-trivial to adapt provided program in this example to socket activation in
inetd(8)(tbh, maybe I should try when I get a chance). What would be nice is if there was a consistent set of expectations for daemons about socket-activation behavior/features, so it’d be easier to write portable programs, and then ship system-specific configs for the various management tools (systemd, SMF, rc/inetd). Wouldn’t be surprised if that ship has sailed though.I don’t see why not? They’re just POSIX sh scripts.
“Avoiding complexity with a large has-been-an-init system that absorbs roughly half of your daemons and prefers binary logs to actually greppable logs”.
Well…
I’m pretty happy with
journalctl | grep. Have you heard of unix pipes? They’re a pretty powerful tool for composing software…The binary log thing is a fair point, but “absorbs roughly half of your daemons” implies that you then have roughly half as many standalone daemons interacting with each other. I think there’s a good case to be made that that’s a reduction in total system complexity, even if it means systemd, considered in isolation, is more complex.
But then again, if you have a lot of standalone daemons, it’s probably because they don’t depend on each other; they’re probably not as coupled as systemd daemons are. The coupling, I believe, is what leads to systemd having more complexity because decoupling makes the whole system more modular. [1]
[1]: https://judecnelson.blogspot.com/2014/09/systemd-biggest-fallacies.html, see Fallacy #1.