My view when systemd first crossed my radar (around the time Debian was making it’s decision about supporting/adopting systemd or not), I of course saw a lot of commentary about the negative aspects of the project.
I’d imagine those criticism are still true: it seems unlikely the people involved in this have softened their views or language.
But what has changed is my appreciation for the technical benefits of systemd as an init manager (and a few related tasks, e.g. the timers stuff to replace cron). I’m not that thrilled about the ever-increasing scope of the project as a whole, but I don’t use Linux as a desktop OS, so I think I’m shielded from a lot of the impact it has on e.g. Gnome etc.
I tried (and eventually succeeded, from memory) to write an init script for a process, using the Debian provided ‘start-stop-daemon’ and an existing init file as a guide. This was not a productive or enjoyable experience for me, at all. I’d say I’m quite comfortable with shell as a scripting language in general, but the idiosyncrasies of producing a reliable init script were confusing and non obvious to me at the time.
These days, I don’t think twice about either creating new systemd units, or using drop-ins to customise existing ones - does everything always work the first time? No. Is it an order of magnitude easier than a sysvinit script (IMO) to identify why it doesn’t do what you expect, and find a solution that will do what you expect? Generally yes.
This is the thing I’ve loved most about systemd. Writing an init script for one distribution is a PITA. Writing one that works on multiple different linux distros makes me tear my effing hair out. The complexity of writing a shell script that conforms to certain parameters with a wide variety of “helpers” seems out of whack with what I want to do: ie: fire up a process and leave it in the background.
Also, an init system that cannot track what child processes it has started, and needs a complicated system of pidfile management to control the process that doesn’t even work in many cases (it breaks as soon as the machine or process crashes, or gets killed without the proper signal being sent beforehand), is barely able to call itself “an init system”. It’s just what we always had, so in the proper UNIX way, everyone just dealt with it and nobody pointed out the big elephant in the room, in that it sucked hard.
Now. systemd has it’s problems, and over-reach is definitely in there. I can’t count the number of times that I’ve had to figure out the “new hotness” way of configuring a GD static network interface, and every time I look at it systemd has new capabilities that it probably shouldn’t have. ( Following jwz’s maxim, I can’t wait for the systemd mail client), but pretending sysvinit was all good was insanity.
It always bothers me a bit when systemd is compared to just init scripts, because those are not the only two options available (and never have been). I don’t really blame the casual Linux user like you, but on more than a few occasions the systemd people used “well, in init scripts it sucks even worse” as fallacious defence of their own shortcoming (you don’t need to rant, rave, and insult to be toxic).
The only commonly used init system in Linux, used by all the major distros, was sysvinit. Yes, there was OpenRC, and upstart, and others, but none of them were ever widely adopted. Yes, Ubuntu developed upstart and it solved some of the problems in sysvinit, but was never widely adopted, so if you were writing a daemon for anything other than ubuntu, it was back to sysvinit, and all the pain that entailed. It’s great that there are other init systems that solve the problem, but if none of them are ever adopted, there’s really no value in it for people who write daemons.
OpenRC is essentially just a continuation of sysV init, with all problems with it. I actually just have OpenRC start runit on my Alpine system, as that’s just easier than writing those scripts. I never looked at upstart.
daemontools and the various projects it inspired (runit, s6, freedt, a few others) always seemed like a good solution to me. It solves most of the issues from SysV init and gives 90% of systemd for most use cases, but in a much smaller package. I don’t know why it was never widely adopted; perhaps it was timing, lack of documentation, lack of advocacy, or something else.
For daemontools, it’s likely the fact that djb has a tendency to release a project and then immediately stop maintaining it, so there becomes a slurry of patches with no canonical blessed set for distros to maintain, and no upstream merging them.
Yeah, that partly explains daemontools; there’s also the whole license issue, which didn’t become clear until 2007 (when djb finally released his stuff as public domain; before that he rejected that a license was needed at all; also a problem with qmail, djbdns) which is why people needed to maintain patches instead of a “daemontools-maintianed” repo or whatnot.
But runit stems from 2004, and there are other daemontools-like systems from the same time period. So I don’t think that djb’s idiosyncrasy fully explain it.
Ah, daemontools. I can answer that one. Daemontools started as a project by Dan J Bernstein (hereafter referred to as djb), who is a brilliant mathematician and cryptologist, but the license for some of his projects, including daemontools is … troubling to a lot of the distributions, and he can be, well, prickly, to put it politely.
Wikipedia has the license as Public Domain, but ISTR that if you didn’t distribute it precisely as was specified he would complain a lot. It meant that a lot of distributions that wanted to use his software shied away from it.
It’s pretty awesome, but it still doesn’t solve a lot of the pain points that systemd solves.
This is all vague rememberings from about 20 years ago, but having dealt with djb in the past, I can definitely understand the reticence on the part of distributions to package his software, or rely on it for critical functionality. He makes Lennart look like a softy in comparison.
Even if you were just targeting Ubuntu, upstart was pretty buggy and feature deficient. You could wedge it so bad that you had to reboot while iterating on a job. I’m pretty happy that systemd takes a features-welcome approach. Every time I think that I’ll have to use a shell stub I can find what I need with some man page searching.
Right, I’m aware there are other options out there, and you’re right they should be part of the conversation when a distro is making decisions about what init to support in its packages.
But, for someone using a distro, even someone writing packages for a distro, you need to support what the distro supports. Yes you could also support extra init systems but you need to support what the distro supports.
I’ve had all sorts of things -1’d as “incorrect” or “troll” where I really struggle seeing how either can apply. Write a lengthy reply going out of my way to make sure it’s nuanced: -2 troll… Great, thanks 😒 Like you said: welcome to the internet 🤷♂️
I’ve noticed this as well. Lobsters has always had explicit downvote reasons to discourage downvoting for silly things like “I disagree” or “your opinion clashes with my worldview and I’m salty.” But as we’ve grown I see those kinds of downvotes more and more.
Systemd definitely achieves several good things, but it has technical, social and political faults. It is an improvement over init scripts, but it is also many other things — and some of those are strict regressions (I am thinking specifically of how it wantonly breaks nohup). Is systemd-as-it-is an improvement over a better system which doesn’t exist? I really don’t know.
The kill-user-processes-on logout thing is no doubt annoying the first time you encounter it, but I’d suggest that it’s at worst a case of incorrect defaults (I honestly don’t even know what the upstream or distro defaults are).
I’m certain it has technical/approach faults, but in the grand scheme of things that one is pretty easy to ‘solve’.
One can no more dissuade a visionary of this kind than one can dissuade a member of the Fabian Society from the virtues of global humanitarian government, but then neither will the vox populi of provincial yokels be of any use in countering it. One can only stoically resign to the pull of inexorable necessity.
With the help of an Internet search query or two, and a good deal of re-reading, I could probably figure out what this means. But I am le tired. And I haven’t even gotten through the first chapter.
This brings up a different controversy: Is writing like this useful to stretch peoples minds, help them grow? Or does it just discourage people and make them feel dumb? Is writing a form of art, or a means of communication, or both?
From my perspective, it looks more focused on artistry than actually communicating and transferring knowledge. Which isn’t necessarily a bad thing.
From my perspective, it looks more focused on artistry than actually communicating and transferring knowledge.
Your perspective may change if you finish reading the document before dashing off to post a comment about the style instead of the substance of the source material. :)
It goes on to give a lot of historical context on previous attempts and on systems issues.
That’s the point though - I’d happily read the whole thing, if I had the mental capacity to do so. I’m sure there’s plenty of good info in there, but it’s buried under so much prose that I feel like I need a college degree to begin understanding it.
The point that you mentioned is a local maximum in terms of hard-to-read prose, and (from my reading up to section 2.3) I believe it is a global maximum too - the text is much nicer almost from that passage onward and I only found one other spot that I believe to be particularly egregious. I encourage you to keep going! I very much dislike the Unix culture and tooling in general, but found this article to be fascinating anyway.
Not every reader is native English speaker (and reader). Reading this style is very demanding and makes it harder to focus on the main topic. I have no idea if there are expressions hidden that I don’t know yet, or if a reference in the text is key to the final point.
You make the mistake of implying that there’s an objective way to communicate clearly and it’s not instead dependent on the listeners.
Most tech documentations, even when assuming no previous knowledge from the reader, would be unreadable for most people for the sole reason of their structure and dryness.
You’re witnessing something written by a person that is on the boundary between different worlds and can merge them into a single piece of writing and this confuses you. It’s normal and probably the author did it deliberately.
Lot of us programmers on the left are keen in rupturing the cultural and social bubble in which the tech sector is entrenched and I wouldn’t be surprised if the author wrote the piece in this way to deliberately challenge the stylistic elements of engineer-oriented writing.
I agree, but I think it’s just interesting that the author seems to have raised the educational bar, not lowered it. Now you have to be an engineer AND a literary genius to understand what’s being written.
And I’m not saying this is necessarily a bad thing. I’m just noting that it’s unapproachable for people like me. I’m sure after a bit more education and time, I’ll appreciate it too.
You don’t have to be a genius to understand that paragraph, just literate. I understood it just fine and the highest formal education I’ve received is community college.
Anyone who works in software is accustomed to googling tech jargon they’re unfamiliar with and learning as they go. I think it is interesting that you balk at doing this in a humanities context and perhaps that was the point.
I think you hit the nail on the head. It sounds to me like you’ve put in the hard work to understand and appreciate this kind of writing, and that is truly awesome (no sarcasm there). For me, I could totally figure this out, no doubt about it. But it would still be quite the slog, because I haven’t invested quite as much hard work as you have into the humanities yet. And that’s ok.
That’s why I say this “isn’t necessarily a bad thing.” I could use some more development in my humanities skills! But if the goal is to communicate ideas, then writing like this will only communicate ideas to a gifted few who are good at both engineering and humanities. Again: Not necessarily a bad thing.
Lot of us programmers on the left are keen in rupturing the cultural and social bubble in which the tech sector is entrenched
And yet you at the same time alienate readers who’ve never studied literature in English, nor English literature. More often than not from second and third world countries.
We write in different ways to reach different audiences in different ways. I never said everything should be written this way. There’s a growing idea of writing more propaganda disguised as tech opinion pieces using a language that programmers like. I mean, the far-right has been doing that for three decades now, it’s time for the left to react.
The “poking” at the bubble is complementary and clearly doesn’t aim at reaching the masses, but just at bringing out those that have the potential to connect with those readings.
Also I’m not American, I’m not an English native speaker and in “second and third world countries” humanities are usually valued a lot more than in the protestant anglosphere, so I don’t get your point.
This is the “English (Literary)” locale where it is assumed you know your leftist lore (and French philosophy). Not my thing but sometimes the references lead to interesting wikipedia pages.
This brings up a different controversy: Is writing like this useful to stretch peoples minds, help them grow? Or does it just discourage people and make them feel dumb? Is writing a form of art, or a means of communication, or both?
When I was younger I thought that impressing people with my vocabulary is the utmost purpose of my writing, but as years have gone on I have realized that this is just mental peacocking (to use a friendly term). The real greatness is to convey information (or even humor) using simple language while staying informative and engaging. I have been way more impressed with authors being able to write like this than people who just show they own a thesaurus.
This brings up a different controversy: Is writing like this useful to stretch peoples minds, help them grow? Or does it just discourage people and make them feel dumb? Is writing a form of art, or a means of communication, or both?
I would argue the former. You’ve learned a little more about the world by virtue of looking up a literary or cultural reference you didn’t previously understand. I’ve done this with plenty of texts I’ve read, in English and particularly in other languages (English is my cradletongue). Of course you weren’t obligated to do so - you could’ve decided that this article about systemd written using these kinds of literary references wasn’t worth your time to understand, and if enough people thought similarly, it would lose a substantial portion of its potential audience. But using literary and cultural references is something that writers and speakers do in pretty much every genre of human communication; and it’s only because this author happened to choose ones that you were unfamiliar with, that you found it something worth remarking on.
I found it really funny. To me, that kind of descriptiveness adds humor, but maybe I’m the only one? Like putting an eccentric accent on a movie character.
I think the article would be much longer if it wasn’t for the use of various literary devices for compression.
Maybe that would be for the better as more people would benefit from reading it.
On the other hand, it’s useful to have someone shine a light on gaps in your knowledge of things which used to be well known. Certainly I don’t think that a lot of the things referred to in the writing should be particularly exotic, yet sadly they are.
Finally, I personally get some enjoyment from reading things like these. It’s the kind of enjoyment I get from reading old books. Difficult to read but full of very interesting ideas.
Is writing like this useful to stretch peoples minds
While I do not particularly like the style of this text, I appreciate any kind of writing style that does not resemble the current “ted-talk” style uncultured vomit that is becoming the style of almost everybody today.
I”m less offended by the use of some obscure terms and references than by the implicit comparison of a Gnome maintainer to a follower of Stalin:
And if we are to take the “revolution OS” metaphor further, then Bassi’s position is not unlike Stalin’s defense of the need of a vanguard party in The Foundations of Leninism (1924), with those opposed consequently in the role of Trotskyites, Zinovievites and ultra-leftists: “The theory of worshipping spontaneity is decidedly opposed to giving the spontaneous movement a politically conscious, planned character. It is opposed to the Party marching at the head of the working class, to the Party raising the masses to the level of political consciousness, to the Party leading the movement; it is in favour of the politically conscious elements of the movement not hindering the movement from taking its own course; it is in favour of the Party only heeding the spontaneous movement and dragging at the tail of it.”
Arguably, in many historical cases, such ways of expression were used to hide the real meaning, a sort of emphemization. It was especially used when it came to criticizing a point of view, to filter out as many people as possible to not bear the consequences of saying clearly what you intended to say. The more fluff and ambiguity the more you can hide behind it.
You can’t dissuade visionaries like this, and the popular bloggers won’t help you counter them either. All you can do is give in [and use the visionaries’ stuff?]
Without the context of the surrounding paras I don’t really know what they were getting at.
I think people write like this because it’s fun for the writer, not necessarily for the reader ;)
Edit: the context is:
The Fabian Society is a British internationalist lefty think tank. Because they’re internationalist and lefty (but not anarchists), they’d probably really like a global humanitarian government.
“vox populi” is latin for “voice of the people” and sometimes means the “opinion of the people”, but that bit is a bit clumsy anyway.
The technical review is more damning than Poettering’s dramatics. Nondeterministic transactions? Features preserved because backwards compat but provide a massive footgun? Use of “reboot blocker” tasks? Wow!
It strikes me that systemd has some real architectural mistakes and needs to bite the bullet and redesign the inconsistencies, nondeterminisms. Seeing all that, I practically thought “maybe using SQLite would’ve resolved a lot of state tracking/migration pathing”
For what it is worth, SMF (the Solaris service management thing) uses sqlite for its storage. SMF has its own set of issues, but lack of atomicity isn’t one.
Does anyone know of a good book that treats systemd somewhat comprehensively?
At this point, I’ve decided my feelings about systemd don’t matter; it’s clearly here to stay. System administration is not my day job, so I’ve been able to get by with only minimal and superficial knowledge of it. I’ll have to re-learn a lot of tasks that I knew how to do (or could at least figure out how to do) using the previous init systems.
Whenever I do, I find I’m always googling (or reading man pages) in frustration. It’s a very extensive system. Command names are long and non-obvious. That’s okay. After all, if I managed to learn git, I should be able to learn systemd.
But I’d like to do so at a leisurely pace, instead of scraping documentation together from (frequently severely outdated) blog posts and man pages.
So is there a good book that’s up-to-date? Or is Red Hat certification study material my best hope?
At this point, I’ve decided my feelings about systemd don’t matter; it’s clearly here to stay.
That’s not totally clear to me. The two biggest bits of the Linux ecosystem are Android and cloud deployments. Android does not use systemd, it uses its own thing. Cloud deployments increasingly use things like Kubernetes to deploy containers. They may use systemd on the oustide (does containerd depend on systemd yet?) but typically the containers don’t use systemd on the inside. Over time, I expect the things on the outside to be simplified and systemd is an obvious contender to go because it doesn’t add much value in this space.
Systemd is really only dominant in individually managed servers and desktop deployments, neither of which are particularly large or growing parts of the overall ecosystem.
Plus, nothing is permanent in the world of tech :). At this point, systemd is mature enough, and deployed widely enough, that I’m beginning to see a bunch of anti-patterns. In my experience, this is when people begin thinking of the next step. Anti-patterns aren’t just a symptom of incompetent users/developers, they’re a sign that a technology is reaching its limits.
A pattern that I’m seeing increasingly often in embedded systems is something that I’ll just go ahead and call the “big init service”, for lack of a better word. Basically, there’s a unit file that runs a shell script, which runs all the application-specific programs (random example from the latest gadget I’ve worked on: a zeromq-based daemon, a logging service – I don’t know why that’s a thing, I was paid just to make the previous daemon stop crashing –, a pseudo-watchdog timer, a license manager, some serial-listening daemon, and a Wayland session in kiosk mode). Basically anything that didn’t have a Yocto package in a base layer so that you could just list it as enabled/disabled at build time.
Being the helpful idiot that I am, I asked one my customers why those aren’t separate init services (I generally know better, but we had a history together and figured it wouldn’t hurt). They told me they knew it was possible but it was a lot of hassle, and they frequently had to launch some of these programs with various extra flags or environment variables for debugging or development purposes, or tweak various parameters when testing. Plus they were all designed to work together anyway. It was a lot more convenient to be able to start them all at once, stop them all at once, and tweak how that’s done by just changing a few lines in a shell script, than to mess with half a dozen unit files. I offered a few suggestions about how systemd can manage that. Turns out they’d tried each and every one of them, and never managed to get it to work reliably – and figured they’d rather write a clunky script than risk dealing with all sorts of flops in the field.
(Edit: to be clear, I think systemd can actually handle that case pretty well – but, indeed, I guess it is a lot of hassle, especially if you want to ensure it works the same way every time, which is very much important for unattended devices where interactive access isn’t always easy to get).
What advantage does systemd bring in this use case in the first place? It sounds to me that they’re basically bypassing systemd? Or are there other parts that bring value?
I tend to use containers over VMs specifically because I don’t have to configure a process manager or a log collector or an ssh daemon or host metrics or anything else. More importantly, developers don’t have to know how to do these things, so now they are empowered to own a greater share of the “ops”, and they aren’t bottlenecking on communication and coordination with an ops function.
They may use systemd on the oustide (does containerd depend on systemd yet?) but typically the containers don’t use systemd on the inside.
You must work in a much more aesthetically pleasing corner of the software world than I do! At this point I’m rarely surprised to find systemd in an Enterprise Container™.
That might be true, but while Gartner might have some say about what kind of work I’m likely to do in the future, it has very little influence on my personal choice of desktop, and the people who maintain open source desktop linux have mostly chosen systemd.
That might be true, but while Gartner might have some say about what kind of work I’m likely to do in the future, it has very little influence on my personal choice of desktop, and the people who maintain open source desktop linux have mostly chosen systemd.
I don’t disagree, but I suspect that systemd will make part of the ecosystem increasingly disconnected from everything else. I wouldn’t be at all surprised if Android started to encroach on the desktop Linux market. Android works surprisingly well on devices with keyboards and mice instead of touchscreens and now supports split screen mode, which is what fans of tiling window managers have been telling us all for ages is better than multiple windows. If you use F-Droid, you can install a load of F/OSS apps, including a chroot environment for GNU/Linux things and even an X server so you can run graphical non-Android *NIX apps. There’s even a native OpenOffice port. At the same time, you can run things like MS Office or any of the rest of a huge pile of other supported proprietary software.
If you’re a hobbyist developer, writing an Android app rather than something for GNOME or KDE gives you a lot more potential users, a lot more potential collaborators, and a set of more marketable skills if you want to eventually move into commercial (open source or proprietary) development. How long do you think GNU/systemd/{GTK/Qt}/{X11/Wayland}/Linux is going to be more popular than Android for desktops?
That’s an interesting question. Popularity - for me - is not a reason to jump ship to another system, at least not for personal use. You’re a FreeBSD developer; if you’d listened to the cool kids on slashdot back in the day you’d have abandoned that doomed ship a long time ago ;)
But you’ve made me wonder what my criterium for choosing a platform is. I’ve always told myself it’s about antifeatures (as defined by Benjamin Mako Hill), which is why Android doesn’t appeal to me. But arguably, systemd qualifies as an antifeature, and it nevertheless went on to dominate Linux distributions.
You’re a FreeBSD developer; if you’d listened to the cool kids on slashdot back in the day you’d have abandoned that doomed ship a long time ago ;)
In hindsight, that might have been the right call. At the moment, I’m working more on Linux than on FreeBSD. Some things are nicer on one, some on the other (my ideal *NIX would have clone, futex, ZFS, capsicum and jails, for example). If Linux were not GPL’d, I think I’d be working now on bringing the features that I miss from FreeBSD across and give up on FreeBSD entirely. That said, I see Windows and *NIX as legacy systems at this point and I’m more interested in working on the things that will replace them.
Well, there is systemd-nspawn. Which is a sorta alternative to LXC. I haven’t tried either yet, so I can’t tell anything about their respective qualities.
I’ve found the systemd man pages to be somewhat complete and comprehensible. Give man systemd a try and then you can gradually discover other tools by jumping from one to the next in the “see also” sections. It’s not the best approach to learning, but it’s not bad either.
Exactly. This is what I’ve come to appreciate about systemd. Compared to what Linux had, it’s reasonably consistent, coherent, and thorough. Sure, it makes a few questionable decisions, sure it has had a rocky road at times (I recall my system locking up on shutdown in the early days thanks to something in journald), and sure there are individual parts of it for which I might personally prefer an alternative (runit for process management, cronie, syslog), but taken as a whole I feel so much more comfortable with what we have now. It gives a certain level of polish I’ve always felt lacking in Linux (OpenBSD has this coherent feel too, without systemd, but I can appreciate both approaches). And most importantly, despite the fears, it doesn’t seem to have affected the viability of alternatives. There are still plenty of distributions that avoid systemd entirely.
I’ve recently been looking for a Linux distribution supported (by default) by cloud providers that feels reasonably clean. I’ve been using OpenBSD but sometimes a Linux just makes certain things easier and I needed the option. Long story short I ended up on Fedora and the one part I really didn’t like was NetworkManager. It felt a bit of a mess with documentation scattered all over, disorganised manpages, and legacy formats (ifcfg-rh) just to confuse things. Then I realised that I could just use systemd-networkd which was well documented, far more simple, easy to configure with familiar syntax, and pretty broadly available (in base installs) in case I ever need to switch. l actually ended up using NetworkManager’s keyfile plugin, but discovering that was a chore, and I just want these things to work.
Newest edition of UNIX and Linux System Administration Handbook has a good chapter about systemd, I believe, but I am not sure if it will be as comprehensive as you’d want it.
As someone who does like a lot of the systemd vision in practice (disclaimer up front) even if the implementation could be a lot better, this is a good (but biased, but it’s honest about it) summary on kernel/userland trends over the past few decades, and addresses the gap between people who do Linux in industry versus hobby and the contradictory goals that ensue.
Section 3 of this should be broken out and merged into the systemd man pages. I’ve never seen a clear description of what systemd actually does until now. This explains so much systemd behavior that I’ve just stared at in confusion before. Units are just a graph for generating jobs! I get it now!
After reflecting on the article, I think even if systemd turns out to be a “dead end” like Utopia and subsumed by kernel subsystems, a lot of those subsystems and their direction were a result of systemd and Utopia before it; they would have likely never went in that direction without userland impetus. (Userland is also easier to experiment with than the kernel, so…)
I think the next step for startup systems is to use systemd external api but replace the internals with a thin layer to use init, rc and service management. I think app devs like that they can use one interface to add their apps as demons, etc.
Among the damages caused by fragmentation, according to Bernard, are upstream maintainers “adding a permanent dock, icons on the desktop, re-enabling the systray” to the DE.
I guess that’s supposed to be downstream maintainers, or distro maintainers (to the dismay of upstream gnome).
I can’t find a way to leave comments or email the author though, pretty much all the links on that website end up bringing you to a mastodon instance.
Basically socket activation has a service manager keep a network socket open and it starts the service being served up (if it’s not already running) when a new client connects. In systemd, it can be used so the SSH daemon to has a separate instance of sshd per user connection.
My view when systemd first crossed my radar (around the time Debian was making it’s decision about supporting/adopting systemd or not), I of course saw a lot of commentary about the negative aspects of the project.
I’d imagine those criticism are still true: it seems unlikely the people involved in this have softened their views or language.
But what has changed is my appreciation for the technical benefits of systemd as an init manager (and a few related tasks, e.g. the timers stuff to replace cron). I’m not that thrilled about the ever-increasing scope of the project as a whole, but I don’t use Linux as a desktop OS, so I think I’m shielded from a lot of the impact it has on e.g. Gnome etc.
I tried (and eventually succeeded, from memory) to write an init script for a process, using the Debian provided ‘start-stop-daemon’ and an existing init file as a guide. This was not a productive or enjoyable experience for me, at all. I’d say I’m quite comfortable with shell as a scripting language in general, but the idiosyncrasies of producing a reliable init script were confusing and non obvious to me at the time.
These days, I don’t think twice about either creating new systemd units, or using drop-ins to customise existing ones - does everything always work the first time? No. Is it an order of magnitude easier than a sysvinit script (IMO) to identify why it doesn’t do what you expect, and find a solution that will do what you expect? Generally yes.
Edit: typo
This is the thing I’ve loved most about systemd. Writing an init script for one distribution is a PITA. Writing one that works on multiple different linux distros makes me tear my effing hair out. The complexity of writing a shell script that conforms to certain parameters with a wide variety of “helpers” seems out of whack with what I want to do: ie: fire up a process and leave it in the background.
Also, an init system that cannot track what child processes it has started, and needs a complicated system of pidfile management to control the process that doesn’t even work in many cases (it breaks as soon as the machine or process crashes, or gets killed without the proper signal being sent beforehand), is barely able to call itself “an init system”. It’s just what we always had, so in the proper UNIX way, everyone just dealt with it and nobody pointed out the big elephant in the room, in that it sucked hard.
Now. systemd has it’s problems, and over-reach is definitely in there. I can’t count the number of times that I’ve had to figure out the “new hotness” way of configuring a GD static network interface, and every time I look at it systemd has new capabilities that it probably shouldn’t have. ( Following jwz’s maxim, I can’t wait for the systemd mail client), but pretending sysvinit was all good was insanity.
It always bothers me a bit when systemd is compared to just init scripts, because those are not the only two options available (and never have been). I don’t really blame the casual Linux user like you, but on more than a few occasions the systemd people used “well, in init scripts it sucks even worse” as fallacious defence of their own shortcoming (you don’t need to rant, rave, and insult to be toxic).
The only commonly used init system in Linux, used by all the major distros, was sysvinit. Yes, there was OpenRC, and upstart, and others, but none of them were ever widely adopted. Yes, Ubuntu developed upstart and it solved some of the problems in sysvinit, but was never widely adopted, so if you were writing a daemon for anything other than ubuntu, it was back to sysvinit, and all the pain that entailed. It’s great that there are other init systems that solve the problem, but if none of them are ever adopted, there’s really no value in it for people who write daemons.
OpenRC is essentially just a continuation of sysV init, with all problems with it. I actually just have OpenRC start runit on my Alpine system, as that’s just easier than writing those scripts. I never looked at upstart.
daemontools and the various projects it inspired (runit, s6, freedt, a few others) always seemed like a good solution to me. It solves most of the issues from SysV init and gives 90% of systemd for most use cases, but in a much smaller package. I don’t know why it was never widely adopted; perhaps it was timing, lack of documentation, lack of advocacy, or something else.
For daemontools, it’s likely the fact that djb has a tendency to release a project and then immediately stop maintaining it, so there becomes a slurry of patches with no canonical blessed set for distros to maintain, and no upstream merging them.
Yeah, that partly explains daemontools; there’s also the whole license issue, which didn’t become clear until 2007 (when djb finally released his stuff as public domain; before that he rejected that a license was needed at all; also a problem with qmail, djbdns) which is why people needed to maintain patches instead of a “daemontools-maintianed” repo or whatnot.
But runit stems from 2004, and there are other daemontools-like systems from the same time period. So I don’t think that djb’s idiosyncrasy fully explain it.
Ah, daemontools. I can answer that one. Daemontools started as a project by Dan J Bernstein (hereafter referred to as djb), who is a brilliant mathematician and cryptologist, but the license for some of his projects, including daemontools is … troubling to a lot of the distributions, and he can be, well, prickly, to put it politely.
Wikipedia has the license as Public Domain, but ISTR that if you didn’t distribute it precisely as was specified he would complain a lot. It meant that a lot of distributions that wanted to use his software shied away from it.
It’s pretty awesome, but it still doesn’t solve a lot of the pain points that systemd solves.
This is all vague rememberings from about 20 years ago, but having dealt with djb in the past, I can definitely understand the reticence on the part of distributions to package his software, or rely on it for critical functionality. He makes Lennart look like a softy in comparison.
runit is used in Void Linux which is a somewhat popular non-SystemD distro. It’s straight-forward to use.
Yeah, I use Void; it’s comparatively small though (although gaining in popularity a bit as an Arch alternative, partly thanks to runit).
Even if you were just targeting Ubuntu, upstart was pretty buggy and feature deficient. You could wedge it so bad that you had to reboot while iterating on a job. I’m pretty happy that systemd takes a features-welcome approach. Every time I think that I’ll have to use a shell stub I can find what I need with some man page searching.
Right, I’m aware there are other options out there, and you’re right they should be part of the conversation when a distro is making decisions about what init to support in its packages.
But, for someone using a distro, even someone writing packages for a distro, you need to support what the distro supports. Yes you could also support extra init systems but you need to support what the distro supports.
Yeah, I’m not faulting you for not mentioning it in your previous comment, it’s more of a general commentary about the wider discussion 😅
I’m genuinely curious how someone can downvote an opinion as “incorrect”.
Oh wait never mind I just remembered this is the internet and someone will always be there to say “you’re wrong”
I’ve had all sorts of things -1’d as “incorrect” or “troll” where I really struggle seeing how either can apply. Write a lengthy reply going out of my way to make sure it’s nuanced: -2 troll… Great, thanks 😒 Like you said: welcome to the internet 🤷♂️
I’ve noticed this as well. Lobsters has always had explicit downvote reasons to discourage downvoting for silly things like “I disagree” or “your opinion clashes with my worldview and I’m salty.” But as we’ve grown I see those kinds of downvotes more and more.
Systemd definitely achieves several good things, but it has technical, social and political faults. It is an improvement over init scripts, but it is also many other things — and some of those are strict regressions (I am thinking specifically of how it wantonly breaks
nohup). Is systemd-as-it-is an improvement over a better system which doesn’t exist? I really don’t know.The kill-user-processes-on logout thing is no doubt annoying the first time you encounter it, but I’d suggest that it’s at worst a case of incorrect defaults (I honestly don’t even know what the upstream or distro defaults are).
I’m certain it has technical/approach faults, but in the grand scheme of things that one is pretty easy to ‘solve’.
With the help of an Internet search query or two, and a good deal of re-reading, I could probably figure out what this means. But I am le tired. And I haven’t even gotten through the first chapter.
This brings up a different controversy: Is writing like this useful to stretch peoples minds, help them grow? Or does it just discourage people and make them feel dumb? Is writing a form of art, or a means of communication, or both?
From my perspective, it looks more focused on artistry than actually communicating and transferring knowledge. Which isn’t necessarily a bad thing.
It’s called “purple prose”.
That’s exactly what I was referring to, TIL.
Your perspective may change if you finish reading the document before dashing off to post a comment about the style instead of the substance of the source material. :)
It goes on to give a lot of historical context on previous attempts and on systems issues.
That’s the point though - I’d happily read the whole thing, if I had the mental capacity to do so. I’m sure there’s plenty of good info in there, but it’s buried under so much prose that I feel like I need a college degree to begin understanding it.
The point that you mentioned is a local maximum in terms of hard-to-read prose, and (from my reading up to section 2.3) I believe it is a global maximum too - the text is much nicer almost from that passage onward and I only found one other spot that I believe to be particularly egregious. I encourage you to keep going! I very much dislike the Unix culture and tooling in general, but found this article to be fascinating anyway.
I sampled towards the end and found the prose similarly tiresome.
Not every reader is native English speaker (and reader). Reading this style is very demanding and makes it harder to focus on the main topic. I have no idea if there are expressions hidden that I don’t know yet, or if a reference in the text is key to the final point.
You make the mistake of implying that there’s an objective way to communicate clearly and it’s not instead dependent on the listeners.
Most tech documentations, even when assuming no previous knowledge from the reader, would be unreadable for most people for the sole reason of their structure and dryness.
You’re witnessing something written by a person that is on the boundary between different worlds and can merge them into a single piece of writing and this confuses you. It’s normal and probably the author did it deliberately.
Lot of us programmers on the left are keen in rupturing the cultural and social bubble in which the tech sector is entrenched and I wouldn’t be surprised if the author wrote the piece in this way to deliberately challenge the stylistic elements of engineer-oriented writing.
I agree, but I think it’s just interesting that the author seems to have raised the educational bar, not lowered it. Now you have to be an engineer AND a literary genius to understand what’s being written.
And I’m not saying this is necessarily a bad thing. I’m just noting that it’s unapproachable for people like me. I’m sure after a bit more education and time, I’ll appreciate it too.
You don’t have to be a genius to understand that paragraph, just literate. I understood it just fine and the highest formal education I’ve received is community college.
Anyone who works in software is accustomed to googling tech jargon they’re unfamiliar with and learning as they go. I think it is interesting that you balk at doing this in a humanities context and perhaps that was the point.
I think you hit the nail on the head. It sounds to me like you’ve put in the hard work to understand and appreciate this kind of writing, and that is truly awesome (no sarcasm there). For me, I could totally figure this out, no doubt about it. But it would still be quite the slog, because I haven’t invested quite as much hard work as you have into the humanities yet. And that’s ok.
That’s why I say this “isn’t necessarily a bad thing.” I could use some more development in my humanities skills! But if the goal is to communicate ideas, then writing like this will only communicate ideas to a gifted few who are good at both engineering and humanities. Again: Not necessarily a bad thing.
If you want to start from somewhere, I suggest you read Geek Sublime from Vikhram Chandra. He’s both a programmer and a novelist.
And yet you at the same time alienate readers who’ve never studied literature in English, nor English literature. More often than not from second and third world countries.
We write in different ways to reach different audiences in different ways. I never said everything should be written this way. There’s a growing idea of writing more propaganda disguised as tech opinion pieces using a language that programmers like. I mean, the far-right has been doing that for three decades now, it’s time for the left to react.
The “poking” at the bubble is complementary and clearly doesn’t aim at reaching the masses, but just at bringing out those that have the potential to connect with those readings.
Also I’m not American, I’m not an English native speaker and in “second and third world countries” humanities are usually valued a lot more than in the protestant anglosphere, so I don’t get your point.
This is the “English (Literary)” locale where it is assumed you know your leftist lore (and French philosophy). Not my thing but sometimes the references lead to interesting wikipedia pages.
When I was younger I thought that impressing people with my vocabulary is the utmost purpose of my writing, but as years have gone on I have realized that this is just mental peacocking (to use a friendly term). The real greatness is to convey information (or even humor) using simple language while staying informative and engaging. I have been way more impressed with authors being able to write like this than people who just show they own a thesaurus.
I would argue the former. You’ve learned a little more about the world by virtue of looking up a literary or cultural reference you didn’t previously understand. I’ve done this with plenty of texts I’ve read, in English and particularly in other languages (English is my cradletongue). Of course you weren’t obligated to do so - you could’ve decided that this article about systemd written using these kinds of literary references wasn’t worth your time to understand, and if enough people thought similarly, it would lose a substantial portion of its potential audience. But using literary and cultural references is something that writers and speakers do in pretty much every genre of human communication; and it’s only because this author happened to choose ones that you were unfamiliar with, that you found it something worth remarking on.
I found it really funny. To me, that kind of descriptiveness adds humor, but maybe I’m the only one? Like putting an eccentric accent on a movie character.
i found it pretty funny too :)
I think the article would be much longer if it wasn’t for the use of various literary devices for compression.
Maybe that would be for the better as more people would benefit from reading it.
On the other hand, it’s useful to have someone shine a light on gaps in your knowledge of things which used to be well known. Certainly I don’t think that a lot of the things referred to in the writing should be particularly exotic, yet sadly they are.
Finally, I personally get some enjoyment from reading things like these. It’s the kind of enjoyment I get from reading old books. Difficult to read but full of very interesting ideas.
While I do not particularly like the style of this text, I appreciate any kind of writing style that does not resemble the current “ted-talk” style uncultured vomit that is becoming the style of almost everybody today.
I”m less offended by the use of some obscure terms and references than by the implicit comparison of a Gnome maintainer to a follower of Stalin:
Arguably, in many historical cases, such ways of expression were used to hide the real meaning, a sort of emphemization. It was especially used when it came to criticizing a point of view, to filter out as many people as possible to not bear the consequences of saying clearly what you intended to say. The more fluff and ambiguity the more you can hide behind it.
EDIT as this comment says, it’s called “purple prose”.
FWIW, here’s my translation:
Without the context of the surrounding paras I don’t really know what they were getting at.
I think people write like this because it’s fun for the writer, not necessarily for the reader ;)
Edit: the context is:
The Fabian Society is a British internationalist lefty think tank. Because they’re internationalist and lefty (but not anarchists), they’d probably really like a global humanitarian government.
“vox populi” is latin for “voice of the people” and sometimes means the “opinion of the people”, but that bit is a bit clumsy anyway.
The technical review is more damning than Poettering’s dramatics. Nondeterministic transactions? Features preserved because backwards compat but provide a massive footgun? Use of “reboot blocker” tasks? Wow!
It strikes me that systemd has some real architectural mistakes and needs to bite the bullet and redesign the inconsistencies, nondeterminisms. Seeing all that, I practically thought “maybe using SQLite would’ve resolved a lot of state tracking/migration pathing”
For what it is worth, SMF (the Solaris service management thing) uses sqlite for its storage. SMF has its own set of issues, but lack of atomicity isn’t one.
Does anyone know of a good book that treats systemd somewhat comprehensively?
At this point, I’ve decided my feelings about systemd don’t matter; it’s clearly here to stay. System administration is not my day job, so I’ve been able to get by with only minimal and superficial knowledge of it. I’ll have to re-learn a lot of tasks that I knew how to do (or could at least figure out how to do) using the previous init systems.
Whenever I do, I find I’m always googling (or reading man pages) in frustration. It’s a very extensive system. Command names are long and non-obvious. That’s okay. After all, if I managed to learn git, I should be able to learn systemd.
But I’d like to do so at a leisurely pace, instead of scraping documentation together from (frequently severely outdated) blog posts and man pages.
So is there a good book that’s up-to-date? Or is Red Hat certification study material my best hope?
That’s not totally clear to me. The two biggest bits of the Linux ecosystem are Android and cloud deployments. Android does not use systemd, it uses its own thing. Cloud deployments increasingly use things like Kubernetes to deploy containers. They may use systemd on the oustide (does containerd depend on systemd yet?) but typically the containers don’t use systemd on the inside. Over time, I expect the things on the outside to be simplified and systemd is an obvious contender to go because it doesn’t add much value in this space.
Systemd is really only dominant in individually managed servers and desktop deployments, neither of which are particularly large or growing parts of the overall ecosystem.
Plus, nothing is permanent in the world of tech :). At this point, systemd is mature enough, and deployed widely enough, that I’m beginning to see a bunch of anti-patterns. In my experience, this is when people begin thinking of the next step. Anti-patterns aren’t just a symptom of incompetent users/developers, they’re a sign that a technology is reaching its limits.
A pattern that I’m seeing increasingly often in embedded systems is something that I’ll just go ahead and call the “big init service”, for lack of a better word. Basically, there’s a unit file that runs a shell script, which runs all the application-specific programs (random example from the latest gadget I’ve worked on: a zeromq-based daemon, a logging service – I don’t know why that’s a thing, I was paid just to make the previous daemon stop crashing –, a pseudo-watchdog timer, a license manager, some serial-listening daemon, and a Wayland session in kiosk mode). Basically anything that didn’t have a Yocto package in a base layer so that you could just list it as enabled/disabled at build time.
Being the helpful idiot that I am, I asked one my customers why those aren’t separate init services (I generally know better, but we had a history together and figured it wouldn’t hurt). They told me they knew it was possible but it was a lot of hassle, and they frequently had to launch some of these programs with various extra flags or environment variables for debugging or development purposes, or tweak various parameters when testing. Plus they were all designed to work together anyway. It was a lot more convenient to be able to start them all at once, stop them all at once, and tweak how that’s done by just changing a few lines in a shell script, than to mess with half a dozen unit files. I offered a few suggestions about how systemd can manage that. Turns out they’d tried each and every one of them, and never managed to get it to work reliably – and figured they’d rather write a clunky script than risk dealing with all sorts of flops in the field.
(Edit: to be clear, I think systemd can actually handle that case pretty well – but, indeed, I guess it is a lot of hassle, especially if you want to ensure it works the same way every time, which is very much important for unattended devices where interactive access isn’t always easy to get).
What advantage does systemd bring in this use case in the first place? It sounds to me that they’re basically bypassing systemd? Or are there other parts that bring value?
There aren’t, at least not for them, but at this point it’s so hard to yank it out of (some) systems that they’d rather bypass it.
I tend to use containers over VMs specifically because I don’t have to configure a process manager or a log collector or an ssh daemon or host metrics or anything else. More importantly, developers don’t have to know how to do these things, so now they are empowered to own a greater share of the “ops”, and they aren’t bottlenecking on communication and coordination with an ops function.
You must work in a much more aesthetically pleasing corner of the software world than I do! At this point I’m rarely surprised to find systemd in an Enterprise Container™.
That might be true, but while Gartner might have some say about what kind of work I’m likely to do in the future, it has very little influence on my personal choice of desktop, and the people who maintain open source desktop linux have mostly chosen systemd.
I don’t disagree, but I suspect that systemd will make part of the ecosystem increasingly disconnected from everything else. I wouldn’t be at all surprised if Android started to encroach on the desktop Linux market. Android works surprisingly well on devices with keyboards and mice instead of touchscreens and now supports split screen mode, which is what fans of tiling window managers have been telling us all for ages is better than multiple windows. If you use F-Droid, you can install a load of F/OSS apps, including a chroot environment for GNU/Linux things and even an X server so you can run graphical non-Android *NIX apps. There’s even a native OpenOffice port. At the same time, you can run things like MS Office or any of the rest of a huge pile of other supported proprietary software.
If you’re a hobbyist developer, writing an Android app rather than something for GNOME or KDE gives you a lot more potential users, a lot more potential collaborators, and a set of more marketable skills if you want to eventually move into commercial (open source or proprietary) development. How long do you think GNU/systemd/{GTK/Qt}/{X11/Wayland}/Linux is going to be more popular than Android for desktops?
That’s an interesting question. Popularity - for me - is not a reason to jump ship to another system, at least not for personal use. You’re a FreeBSD developer; if you’d listened to the cool kids on slashdot back in the day you’d have abandoned that doomed ship a long time ago ;)
But you’ve made me wonder what my criterium for choosing a platform is. I’ve always told myself it’s about antifeatures (as defined by Benjamin Mako Hill), which is why Android doesn’t appeal to me. But arguably, systemd qualifies as an antifeature, and it nevertheless went on to dominate Linux distributions.
In hindsight, that might have been the right call. At the moment, I’m working more on Linux than on FreeBSD. Some things are nicer on one, some on the other (my ideal *NIX would have clone, futex, ZFS, capsicum and jails, for example). If Linux were not GPL’d, I think I’d be working now on bringing the features that I miss from FreeBSD across and give up on FreeBSD entirely. That said, I see Windows and *NIX as legacy systems at this point and I’m more interested in working on the things that will replace them.
Just you wait a few months, and systemd will also do container orchestration! j/k :)
Well, there is systemd-nspawn. Which is a sorta alternative to LXC. I haven’t tried either yet, so I can’t tell anything about their respective qualities.
I’ve found the systemd man pages to be somewhat complete and comprehensible. Give
man systemda try and then you can gradually discover other tools by jumping from one to the next in the “see also” sections. It’s not the best approach to learning, but it’s not bad either.Exactly. This is what I’ve come to appreciate about systemd. Compared to what Linux had, it’s reasonably consistent, coherent, and thorough. Sure, it makes a few questionable decisions, sure it has had a rocky road at times (I recall my system locking up on shutdown in the early days thanks to something in journald), and sure there are individual parts of it for which I might personally prefer an alternative (runit for process management, cronie, syslog), but taken as a whole I feel so much more comfortable with what we have now. It gives a certain level of polish I’ve always felt lacking in Linux (OpenBSD has this coherent feel too, without systemd, but I can appreciate both approaches). And most importantly, despite the fears, it doesn’t seem to have affected the viability of alternatives. There are still plenty of distributions that avoid systemd entirely.
I’ve recently been looking for a Linux distribution supported (by default) by cloud providers that feels reasonably clean. I’ve been using OpenBSD but sometimes a Linux just makes certain things easier and I needed the option. Long story short I ended up on Fedora and the one part I really didn’t like was NetworkManager. It felt a bit of a mess with documentation scattered all over, disorganised manpages, and legacy formats (ifcfg-rh) just to confuse things. Then I realised that I could just use systemd-networkd which was well documented, far more simple, easy to configure with familiar syntax, and pretty broadly available (in base installs) in case I ever need to switch. l actually ended up using NetworkManager’s keyfile plugin, but discovering that was a chore, and I just want these things to work.
One of the early adopters of systemd, Arch Linux, has fine cheat sheet -like documentation that goes through many things: https://wiki.archlinux.org/index.php/Systemd
Systemd isn’t so complex that you should be looking at certifications.
Newest edition of UNIX and Linux System Administration Handbook has a good chapter about systemd, I believe, but I am not sure if it will be as comprehensive as you’d want it.
As someone who does like a lot of the systemd vision in practice (disclaimer up front) even if the implementation could be a lot better, this is a good (but biased, but it’s honest about it) summary on kernel/userland trends over the past few decades, and addresses the gap between people who do Linux in industry versus hobby and the contradictory goals that ensue.
Section 3 of this should be broken out and merged into the systemd man pages. I’ve never seen a clear description of what systemd actually does until now. This explains so much systemd behavior that I’ve just stared at in confusion before. Units are just a graph for generating jobs! I get it now!
After reflecting on the article, I think even if systemd turns out to be a “dead end” like Utopia and subsumed by kernel subsystems, a lot of those subsystems and their direction were a result of systemd and Utopia before it; they would have likely never went in that direction without userland impetus. (Userland is also easier to experiment with than the kernel, so…)
I think the next step for startup systems is to use systemd external api but replace the internals with a thin layer to use init, rc and service management. I think app devs like that they can use one interface to add their apps as demons, etc.
For example, dnscrypt-proxy usage of systemd: https://github.com/DNSCrypt/dnscrypt-proxy/wiki/systemd
This is too verbose.
I guess that’s supposed to be downstream maintainers, or distro maintainers (to the dismay of upstream gnome).
I can’t find a way to leave comments or email the author though, pretty much all the links on that website end up bringing you to a mastodon instance.
Could anybody explain what socket activation is?
Basically socket activation has a service manager keep a network socket open and it starts the service being served up (if it’s not already running) when a new client connects. In systemd, it can be used so the SSH daemon to has a separate instance of sshd per user connection.
Here’s more words: http://0pointer.de/blog/projects/socket-activation.html
I am wrong or does this sound a bit like message passing for services?
It is so hard to read a long essay about something that sucks.
I did not find this comment conducive to discussion; evidently others feel similarly.
Also, for what it’s worth, sometimes reading long essays helps us defend to others our beliefs as to why something sucks.