1.  

    I feel like I’ve put too many tags, but there isnt a catchall tag for systems administrations like there is for programming.

    I have a linux / bsd sys admin interview on tuesday morning, so I’m looking up questions.

    1.  

      Good luck.

    1.  
      • $COMPANY - I added support for multiple configs (i.e. multiple vrrp_instance blocks for keepalived.conf) in Koalephant Failover Manager over the weekend, I need to test it a little more and then make a release.

      • $CLIENT1 - still mostly progressing with Bamboo Model migration, and I’ll be using the above release of Failover Manager to add a second floating (private) IP to the LB’s, so DB cluster traffic doesn’t traverse the public floating IP

      • $PERSONAL - I’ve started playing with D, as a potential path forward for some of the Koalephant tools that get too big/complex to be easy to maintain as shell programs.

      • $HOME - A couple more outside lights to change the switches/install and I can call that job done for now.

      1. 10

        Redis is an example of this paradigm. Today, most cloud providers offer Redis as a managed service over their infrastructure and enjoy huge income from software that was not developed by them.

        This makes no sense. Redis was not developped by Redis Labs either, they only hired Antirez a few years ago. Before that, they were doing exactly what they criticize today. I would even say they weren’t the best actor in that space, e.g. OpenRedis was founded by people truly involved in the community…

        I won’t say any more before I see a direct take from Salvatore on this issue.

        In any case, commercially, I think this is a huge mistake. If they persist Redis is going to be forked and the fork will eventually win, a la MariaDB.

        EDIT: I had misread the announcement, this is only about modules which will be Apache 2 + Commons Clause, while the Redis core will remain BSD. I am fine with that even though I think AGPL would make more sense if you want those modules to become popular. Enforcing a monopoly on hosting is never a good idea.

        From various comments I can read online it looks like I was not the only one to misunderstand. That Redis was unaffected should probably have been the first line of that post, in bold…

        EDIT 2: Salvatore himself is tweeting about the issue on Twitter right now. https://twitter.com/antirez/status/1032192722755571714

        1. 7

          MariaDB isn’t the bastion of open source people make it out to be.

          Since the original claim that a) Oracle would close-source MySQL, and/or let it stagnate to force adoption of their commercial offerings, and thus b) MariaDB would fork MySQL and maintain feature parity, in an open source model:

          • Oracle have released several new versions, with major new features/functionality/SQL compatibility
          • Oracle have maintained the existing Open Source licence
          • MariaDB have broken compatibility in numerous places
          • MariaDB have backflipped and made one of their components for running a HA cluster, closed source.

          Honestly if your (the reader, not @catwell) app/company needs MySQL and isn’t paying for a commercial licence from Oracle, IMO you’d be stupid not to use either Persona Server or Percona Cluster. Actual feature parity (i.e. they constantly rebase from upstream) and a clear business model: pay for support/advice, the software is completely free.

          1. 2

            clear business model: pay for support/advice, the software is completely free.

            That business model is what Redis Labs is trying to address. There’s a serious problem for companies with that model if a large hosting provider like AWS that more and more people are moving to can come in and offer a “as a service” version that cuts off the support revenue stream. At that point, AWS can benefit from the work that said company is doing without contributing anything back.

            Open source software in general and open source business models often assume that you won’t have parasitic players in the market who derive value from the work of others but contribute nothing in return. The current system is going to have to change eventually to account for that.

            It might end up being that all open source software is produced by companies that aren’t “product” companies. For example, Google spinning out K8S and not attempting to make money off the software. LinkedIn getting an advantage out of opening up Kafka etc. In that world, eventually, there will be very few companies like RedisLabs, CockroachDB, InfluxDB etc that are trying to be product companies. The large move “to the cloud” that is underway is a huge disruption to that previously OSS business model. I think a model that many will try will be to take an open source product and provide close sourced, additional functionality around those codebases (thereby sidestepping licenses like the AGPL) and doing managed hosting and as a service hosting within the big clouds like AWS, GCP, and Azure.

            1. 2

              There’s a serious problem for companies with that model if a large hosting provider like AWS that more and more people are moving to can come in and offer a “as a service” version that cuts off the support revenue stream.

              Percona provide support services for customers who use AWS’ various mysql flavoured “DB as a Service” offerings.

              This is not that different IMO than what Rackspace did - they took their Ops/Arch experience, and offered it as a service, regardless of who hosts the underlying machines.

          2. 3

            Oracle MySQL is alive and well and there is no sign of MariaDB winning.

            1. 0

              Re: AGPL

              The 3 modules that were reliscened were previously AGPL. AGPL doesn’t provide the protections that Redis Labs is seeking. Most OSS companies have a business model that revolves around support. If a large hosting provider like Amazon comes in and provides an “as a service” version, that cuts off a primary revenue stream. If said hosting provider doesn’t produce improvements to the codebase then AGPL doesn’t matter.

            1. 3

              Its not my month apparently, so this week I’m recovering from a stomach bug, and continuing with mostly the same as last week

              • $CLIENT1: ongoing dev, ZF1 > Bamboo v5 model migration, and helping one of the other devs with getting his CSP into the provisioning system.
              • $COMPANY: ongoing, albeit likely slower/less focussed, progress on Bamboo v6 changes.
              • $HOME: due to work, and then the aforementioned stomach bug, I got basically nothing done on home wiring last week (I did manage to replace an unwanted mature Banana plant with Black Bamboo though), so that’s still on the agenda for this week.
              1. 6

                Continuing to renovate the 1940s house I bought in March. I took it down to the brick shell and I’m slowly building it back up each weekend.

                I think I’ve had one weekend spare since then where I’ve not been doing DIY. But physical and mental work dovetail quite nicely. On the weekend my brain rests, in the week my body rests.

                1. 1

                  Sounds like you’re living my dream! Are you going to be adding some of those much needed geeky accessories like Cat6e throughout and full house automation?

                  1. 4

                    Given the reality of things I think the really geeky thing is not to use “home automation”.

                    1. 4

                      Only if you’re not engineering it yourself ;) I’m just as wary of third party IOT/“home automation” as the next paranoid engineer with experience in these things but the idea of developing a super-secure, self hosted set up makes me tingle with excitement.

                      For context, having an air gaped office network located within a Faraday cage is my ideal office solution.

                      1. 2

                        I think your office was featured in this scene of a movie at least one American in Russia thinks was historically accurate despite talking about future events. ;)

                      2. 2

                        Yeah. We’re supposed to be the smart rebels to dangerous trends in tech.

                        Alternatively, we’d do it on a super-secure setup limited to things that can’t be used to spy on you or burn your house down. No Internet connected ovens, dryers, coffee makers, or toasters for example. The threat model still allows one to have a local network and computer controlling the rest, though. High-assurance VPN’s are useful in this scenario, too, if one wants the remote functionality with minimal risk. Mikro-SINA is a good example of such architectures.

                  1. 2

                    I’m still looking for a freelance designer to

                    • finish some internal work for Koalephant (such as, the design for our site!)
                    • be able to send work to/pull onto projects (obviously pending availability)

                    Edit: my email is in my profile if anyone wants to get in touch.

                    1. 3

                      After some time off recovering from a case of ‘metal in the eyeball’, I’m getting back into a number of things:

                      • For $CLIENT1, regular ongoing dev, plus still working on the Model migration.

                      • For $COMPANY, I’ve started development (most of which is refactoring/changing existing things) for Bamboo v6

                      • For $HOME, I still have a bunch of wiring to do, to get the existing outside lights switched from inside the house, and to light the new room ‘properly’. Additionally, it turns out wifi signals don’t penetrate solid concrete very well, and 10+GB media files don’t stream well over shitty wifi, so it looks like I’ll be running ethernet through at least one concrete wall, so movies don’t constantly pause to re-buffer.

                      1. 8

                        Well I have just finished a release of my build tool, which I’m very pleased with. I have been trying to “prove” it’s utility by builting existing software with it. It was able to make scripts with it to build jq and bash, so it’s pretty effective.

                        1. 3

                          Do you have examples? I’d like to see the build script for jq.

                          1. 2

                            the script is here https://gist.github.com/rain-1/f3434f4b12147d5ef62369e511a184de#file-jq-makesfile I left in the make bits as comments for comparisons sake

                            which generates this input for the ‘makes’ tool: https://gist.github.com/rain-1/f3434f4b12147d5ef62369e511a184de#file-jq-example

                            and then makes processes those build rules to execute them in parallel respecting dependency order.

                            the bash scripts themselves that generate files are pretty ugly, my tool so far is just the engine part. A nice declarative UI to produce the TSV files can be made separately.

                            1. 2

                              So is your goal to write these makes files by hand or generate them with scripts?

                              1. 1

                                they can be generated by a DSL either something similar to the make language, or embedded as a python library.

                                1. 2

                                  Neat, thanks!

                              2. 1

                                Any reason your scripts target /bin/sh when they’re actually Bash scripts?

                                They won’t run, as-is on Debian, for example.

                                1. 2

                                  oh thats just a mistake

                          1. 34

                            Good talk.

                            I recently used systemd “in anger” for the first time on a raspi device to orchestrate several scripts and services, and I was pleasantly surprised (but also not surprised, because the FUD crowd is becoming more and more fingerprintable to me). systemd gives me lifecycle, logging, error handling, and structure, declaratively. It turns out structure and constraints are really useful, this is also why go has fast dependency resolution.

                            It violates unix philosohpy

                            That accusation was also made against neovim. The people muttering this stuff are slashdot markov chains, they don’t have any idea what they’re talking about.

                            1. 22

                              The declarative units are definitely a plus. No question.

                              I was anti-systemd when it started gaining popularity, because of the approach (basically kitchen-sinking a lot of *NIX stuff into a single project) and the way the project leader(s) respond to criticism.

                              I’ve used it since it was default in Debian, and the technical benefits are very measurable.

                              That doesnt mean the complaints against it are irrelevant though - it does break the Unix philosophy I think most people are referring to:

                              Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”.

                              1. 30

                                If you believe composability (one program’s output is another program’s input) is an important part of The Unix Philosophy, then ls violates it all day long, always has, likely always will. ls also violates it by providing multiple ways to sort its output, when sort is right there, already doing that job. Arguably, ls formatting its output is a violation of Do One Thing, because awk and printf exist, all ready to turn neat columns into human-friendly text. My point is, The Unix Philosophy isn’t set in stone, and never has been.

                                1. 7

                                  Didn’t ls predate the Unix Philosophy? There’s a lot of crufthistory in unix. dd is another example.

                                  None of that invalidates the philosophy that arose through an extended design exploration and process.

                                  1. 4

                                    nobody said it’s set in stone; it’s a set of principles to be applied based on practicality. like any design principle, it can be applied beyond usefulness. some remarks:

                                    • i don’t see where ls violates composability. the -l format was specifically designed to be easy to grep.
                                    • the sorting options are an example of practicality. they don’t require a lot of code, and would be much more clumsy to implement as a script (specifically when you don’t output the fields you’re sorting on)
                                    • about formatting, i assume you’re referring to columniation, which to my knowledge was not in any version of ls released by Bell Labs. checking whether stdout is a terminal is indeed an ugly violation.
                                    1. 6

                                      i don’t see where ls violates composability. the -l format was specifically designed to be easy to grep.

                                      People have written web pages on why parsing the output of ls is a bad idea. Using ls -l doesn’t solve any of these problems.

                                      As a matter of fact, the coreutils people have this to say about parsing the output of ls:

                                      However ls is really a tool for direct consumption by a human, and in that case further processing is less useful. For futher processing, find(1) is more suited.

                                      Moving on…

                                      the sorting options are an example of practicality. they don’t require a lot of code, and would be much more clumsy to implement as a script (specifically when you don’t output the fields you’re sorting on)

                                      This cuts closer to the point of what we’re saying, but here I also have to defend my half-baked design for a True Unix-y ls Program: It would always output all the data, one line per file, with filenames quoted and otherwise prepared such that they always stick to one column of one line, with things like tab characters replaced by \t and newline characters replaced by \n and so on. Therefore, the formatting and sorting programs always have all the information.

                                      But, as I said, always piping the output of my ls into some other script would be clumsier; it would ultimately result in some “human-friendly ls” which has multiple possible pipelines prepared for you, selectable with command-line options, so the end result looks a lot like modern ls.

                                      about formatting, i assume you’re referring to columniation, which to my knowledge was not in any version of ls released by Bell Labs. checking whether stdout is a terminal is indeed an ugly violation.

                                      I agree that ls shouldn’t check for a tty, but I’m not entirely convinced no program should.

                                      1. 4

                                        just because some people discourage composing ls with other programs doesn’t mean it’s not the unix way. some people value the unix philosophy and some don’t, and it’s not surprising that those who write GNU software and maintain wikis for GNU software are in the latter camp.

                                        your proposal for a decomposed ls sounds more unixy in some ways. but there are still practical reasons not to do it, such as performance and not cluttering the standard command lexicon with ls variants (plan 9 has ls and lc; maybe adding lt, lr, lu, etc. would be too many names just for listing files). it’s a subtle point in unix philosophy to know when departing from one principle is better for the overall simplicity of the system.

                                  2. 25

                                    With all due respect[1], did your own comment hit your fingerprint detector? Because it should. It’s extrapolating wildly from one personal anecdote[2], and insulting a broad category of people without showing any actual examples[3]. Calling people “markov chains” is fun in the instant you write it, but contributes to the general sludge of ad hominem dehumanization. All your upvoters should be ashamed.

                                    [1] SystemD arouses strong passions, and I don’t want this thread to devolve. I’m pointing out that you’re starting it off on the wrong foot. But I’m done here and won’t be responding to any more name-calling.

                                    [2] Because God knows, there’s tons of badly designed software out there that has given people great experiences in the short term. Design usually matters in the long term. Using something for the first time is unlikely to tell you anything beyond that somebody peephole-optimized the UX. UX is certainly important, rare and useful in its own right. But it’s a distinct activity.

                                    [3] I’d particularly appreciate a link to NeoVim criticism for being anti-Unix. Were they similarly criticizing Vim?

                                    1. 9

                                      [3] I’d particularly appreciate a link to NeoVim criticism for being anti-Unix. Were they similarly criticizing Vim?

                                      Yes, when VIM incorporated a terminal. Which is explicitly against its design goals. From the VIM 7.4 :help design-not

                                      VIM IS... NOT                                           *design-not*
                                      
                                      - Vim is not a shell or an Operating System.  You will not be able to run a
                                        shell inside Vim or use it to control a debugger.  This should work the
                                        other way around: Use Vim as a component from a shell or in an IDE.
                                        A satirical way to say this: "Unlike Emacs, Vim does not attempt to include
                                        everything but the kitchen sink, but some people say that you can clean one
                                        with it.  ;-)"
                                      

                                      Neo-VIM appears to acknowledge their departure from VIM’s initial design as their :help design-not has been trimmed and only reads:

                                      NVIM IS... NOT                                          design-not
                                      
                                      Nvim is not an operating system; instead it should be composed with other
                                      tools or hosted as a component. Marvim once said: "Unlike Emacs, Nvim does not
                                      include the kitchen sink... but it's good for plumbing."
                                      

                                      Now as a primarily Emacs user I see nothing wrong with not following the UNIX philosophy, but at it is clear that NeoVIM has pushed away from that direction. And because that direction was an against their initial design it is reasonable for users that liked the initial design to criticism NeoVIM because moving further away from the UNIX philosophy.

                                      Not that VIM hadn’t already become something more than ‘just edit text’, take quickfix for example. A better example of how an editor can solve the same problem by adhering to the Unix Philosophy of composition through text processing would be Acme. Check out Acme’s alternative to quickfix https://youtu.be/dP1xVpMPn8M?t=551

                                      1. 0

                                        akkartik, which part of my comment did you identify with? :) FWIW, I’m fond of you personally.

                                        I’d particularly appreciate a link to NeoVim criticism for being anti-Unix

                                        Every single Hacker News thread about Neovim.

                                        Were they similarly criticizing Vim?

                                        Not until I reply as such–and the response is hem-and-haw.

                                        1. 9

                                          To be fair I don’t think the hacker news hive mind is a good judge of anything besides what is currently flavour of the week.

                                          Just yesterday I had a comment not just downvoted but flagged and hidden-by-default, because I suggested Electron is a worse option than a web app.

                                          HN is basically twitter on Opposite Day: far too happy to remove any idea even vaguely outside what the group considers “acceptable”.

                                          1. 4

                                            Indeed, I appreciate your comments as well in general. I wasn’t personally insulted, FWIW. But this is precisely the sort of thing I’m talking about, the assumption that someone pushing back must have their identity wrapped up in the subject. Does our community a disservice.

                                            1. 0

                                              OTOH, I spent way too much of my life taking the FUD seriously. The mantra-parroting drive-by comments that are common in much of the anti-systemd and anti-foo threads should be pushed back. Not given a thoughtful audience.

                                              1. 2

                                                Totally fair. Can you point at any examples?

                                                1. 3

                                                  https://news.ycombinator.com/item?id=7289935

                                                  The old Unix ways are dying… … Vim is, in the spirit of Unix, a single purpose tool: it edits text.

                                                  https://news.ycombinator.com/item?id=10412860

                                                  thinks that anything that is too old clearly has some damage and its no longer good technology, like the neovim crowd

                                                  Also just search for “vim unix philosophy” you’ll invariably find tons of imaginary nonsense:

                                                  https://hn.algolia.com/?query=vim%20unix%20philosophy&sort=byPopularity&prefix&page=0&dateRange=all&type=comment

                                                  Please don’t make me search /r/vim :D

                                                  1. 4

                                                    thinks that anything that is too old clearly has some damage and its no longer good technology, like the neovim crowd

                                                    That’s not saying that neovim is ‘anti-Unix philosophy’, it’s saying that neovim is an example of a general pattern of people rewriting and redesigning old things that work perfectly well on the basis that there must be something wrong with anything that’s old.

                                                    Which is indeed a general pattern.

                                                    1. 1

                                                      That’s not saying that neovim is ‘anti-Unix philosophy’

                                                      It’s an example of (unfounded) fear, uncertainty, and doubt.

                                                      rewriting and redesigning old things that work perfectly well on the basis that there must be something wrong with anything that’s old.

                                                      That’s a problem that exists, but attaching it to project X out of habit, without justification, is the pattern I’m complaining about. In Neovim’s case it’s completely unfounded and doesn’t even make sense.

                                                      1. 1

                                                        It’s not unfounded. It’s pretty obvious that many of the people advocating neovim are doing so precisely because they think ‘new’ and ‘modern’ are things that precisely measure the quality of software. They’re the same people that change which Javascript framework they’re using every 6 weeks. They’re not a stereotype, they’re actual human beings that actually hold these views.

                                                        1. 2

                                                          Partial rewrite is one of the fastest ways to hand off software maintainership, though. And vim needed broader maintainer / developer community.

                                                          1. 0

                                                            Vim’s maintainer/developer community is more than sufficient. It’s a highly extensible text editor. Virtually anything can be done with plugins. You don’t need core editor changes very often if at all, especially now that the async stuff is in there.

                                                            1. 3

                                                              You don’t need core editor changes very often if at all, especially now that the async stuff is in there.

                                                              Which required pressure from NeoVim, if I understood the situation correctly. Vim is basically a one-man show.

                                                    2. 2

                                                      Thanks :) My attitude is to skip past crap drive-by comments as beneath notice (or linking). But I interpreted you to be saying FUD (about SystemD) that you ended up taking seriously? Any of those would be interesting to see if you happen to have them handy, but no worries if not.

                                                      Glad to have you back in the pro-Neovim (which is not necessarily anti-Vim) camp!

                                          2. 20

                                            What is FUD is this sort of comment: the classic combination of comparing systemd to the worst possible alternative instead of the best actual alternative with basically claiming everyone that disagrees with you is a ‘slashdot markov chain’ or similar idiotic crap.

                                            On the first point, there are lots of alternatives to sysvinit that aren’t systemd. Lots and lots and lots. Some of them are crap, some are great. systemd doesn’t have a right to be compared only to what it replaced, but also all the other things that could have replaced sysvinit.

                                            On the second point, it’s just bloody rude. But it also shows you don’t really understand what people are saying. ‘I think [xyz] violates the unix philosophy’ is not meaningless. People aren’t saying it for fun. They’re saying it because they think it’s true, and that it’s a bad thing. If you don’t have a good argument for the Unix philosophy not matter, or you think systemd doesn’t actually violate it, please go ahead and explain that. But I’ve never actually seen either of those arguments. The response to ‘it violates the Unix philosophy’ is always just ‘shut up slashdotter’. Same kind of comment you get when you say anything that goes against the proggit/hn hivemind that has now decided amongst other things that: microsoft is amazing, google is horrible, MIT-style licenses are perfect, GPL-style licenses are the devil-incarnate, statically typed languages are perfect, dynamically typed languages are evil, wayland is wonderful, x11 is terrible, etc.

                                            1. 8

                                              claiming everyone that disagrees with you is a ‘slashdot markov chain’ or similar idiotic crap

                                              My claim is about the thoughtless shoveling of groundless rumors. Also I don’t think my quip was idiotic.

                                              there are lots of alternatives to sysvinit that aren’t systemd

                                              That’s fine, I never disparaged alternatives. I said: systemd is good and I’m annoyed that the grumblers said it wasn’t.

                                              1. 2

                                                It’s not good though, for all the reasons that have been said. ‘Better than what you had before’ and ‘good’ aren’t the same thing.

                                                1. 1

                                                  seriously. If you don’t like systemd, use something else and promote its benefits. Tired of all the talking down of systemd. It made my life so much easier.

                                                  1. 1

                                                    seriously. If you like systemd, use it and shut up about it. Tired of all the talking up of systemd as if it’s actually any better than its alternatives, when it is objectively worse, and is poorly managed by nasty people.

                                                    1. 4

                                                      Have you watched the video this thread is about? Because you really sound like the kind of dogmatist the presenter is talking about.

                                                      If you like systemd, use it and shut up about it

                                                      Also, isn’t this a double-standard, since when it comes to complaining about systemd, this attitude doesn’t seem that prevalent.

                                                      1. 2

                                                        No, because no other tool threatens the ecosystem like systemd does.

                                                        Analogy: it wasn’t a double-standard 10 years ago to complain about Windows and say ‘if you like Windows, use it and shut up about it’.

                                                        1. 3

                                                          I see this kind of vague criticism when it comes to systemd alot. What ecosystem is it really breaking? It’s all still open source, there aren’t any proprietary protocols or corporate patents that prevent people from modifying the software to not have to rely on systemd. This “threat”, thr way I see it, has turned out to be at most a “ minor inconvenience “.

                                                          I suppose you’re thinking about examples like GNOME, but on the one hand, GNOME isn’t a unix-dogmatist project, but instead they aim to create a integrated desktop experience, consciously trading this in for ideal modularity – and on the other, projects like OpenBSD have managed to strip out what required systemd and have a working desktop environment. Most other examples, of which I know, have a similar pattern.

                                            2. 6

                                              I think that the problem is fanboyism, echo chambers and ideologies.

                                              I might be wrong, so please don’t consider this an accusation. But you writing this sounds like someone hearing that systemd is bad, therefore never looking at it, yet copying it. Then one tries it and finding out that baseless prejudices were in fact baseless.

                                              After that the assumption is that everyone else must have been doing the same and one is enlightened now to see it’s actually really cool.

                                              I think that this group behavior and blindly copying opinions is one of the worst things in IT these days, even though of course it’s not limited to this field.

                                              A lot of people criticizing systemd actually looked at systemd, really deep, maybe even built stuff on it, or at least worked with it in production as sysadmin/devop/sre/…

                                              Yes, I have used systemd, yes I understand why decisions we’re taken, where authors if the software were going, read specs of the various parts (journald for example), etc.

                                              I think I have a pretty good understanding compared to at least most people that only saw it from a users perspective (considering writing unit files to be users perspective as well).

                                              So I could write about that in my CV and be happy that I can answer a lot of questions regarding systemd, advocate its usage to create more demand and be happy.

                                              To sum it up: I still consider systemd to be bad on multiple layers, both implementation and some ideas that I considered great but then through using it seeing that it was a wrong assumption. By the way that’s the thing I would not blame anyone for. It’s good that stuff gets tried, that’s how research works. It’s not the first and not the last project that will come out sounding good, to only find out a lot of things either doesn’t make a difference or make it worse.

                                              I am a critic of systemd but I agree that there’s a lot of FUD as well. Especially when there’s people that blame everything, including own incompetence on systemd. Nobody should ever expect a new project to be a magic bullet. That’s just dumb and I would never blame systemd for trying a different approach or for not being perfect. However I think it has problems on many levels. While I think the implementation isn’t really good that’s something that can be fixed. However I think some parts of the concept level are either pretty bad or have turned out to be bad decisions.

                                              I was very aware that especially in the beginning the implementation was bad. A lot got better. That’s to be expected. However next to various design decisions I consider bad I think many more were based on ideas that I think to most people in IT sound good and reasonable but in the specific scenarios that systemd is used it at least in my experience do not work out at all or only work well in very basic cases.

                                              In other words the cases where other solutions are working maybe not optimal, but that aren’t considered a problem worth fixing because the added complexity isn’t worth it systemd really shines. However when something is more complex I think using systemd frequently turns out to be an even worse solution.

                                              While I don’t wanna go into detail because I don’t think this is the right format for an actual analysis I think systemd in this field a lot in common with both configuration management and JavaScript frameworks. They tend to be amazing for use cases that are simple (todo applications for example), but together with various other complexities often make stuff unnecessarily complicated.

                                              And just like with JavaScript frameworks and configuration management there’s a lot of FUD, ideologies, echochambers, following the opinion of some thought leaders, and very little building your own solid opinion.

                                              Long story short. If you criticize something without knowing what it is about then yes that’s dumb and likely FUD. However assuming that’s the only possible reason for someone criticizing software is similarly dumb and often FUD regarding this opinion.

                                              This by the way also works the reverse. I frequently see people liking software and echoing favorable statements for the same reasons. Not understanding what they say, just copying sentences of opinion leaders, etc.

                                              It’s the same pattern, just the reversal, positive instead of negative.

                                              The problem isn’t someone disliking or liking something, but that opinions and thoughts are repeated without understanding which makes it hard to have discussions and arguments that give both sides any valuable insides or learnings

                                              Then things also get personal. People hate on Poetteing and think he is dumb and Poetteing thinks every critic is dumb. Just because that’s a lot of what you see when every statement is blindly echoed.

                                              1. 1

                                                That’s nice, but the implication of the anti-systemd chorus was that sys v init was good enough. Not all of these other “reasonable objections” that people are breathless to mention.

                                                The timbre reminded me of people who say autotools is preferrable to cmake. People making a lot of noise about irrelevant details and ignoring the net gain.

                                                But you writing this sounds like someone hearing that systemd is bad, therefore never looking at it, yet copying it.

                                                No, I’m reacting to the idea that the systemd controversy took up any space in my mind at all. It’s good software. It doesn’t matter if X or Y is technically better, the popular narrative was that systemd is a negative thing, a net-loss.

                                                1. 2

                                                  In your opinion it’s good software and you summed up the “anti-systemd camp” with “sys v init was good enough” even though people from said “anti-systemd camp” on this very thread disagreed that that was their point.

                                                  To give you an entirely different point of view, I’m surprised you don’t want to know anything about a key piece of a flagship server operating systems (taking that one distro is technically an OS) affecting the entire eco system and unrelated OS’ (BSDs etc.) that majorly affects administration and development on Linux-based systems. Especially when people have said there are clear technical reasons for disliking the major change and forced compliance with “the new way”.

                                                  1. 2

                                                    you summed up the “anti-systemd camp” with “sys v init was good enough” even though people from said “anti-systemd camp” on this very thread disagreed that that was their point.

                                                    Even in this very thread no one has actually named a preferred alternative. I suspect they don’t want to be dragged into a discussion of details :)

                                                    affecting the entire eco system and unrelated OS’ (BSDs etc.)

                                                    BSDs would be a great forum for demonstrating the alternatives to systemd.

                                                    1. 2

                                                      Well, considering how many features that suite of software has picked up, there isn’t currently one so that shortens the conversation :)

                                                      launchd is sort of a UNIX alternative too, but it’s currently running only on MacOS and it recently went closed source.

                                              2. 3

                                                It violates unix philosohpy

                                                That accusation was also made against neovim. The people muttering this stuff are slashdot markov chains, they don’t have any idea what they’re talking about.

                                                i don’t follow your reasoning. why is it relevant that people also think neovim violates the unix philosophy? are you saying that neovim conforms to the unix philosophy, and therefore people who say it doesn’t must not know what they’re talking about?

                                                1. 1

                                                  are you saying that neovim conforms to the unix philosophy, and therefore people who say it doesn’t must not know what they’re talking about?

                                                  When the implication is that Vim better aligns with the unix philosophy, yes, anyone who avers that doesn’t know what they’re talking about. “Unix philosophy” was never a goal of Vim (”:help design-not” was strongly worded to that effect until last year, but it was never true anyways) and shows a deep lack of familiarity with Vim’s features.

                                                  Some people likewise speak of a mythical “Vim way” which again means basically nothing. But that’s a different topic.

                                                  1. 1

                                                    vim does have fewer features which can be handled by other tools though right? not that vim is particularly unixy, but we’re talking degrees

                                                2. 1

                                                  The people muttering this stuff are slashdot markov chains, they don’t have any idea what they’re talking about

                                                  I’ll bookmark this comment just for this description.

                                                1. 3

                                                  This is a little late, but this morning was spent removing excess banana plants.

                                                  Adding a downpipe to the new gutter is this afternoons goal, and maybe work on some of the 6.x goals for Bamboo Framework later tonight.

                                                  1. 2

                                                    This sounds really cool (as does the other hg extensions he mentions after absorb, and the whole approach: you shouldn’t need a PhD to use a DVCS with confidence)

                                                    Unfortunately I can’t find much information about actually using this extension. Can anyone expand on that?

                                                    1. 1

                                                      Most but not all of the features they’re talking about are in their repo (some are already built-in):

                                                      https://bitbucket.org/facebook/hg-experimental/

                                                      There’s work to slowly upstream this work, but hg does move slowly.

                                                      Also note that their README is out of date. Sorry about that.

                                                    1. 4

                                                      The builders finished renovating our house yesterday. So now I have 36 sq.m (~400 sq.ft) of new lounge/dining room to put lights, power, furniture in. And the builders are back in ~1 week to add the carport extension, which will mean more lights and power…. so plenty of DIY stuff going on.

                                                      With the builders no longer running hammer-drills at 7am I can also get in more full work “days” (which aren’t really during the day) with $CLIENT1 so, that’ll mean progressing with the Bamboo Models migration mostly.

                                                      1. 3

                                                        A question for anyone who might have context – from this piece it seems like they have a cluster per restaurant, which doesn’t make much sense in terms of complexity versus payoff to my mind. The thing that would make more sense and be very interesting is if they’re having these nodes join a global or regional k8s cluster. Am I misreading this?

                                                        1. 2

                                                          They seem to be using NUCs as their Kubernetes nodes, so the hardware cost isn’t going to be too great.

                                                          I imagine it’s down to a desire to not be dependent on an internet connection to run their POS and restaurant management applications, I’m sure the costs of a connection with an actual SLA are obscene compared to the average “business cable” connection you can use if it doesn’t need to be super reliable.

                                                          1. 3

                                                            Still, restaurants have been using computers for decades. It looks as if they have a tech team that’s trying very hard to apply trendy tools and concepts (Kuberneetes, “edge computing”) to a solved problem. I’d love to be proven wrong, though.

                                                            1. 3

                                                              I’ve never been to one of these restaurants but I can’t imagine anything that needs a literal cluster to run its ordering and payments system.

                                                              Sounds like an over engineered Rube Goldberg machine because of some resume/cv padding.

                                                              1. 2

                                                                While restaurants certainly have been using computers for decades the kind of per location ordering integrations needed for today’s market are pretty diverse:

                                                                • Regular orders
                                                                • Delivery services in area (Postmates, dd, caviar, eat24, ubereats)
                                                                • Native app ordering
                                                                • Coupons
                                                                • App coupons

                                                                If you run a franchise like Chick-fil-A, you don’t want a downtime in the central infrastructure to prevent internet orders at each location, as it would make your franchisees upset that their business was impacted. You also want your franchisees to have easy access to all the ordering methods available in their market. This hits both as it allows them to run general compute using the franchisee’s internet, and easily deploy new integrations, updates, etc w/o an IT person at the location.

                                                                I have a strong suspicion that this is why I see so many Chick-fil-As on almost every food delivery service.

                                                                Beyond that, it’s also easier and cheaper to deploy applications onto a functional k8s/nomad/mesos stack than VMS or other solutions because of available developer interest and commodity hw cost. Most instability I’ve seen in these setups is a function of how many new jobs or tasks are added. Typically if you have pretty stable apps you will have fewer worries than other deployment solutions. Not saying there aren’t risks, but this definitely simplifies things.

                                                                As an aside I would say that while restaurants have been using computers for decades they haven’t necessarily been using them well and lots of the systems were proprietary all in one (hw/sw/support) ‘solutions.’ That’s changed a bit but you’ll still see lots of integrated POS systems that are just a tablet+app+accessories in a nice swivel stand. I’ve walked into places where they were tethering their POS system to someone’s cell phone because the internet was down and the POS app needed internet to checkout (even cash).

                                                              2. 1

                                                                Most retail stores like this use a $400/mo T1 which is 1.5mbit/sec (~185kb/sec) symmetrical – plenty for transaction processing but not much else. Their POS system is probably too chatty to run on such a low bandwidth link.

                                                              3. 1

                                                                It could just be a basic, HA setup or load balancing cluster on several, cheap machines. I recommended these a long time ago as alternatives to AS/400’s or VMS clusters which are highly reliable, but pricey. They can also handle extra apps, provide extra copies of data to combat bitrot, support rolling upgrades, and so on. Lots of possibilities.

                                                                People can certainly screw them up. You want person doing setup to know what they’re doing. I’m just saying there’s benefits.

                                                              1. 12

                                                                a harrowing tale from a GCP customer who had their entire project shut down by Google’s fraud protection system, with full deletion scheduled for 3 business days later

                                                                This is where the advantage of small hosting companies is. With prgmr, I’m dealing with a few friendly sysadmins who email me about my OS freezing, instead of a Corporate Monster with Automatic Fraud Detection (will probably be marketed as AI™ soon) that Deletes My Account.

                                                                1. 2

                                                                  Just use several providers or with several accounts, to be sure that you’re fine. If you’ve been using the rights tools and workflow, this would be totally feasible to achieve and probably a good guarantee.

                                                                  1. 4

                                                                    Are you sure the use of “just” is justified with a suggestion to use multiple providers? Surely the complexity and cost would be significantly different? I’m puzzled.

                                                                    1. 2

                                                                      I guess it all depends of your size and your needs. Although, I do apologies for the “just”, it’s not helping anyone. What I meant is that for many usecases, using tools like terraform,ansible,docker (with all the hype around), can help a lot to be compatible with multiple providers. For example using terraform to setup vms,storage and networks on aws/gcloud/azure (or even on vmware), which is an initial cost to work on, but then would save a lot of time when scaling or just for disaster recovery. The output of terraform can be used by ansible to deploy on the machines.

                                                                      I know there are issues with some providers, but the low cost and the flexibility is also a great advantage.

                                                                      1. 1

                                                                        Thanks for elaborating, that makes sense.

                                                                  2. 1

                                                                    This is exactly why I still use Rimu - if I need to, I can contact the owner of the company, and the rest of the time there are actual people available to provide extra assistance/Info if required.

                                                                  1. 5
                                                                    • For $CLIENT1 it’s mostly following on from last week

                                                                      • The initial model migration was pretty straightforward.
                                                                      • This week I’ll hopefully clear up any remaining questions/uncertainty about the new model system for the other dev’s on the team, and proceed with migrating some more of the ‘simpler’ models across to Bamboo.
                                                                      • Plus of course, regular bug fixes/prod incident handling
                                                                    • For $HOME we’ll probably need to make some decisions about internal light placement soon, so the builders can run wiring inside the panels for us. We’re into wet season now, so it’s nice to know it’ll be all sealed up soon.

                                                                    1. 11

                                                                      One culture note I find really interesting: I remember 3-4 years ago a lot of people were griping about how “full stack” wasn’t real. Almost all of them argued that backend was so complicated you needed a specialist to do it well.

                                                                      Now we’re seeing the exact same articles but now it’s the frontend that’s too complicated.

                                                                      When it comes to specialisation, generalists underestimate the benefits but specialists overestimate the necessity.

                                                                      1. 5

                                                                        I think it’s related to how complicated front end has become.

                                                                        I think the problem is that most people who call themselves “full stack” are, like most of us, quite highly experienced in one area, and have enough working knowledge to get by in the other areas.

                                                                        Every “full stack vs not” discargument I’ve seen has boiled down to “full stack” people claiming that more specialised people are “single skill”.

                                                                        I’ve never met or worked with anyone that did just one thing. In most teams/orgs I’d expect people to have some experience across most of the tech stack - but that doesn’t make them “full stack” any more than it makes me a mechanic because I can change a tire or replace a car battery, or a builder because I can put up a shelf.

                                                                        That doesn’t mean there isn’t a place for people who are (or seemingly claim themselves to be, ala “full stack”) more evenly experienced over the stack than those who specialise, but in my experience these people tend to be the ones who just brush off anything that’s beyond them as “we don’t need to worry about it”.

                                                                        1. 1

                                                                          I acknowledge that you used “most” and “tend” to allow exceptions, but your argument still rubs me the wrong way.

                                                                          I am one of those people who has described himself as a full-stack web developer. I feel comfortable doing this because I have designed and implemented back-ends and front-ends of services that scaled to hundreds of thousands of users. Obviously there exist much larger scales, but I think ~million users will cover the needs of most web services out there and in some countries, like Slovenia where I live, it will cover all of them. It does not seem unreasonable to me to have a term for noting that you can build any part of it if necessary.

                                                                          I do not claim to know everything I need to know at all times, but I do know enough about everything relevant that I can tell where the gaps are and fill them in a reasonable time. I find this perfectly reasonable in the same way as needing to learn a new language for a project does not disqualify a developer from still being a developer.

                                                                          I am not alone and have colleagues who can do the same or better. None of us argue that we are all anyone needs and even on smaller projects it is generally better if people focus on fewer things. Most of my work lately is on front-end and I certainly am not stupid enough to not notice that specialists can do many things better than me. If your project can benefit from that and can afford hiring such person, it would be stupid not to.

                                                                          As you say yourself, full-stack is really just a description for a different distribution of skills and experience over the stack and you can be a competent developer over huge part of it if you pay attention to what and why you are learning something and avoid switching tools and frameworks for the currently fashionable one every half a year.

                                                                          I don’t doubt most full-stack developers are bad at their job in the same way as most of any group of developers are (X specialists, Python developers…). Likewise no group of practitioners of noticeable size lacks individuals disparaging other groups.

                                                                          1. 1

                                                                            I did specifically qualify it as anecdotal:

                                                                            in my experience these people tend to be

                                                                            The rest of your comment just seems to reaffirm what I said though - fullstack is generally just a broader, shallower set of experience rather than narrow, deeper with someone more specialised.

                                                                      1. 1

                                                                        I’d never heard of envoy before this, and looking it’s yaml config I’m not in a rush to dig deeper.

                                                                        1. 1

                                                                          Envoy is really designed for machine configuration (the YAML is not for the faint of heart). Most of the advanced config is done via gRPC APIs. That’s one reason why we created Ambassador to manage the Envoy configuration for you.

                                                                        1. 4

                                                                          Following on from last week:

                                                                          • For $HOME - holy shit, these builders are quick, so no doubt various renovation related things: taking down electrical fixtures in their way, choosing materials, etc

                                                                          • For $CLIENT1 - I’m actually working on the initial ZF1 > Bamboo cutover, which initially means working out if it’s even remotely possible to have a compatibility shim to map whatever the fuck model concept ZF1 uses, into the ~active record pattern Bamboo implements.

                                                                          1. 3

                                                                            This sounds like the old tired line “RAID is not backup”. Which is true, nobody disagrees, and so it is pointless to keep repeating.

                                                                            1. 7

                                                                              FWIW, I continually have discussions with people that believe having a distributed database means they don’t need backups.

                                                                              1. 2

                                                                                It’s less about someone actively disagreeing it’s about people naively thinking it’s the same concept

                                                                              1. 25

                                                                                Nice article. I must admin that I am a systemd fan. I much prefer it to the soup of raw text in rc.d folders. Finally, an init system system for the 1990s.

                                                                                1. 13

                                                                                  I’ve never had a problem doing anything with systemd myself - I think a lot of the hate towards it stems from the attitude of the project owners, and how they don’t make any effort to cooperate with other projects (most notably, IMO, the Linux kernel folks). Here’s a couple of interesting mailing list messages that demonstrate that:

                                                                                  1. 11

                                                                                    Exactly.

                                                                                    I was initially skeptical about the debug ability of a systemd unit, but the documentation covers things to great depth, and I’m a convert to the technical merits. Declarative service files, particularly when you use a ‘drop-in’, are a definite step up from the shell scripts of sysvinit.

                                                                                    The way the project tries to gobble up /everything/ is a concern though, given their interactions (or lack thereof) with other parts of the community.

                                                                                    1. 2

                                                                                      My impression is that the resistance to systemd stems from it not being unixy. Not being Debiany, even.

                                                                                      I use for i in ..., sed, grep, awk, find, kill -SIGHUP, lsof, inotify, tee, and tr all damned day to mange my system, and systemd has left me blind and toothless.

                                                                                      I’m still working on my LFS-based replacement for my various Debian desktops, vms, and laptop.

                                                                                      1. 1

                                                                                        Declarative service files, particularly when you use a ‘drop-in’, are a definite step up from the shell scripts of sysvinit.

                                                                                        I’ve never found “systemd vs sysvinit shell scripts” to be a particularly compelling argument. “Don’t use sysvinit shell scripts” is a perfectly fine argument, but doesn’t say much about systemd. There are loads of init systems out there, and it seemed to me that systemd was never in competition with sysvinit scripts, it was in competition with other new-fangled init systems, especially upstart which was widely deployed by Ubuntu.

                                                                                        1. 1

                                                                                          In the case of Debian, it’s basically sysvinit or systemd.

                                                                                          1. 2

                                                                                            That’s still not much of an argument for systemd; it’s just passing the buck to the Debian developers, and going with whichever they chose. That’s an excellent thing for users, sysadmins, etc. to do, but doesn’t address the actual question (i.e. why did the Debian devs make that choice?).

                                                                                            According to Wikipedia, the initial release of systemd was in 2010, at which point Ubuntu (a very widely-deployed Debian derivative) had been using upstart by default for 4 years.

                                                                                            Debian’s choice wasn’t so much between sysvinit or systemd, it was which non-sysvinit system to use; with the highest-profile contenders being systemd (backed by RedHat) and upstart (backed by Canonical). Sticking with sysvinit would have been an abstain, i.e. “we know it’s bad, but the alternatives aren’t better enough to justify a switch at the moment”. In other words sysvinit’s only “feature” is the fact that it is already in widespread use, with all of the benefits that brings (pre-existing code snippets, documentation, blogposts, troubleshooting forums, etc.).

                                                                                            These days systemd has that “feature” too, since it’s used by so many distros (including Debian, as you say), which was the last nail in sysvinit’s coffin: at this point sysvinit is mostly hanging on as a legacy option (Debian in particular cares very deeply about stability and compatibility). Choosing between Debian sysvinit and Debian systemd isn’t so much a choice of init system, it’s a choice of whether or not to agree with the Debian developers’ choice to switch init system. And that choice was between systemd, upstart, initng, runit, daemontools, dmd, etc. They abstained (stuck with sysvinit) for many years, until around 2015 when the systemd vs upstart competition was resoundingly won by systemd, with Ubuntu switching away from upstart and Debian switching away from sysvinit.

                                                                                            As I saw all of this going on, my interpretation was:

                                                                                            • Around 2005 every popular distro was using sysvinit because of its entrenched base, a few users advocated for alternatives like initng but the distros didn’t find the improvements to be worth the cost.
                                                                                            • Ubuntu switched to upstart, making init systems a hot topic: sysvinit became viewed as legacy, upstart was being looked at closely by other distros and it seemed like, once it got enough real-world usage, many might switch over.
                                                                                            • Systemd appeared, inspired by Apple’s launchd, and gradually gained users. At this point sysvinit was already seen as legacy and the question was what systemd offered that upstart didn’t.
                                                                                            • Debian debated switching init system, and with input from Ubuntu developers they both agreed that systemd was the better option (from my understanding, systemd’s “lazy” approach was a fundamentally better fit to the init problem than upstart’s “eager” approach). Upstart basically died at this point.
                                                                                            • All subsequent debates about systemd focus on how it’s better than sysvinit, which was never really in question.

                                                                                            To me, comparing systemd to sysvinit is like those shampoo adverts which claim their product gives an X% improvement, but the fine-print says that’s compared to not washing ;)

                                                                                            1. 1

                                                                                              OpenRC is drop-in and works perfectly fine. I dropped it in and I’m using it on all my installs with no issues.

                                                                                              1. 1

                                                                                                I think maybe you’ve misunderstood me.

                                                                                                I don’t mean you can install systemd and it will continue to work with sysvinit scripts.

                                                                                                I’m referring to systemd’s “drop-in” unit configurations. You can override specific parameters of a unit without having to replace the whole thing.

                                                                                                https://coreos.com/os/docs/latest/using-systemd-drop-in-units.html