1. 15

    I recently discovered how horribly complicated traditional init scripts are whilst using Alpine Linux. OpenRC might be modern, but it’s still complicated.

    Runit seems to be the nicest I’ve come across. It asks the question “why do we need to do all of this anyway? What’s the point?”

    It rejects the idea of forking and instead requires everything to run in the foreground:

    /etc/sv/nginx/run:

    #!/bin/sh
    exec nginx -g 'daemon off;'
    

    /etc/sv/smbd/run

    #!/bin/sh
    mkdir -p /run/samba
    exec smbd -F -S
    

    /etc/sv/murmur/run

    #!/bin/sh
    exec murmurd -ini /etc/murmur.ini -fg 2>&1
    

    Waiting for other services to load first does not require special features in the init system itself. Instead you can write the dependency directly into the service file in the form of a “start this service” request:

    /etc/sv/cron/run

     #!/bin/sh
     sv start socklog-unix || exit 1
     exec cron -f
    

    Where my implementation of runit (Void Linux) seems to fall flat on its face is logging. I hoped it would do something nice like redirect stdout and stderr of these supervised processes by default. Instead you manually have to create a new file and folder for each service that explicitly runs its own copy of the logger. Annoying. I hope I’ve been missing something.

    The only other feature I can think of is “reloading” a service, which Aker does in the article via this line:

    ExecReload=kill -HUP $MAINPID

    I’d make the argument that in all circumstances where you need this you could probably run the command yourself. Thoughts?

    1. 6

      Where my implementation of runit (Void Linux) seems to fall flat on its face is logging. I hoped it would do something nice like redirect stdout and stderr of these supervised processes by default. Instead you manually have to create a new file and folder for each service that explicitly runs its own copy of the logger. Annoying. I hope I’ve been missing something.

      The logging mechanism works like this to be stable and only lose logs in case runsv and the log service would die. Another thing about separate logging services is that stdout/stderror are not necessarily tagged, adding all this stuff to runsv would just bloat it.

      There is definitively room for improvements as logger(1) is broken since some time in the way void uses it at the moment (You can blame systemd for that). My idea to simplify logging services to centralize the way how logging is done can be found here https://github.com/voidlinux/void-runit/pull/65. For me the ability to exec svlogd(8) from vlogger(8) to have a more lossless logging mechanism is more important than the main functionality of replacing logger(1).

      1. 1

        Ooh thankyou, having a look :)

      2. 6

        Instead you can write the dependency directly into the service file in the form of a “start this service” request

        But that neither solves starting daemons in parallel, or even at all, if they are run in the ‘wrong’ order. Depending on network being setup, for example, brings complexity to each of those shell scripts.

        I’m of the opinion that a dsl of whitelisted items (systemd) is much nicer to handle than writing shell scripts, along with the standardized commands instead of having to know which services that accepts ‘reload’ vs ‘restart’ or some other variation in commands - those kind of niceties are gone when the shell scripts are individually an interface each.

        1. 6

          The runit/daemontools philosophy is to just keep trying until something finally runs. So if the order is wrong, presumably the service dies if a dependent service is not running, in which case it’ll just get restart. So eventually things progress towards a functioning state. IMO, given that a service needs to handle the services it depends on crashing at any time anyways to ensure correct behaviour, I don’t feel there is significant value in encoding this in an init system. A dependent service could also be moved to running on another machine which this would not work in as well.

          1. 3

            It’s the same philosophy as network-level dependencies. A web app that depends on a mail service for some operations is not going to shutdown or wait to boot if the mail service is down. Each dependency should have a tunable retry logic, usually with an exponential backoff.

          2. 4

            But that neither solves starting daemons in parallel, or even at all, if they are run in the ‘wrong’ order.

            That was my initial thought, but it turns out the opposite is true. The services are retried until they work. Things are definitely paralleled – there is not “exit” in these scripts, so there is no physical way of running them in a linear (non-parallel) nature.

            Ignoring the theory: void’s runit provides the second fastest init boot I’ve ever had. The only thing that beats it is a custom init I wrote, but that was very hardware (ARM Chromebook) and user specific.

          3. 5

            Dependency resolving on daemon manager level is very important so that it will kill/restart dependent services.

            runit and s6 also don’t support cgroups, which can be very useful.

            1. 5

              Dependency resolving on daemon manager level is very important so that it will kill/restart dependent services

              Why? The runit/daemontools philsophy is just to try to keep something running forever, so if something dies, just restart it. If one restarts a service, than either those that depend on it will die or they will handle it fine and continue with their life.

              1. 4

                either those that depend on it will die or they will handle it fine

                If they die, and are configured to restart, they will keep bouncing up and down while the dependency is down? I think having dependency resolution is definitely better than that. Restart the dependency, then the dependent.

                1. 4

                  Yes they will. But what’s wrong with that?

                  1. 2

                    Wasted cycles, wasted time, not nearly as clean?

                    1. 10

                      It’s a computer, it’s meant to do dumb things over and over again. And presumably that faulty component will be fixed pretty quickly anyways, right?

                      1. 5

                        It’s a computer, it’s meant to do dumb things over and over again

                        I would rather have my computer do less dumb things over and over personally.

                        And presumably that faulty component will be fixed pretty quickly anyways, right?

                        Maybe; it depends on what went wrong precisely, how easy it is to fix, etc. We’re not necessarily just talking about standard daemons - plenty of places run their own custom services (web apps, microservices, whatever). The dependency tree can be complicated. Ideally once something is fixed everything that depends on it can restart immediately, rather than waiting for the next automatic attempt which could (with the exponential backoff that proponents typically propose) take quite a while. And personally I’d rather have my logs show only a single failure rather than several for one incident.

                        But, there are merits to having a super-simple system too, I can see that. It depends on your needs and preferences. I think both ways of handling things are valid; I prefer dependency management, but I’m not a fan of Systemd.

                        1. 4

                          I would rather have my computer do less dumb things over and over personally.

                          Why, though? What’s the technical argument. daemontools (and I assume runit) do sleep 1 second between retries, which for a computer is basically equivalent to it being entirely idle. It seems to me that a lot of people just get a bad feeling about running something that will immediately crash.

                          Maybe; it depends on what went wrong precisely, how easy it is to fix, etc. We’re not necessarily just talking about standard daemons - plenty of places run their own custom services (web apps, microservices, whatever).

                          What’s the distinction here? Also, with microservices the dependency graph in the init system almost certainly doesn’t represent the dependency graph of the microservice as it’s likely talking to services on other machines.

                          I think both ways of handling things are valid

                          Yeah, I cannot provide an objective argument as to why one should prefer one to the other. I do think this is a nice little example of the slow creep of complexity in systems. Adding a pinch of dependency management here because it feels right, and a teaspoon of plugin system there because we want things to be extensible, and a deciliter of proxies everywhere because of microservices. I think it’s worth taking a moment every now and again and stepping back and considering where we want to spend our complexity budget. I, personally, don’t want to spend it on the init system so I like the simple approach here (especially since with microservies the init dependency graph doesn’t reflect the reality of the service anymore). But as you point out, positions may vary.

                          1. 2

                            Why, though? What’s the technical argument

                            Unnecessary wakeup, power use (especially for a laptop), noise in the logs from restarts that were always bound to fail, unnecessary delay before restart when restart actually does become possible. None of these arguments are particularly strong, but they’re not completely invalid either.

                            We’re not necessarily just talking about standard daemons …

                            What’s the distinction here?

                            I was trying to point out that we shouldn’t make too many generalisations about how services might behave when they have a dependency missing, nor assume that it is always ok just to let them fail (edit:) or that they will be easy to fix. There could be exceptions.

                        2. 2

                          Perhaps wandering off topic, but this is a good way to trigger even worse cascade failures.

                          eg, an RSS reader that falls back to polling every second if it gets something other than 200. I retire a URL, and now a million clients start pounding my server with a flood of traffic.

                          There are a number of local services (time, dns) which probably make some noise upon startup. It may not annoy you to have one computer misbehave, but the recipient of that noise may disagree.

                          In short, dumb systems are irresponsible.

                          1. 2

                            But what is someone supposed to do? I cannot force a million people using my RSS tool not to retry every second on failure. This is just the reality of running services. Not to mention all the other issues that come up with not being in a controlled environment and running something loose on the internet such as being DDoS’d.

                            1. 2

                              I think you are responsible if you are the one who puts the dumb loop in your code. If end users do something dumb, then that’s on them, but especially, especially, for failure cases where the user may not know or observe what happens until it’s too late, do not ship dangerous defaults. Most users will not change them.

                              1. 1

                                In this case we’re talking about init systems like daemontools and runit. I’m having trouble connecting what you’re saying to that.

                        3. 2

                          If those thing bother you, why run Linux at all? :P

                      2. 2

                        N.B. bouncing up and down ~= polling. Polling always intrinsically seems inferior to event based systems, but in practice much of your computer runs on polling perfectly fine and doesn’t eat your CPU. Example: USB keyboards and mice.

                        1. 2

                          USB keyboard/mouse polling doesn’t eat CPU because it isn’t done by the CPU. IIUC the USB controller generates an interrupt when data is received. I feel like this analogy isn’t a good one (regardless). Checking a USB device for a few bytes of data is nothing like (for example) starting a Java VM to host a web service which takes some time to read its config and load its caches only to then fall over because some dependency isn’t running.

                        2. 1

                          Sleep 1 and restart is the default. It is possible to have another behavior by adding a ./finish script to the ./run script.

                      3. 2

                        I really like runit on void. I do like the simplicity of SystemD target files from a package manager perspective, but I don’t like how systemd tries to do everything (consolekit/logind, mounting, xinet, etc.)

                        I wish it just did services and dependencies. Then it’d be easier to write other systemd implementations, with better tooling (I’m not a fan of systemctl or journalctl’s interfaces).

                        1. 1

                          You might like my own dinit (https://github.com/davmac314/dinit). It somewhat aims for that - handle services and dependencies, leave everything else to the pre-existing toolchain. It’s not quite finished but it’s becoming quite usable and I’ve been booting my system with it for some time now.

                      4. 4

                        I’d make the argument that in all circumstances where you need this you could probably run the command yourself. Thoughts?

                        It’s nice to be able to reload a well-written service without having to look up what mechanism it offers, if any.

                        1. 5

                          Runits sv(8) has the reload command which sends SIGHUP by default. The default behavior (for each control command) can be changed in runit by creating a small script under $service_name/control/$control_code.

                          https://man.voidlinux.eu/runsv#CUSTOMIZE_CONTROL

                          1. 1

                            I was thinking of the difference between ‘restart’ and ‘reload’.

                            Reload is only useful when:

                            • You can’t afford to lose a few seconds of service uptime (OR the service is ridiculously slow to load)
                            • AND the daemon supports an on-line reload functionality.

                            I have not been in environments where this is necessary, restart has always done me well. I assume that the primary use cases are high-uptime webservers and databases.

                            My thoughts were along the lines o: If you’re running a high-uptime service, you probably don’t care about the extra effort of writing ‘killall -HUP nginx’ than ‘systemctl reload nginx’. In fact I’d prefer to do that than take the risk of the init system re-interpreting a reload to be something else, like reloading other services too, and bringing down my uptime.

                          2. 3

                            I hoped it would do something nice like redirect stdout and stderr of these supervised processes by default. Instead you manually have to create a new file and folder for each service that explicitly runs its own copy of the logger. Annoying. I hope I’ve been missing something.

                            I used to use something like logexec for that, to “wrap” the program inside the runit script, and send output to syslog. I agree it would be nice if it were builtin.

                          1. 1

                            I liked this article: How we built Watsi Coverage without stable electricity, WiFi, or email which discusses a number of problems that sounds like yours.

                            1. 2

                              I’d actually heard of these guys but hadn’t dug into them before. Man what I would give to have a team of people to do development… They’re running a pretty close parallel to ours but we’re specifically geared towards disease surveillance and outbreak management versus long-term health management (though we do stray into that when countries don’t have any viable alternative in place).

                              Thanks for sharing this, I might actually contact these guys at some point to see if we can integrate in some meaningful ways.

                            1. 3

                              Aaron Swartz made an HTML-to-Markdown converter in 2004, together with Markdown which he apparently ~co-authored (!) : http://www.aaronsw.com/weblog/001189

                              My usage of Swartz’s html2text program, was persisting markdown versions of online articles locally: https://github.com/chelmertz/cleader

                              1. 2

                                Ha, it’s fun to see Bob Ippolito’s slightly outraged comment at the bottom of Aaron’s post: “what’s wrong with RST?”

                              1. -7

                                “CommonMark compliant” … lol yeah, forget it.

                                it’s a fake “standard.” They didn’t think of Markdown, invent it, or do anything to help it’s advance.

                                No one needs to “comply” with “CommonMark. The CommonMark project has been trying to take ownership of Markdown for years. It’s ridiculous and annoying.

                                1. 8

                                  They didn’t think of Markdown, invent it, or do anything to help it’s advance.

                                  You say “it”, when one of the reasons CommonMark exists, is that there is no single “it”, there are divergent implementations/extensions. The original implementation is not a specification, and it ‘has bugs’/is ambiguous.

                                  I am not too well-informed; who is better suited to “do anything to help it’s advance”? And what did CM do wrong?

                                  1. 5

                                    I presume what leeflannery is referring to is that John Gruber is BDFL of Markdown and the only authority. Only he’s more like the Absent Dictator for Life, which resulted in a proliferation of implementations that sometimes conflicted with each other and Markdown and led Jeff Atwood (of StackOverflow) to establish CM.

                                    Atwood’s story is here - https://blog.codinghorror.com/standard-markdown-is-now-common-markdown/

                                    And here’s the Github issue with some more detail - https://github.com/commonmark/CommonMark/issues/19

                                  2. 6

                                    it’s a fake “standard.” They didn’t think of Markdown, invent it, or do anything to help it’s advance.

                                    Every standard is fake, if you want to be pedantic about it. No C compiler has to comply with the ANSI standard, no web server has to implement the HTTP spec. Nor are these standards set in stone (good ones at least), and they develop with trends, new needs and realization of previous shortcomings.

                                    What the CommonMark project want to achieve isn’t to make up some unrelated markup language, or to feel special for themselves, but “propose a standard, unambiguous syntax specification for Markdown, along with a suite of comprehensive tests to validate Markdown implementations”. And as I’ve already mention in this thread, these people aren’t nobodies, but instead it was initiated by some of the more major figures in the “markdown scene”. Sure, they don’t “own” markdown (whatever that is supposed to mean), but they are proposing a common ground to strongly define the syntax and the semantics of a markdown parser, having have already published revisions, updating their specification.

                                    If it’s a good standard, people will adopt it when doing something related to markdown, otherwise they won’t. This doesn’t look like something “stupid”, if you were to ask me, but rather an incentive to create a well defined, common sensical, sane specification, to improve the current state of markdown – and if one doesn’t like it, there’s absolutely no need to worry about it or pay any attention whatsoever to the project.

                                  1. 3

                                    There are a lot of great blog posts written by ‘recursers’. Those posts are usually spread out on different blogs though. Does anyone know if they are aggregated somewhere?

                                    1. 6

                                      There’s an internal tool that we (recursers) use that aggregates them, but unfortunately it’s not public. Perhaps someday someone will write a public view for it. (hint hint to current recursers)

                                      1. 7

                                        FYI, other projects call this their planet. See http://planet.mozilla.org/ or http://planet.debian.org/ or http://planet.ubuntu.com/

                                        1. 1

                                          Thanks, that’s a cool term I hadn’t heard before.

                                        2. 1

                                          I’d really like this.

                                        3. 1

                                          I was just thinking today that a huge benefit of doing Recurse Center was exposure to so many great blogs/bloggers I otherwise wouldn’t have encountered.

                                        1. 5
                                          • A major Vue-based refactor at work, which has been a lot of fun. At the same time I’m a bit paranoid that I’ve been taking too long. If anyone can speak to dealing with that paranoia in healthy ways, it’d help me feel better!
                                          • Working exercise back into my life after a puzzling injury - it seemed like De Quervain’s Tenosynovitis but it took three times as long for symptoms to ease up
                                          • Wedding planning
                                          1. 5

                                            Some tips regarding feeling slow:

                                            • discuss solutions with coworkers
                                            • demo results for stakeholder often
                                            • break down work items before/during coding to get the feeling of checking items off the list
                                            • if it suits your work place: get your work into the main branch early & often, I really think it helps to balance the perfect (but slow) solution vs making tangible progress; it feels nice, and if the issue really drags out so that it needs to be aborted, there was a little bit of progress delivered

                                            I think it is mostly a communications issue. Usually the daily standups catches things that take too long, which should make team members want to help you in someway. Most of the time when I feel like that, it is remedied by talking to someone, or even rubber ducking.

                                            I am sure there are other tips if you google it a bit, good luck!

                                            1. 1

                                              Thank you!

                                          1. 1

                                            One thing I’m always unsure about: Are all these shapes hand-written in SVG? If not, which editor is usually used? I can’t find any good recommendations anywhere and all these presentations omit that part :).

                                            1. 4

                                              I use Inkscape for manual SVG creation and diagrams for programmatic SVG creation.

                                              1. 1

                                                I wasn’t aware of Diagrams! I’m a huge fan of PGF and metapost, but using them for SVG is painful. Inkscape, I find very… clunky?… so I was hoping there was a solution in-between Illustrator and Inkscape.

                                                Thanks!

                                              2. 3

                                                You can use vector-based programs, such as Adobe Illustrator or Inkscape.

                                                1. 1

                                                  Thanks! I know of these, but thanks clarifying, I thought I was missing something!

                                              1. 5

                                                Very nice.

                                                Small bit of feedback: on mobile, Saved is unfortunately placed right under the logo because of how the menu items wraps on small screens. I’ve missclicked/touched a couple of times already, I apparently click the logo a lot.

                                                /Someone with big fingers

                                                1. 7

                                                  It should probably be one of those hamburger menus so the options can drop down and have big hit targets only when you need them.

                                                  1. 6

                                                    That could work, but keeping the most commonly used links visible would save clicks. A bit larger targets would suit me well, but I’m only one data point :)

                                                1. 1

                                                  Something interesting: the debug-me author has included gpg tooling in this, so presumably understands the value of keys, but there seems to be absolutely nothing to stop them just cat ~/.ssh/id_rsa etc, to generally steal anything.

                                                  While the signed-logs bit might help drive this abusive user away, it doesn’t actually prevent the action from occurring.

                                                  1. 3

                                                    debug-me’s strategy seems to be aimed at forgoing black-/whitelisting commands, by having a binary choice of trusting someone with everything or not (I may be wrong, there might be more granular access rights).

                                                    If the developer did do something bad, you’d have proof that they cannot be trusted, which you can share with the world. Knowing that is the case will keep most developers honest.

                                                    https://debug-me.branchable.com/

                                                    When demonstrating “can I view your database?” in the first video, the author wants to believe the best in people, and trusting them with your own computer once they are proven to be who they say they are. Your example seems to be addressed in the long-term, by moving the target to attacking the GPG identity/communication itself.

                                                    I’m just curious, do you have any ideas or workflows that would stop the attack you mention?

                                                    If the user needs help debugging something real, the jail approach seems out of scope unless ‘everything’ is jailed and that mechanism has its own way of sharing sessions. My personal experience is mostly TeamViewer which, just as debug-me, relies on synchronous manual monitoring by the host (i.e. the amount of trust is high).

                                                  1. 2

                                                    I like the layout of Fowler’s Refactoring book and the patterns within. He doesn’t try too hard to sell the values of the patterns, but shows the obective outcome of applying the patterns. To avoid the “I know best, do as I do”, he even provides patterns that are the opposite of each other.

                                                    After doing that, you don’t really have to be explicit about the responsibility and consequences being the developer’s, not some all knowing author.

                                                    I think you need to make the reader/student think and let them come up with the answers themselves, offering guidance and asking hard to answer questions, instead of pushing them into a certain belief. It is too easy to fall into religious arguments otherwise.

                                                    1. 1

                                                      I have only read DJB’s reflections on security from the initial post, and I liked it.

                                                      How would I go about if I wanted to find critique, follow ups or addendums to, for example, that paper? I don’t read a lot of papers, I imagine it takes some skill to find relevant responses to them.

                                                      1. 2

                                                        You can try google searching for link:http://cr.yp.to/qmail/qmailsec-20071101.pdf. This shows who refers to that article elsewhere on the Internet.

                                                      1. 1

                                                        If you would like the functionality markdown-over-htttp-to-pdf, the following works fine and has really nice typography, if you ask me.

                                                        curl https://raw.githubusercontent.com/jquery/jquery/master/README.md | pandoc -f markdown -t latex -o readme.pdf
                                                        

                                                        The dependencies are well worth having installed, considering all of Pandoc’s features.

                                                        1. 9

                                                          jcs, another thought just came to my mind about the “many thousands of usernames” on the user tree. Most of them were invited but don’t really participate at all. Might as well not be thought of as Lobsters “users.” Might be useful to add a filter near top disappearing all inactive accounts to show who the effective user base is. Maybe even just participating on monthly basis if we were aiming for max inclusion. From there, a way to filter the list to see those who submit, vote on, or comment on stories the most. If an order of importance existed, putting submissions or comments with yet-to-be-determined, vote threshold higher than reads or votes since I default to reward active participation.

                                                          This idea came to mind because of my impression when I first came. Site is small and active with many little groups in same overall space focusing on worthwhile things. Neat place. User list looked like a ghost town, though, with piles of names with no contributions or who I’ve never seen comment.

                                                          1. 20

                                                            I visit the site everyday and “contribute” by upvoting the stories I want more off. If a non-robot would request an invite, I would invite them after a minimal amount of interaction.

                                                            Background: I am one of the users with 0 points, who requested an invitation to be able to vote on interesting stories. My own invitation came after having emailed a “comment” to a story directly to the author who had a publicly available email, which worked just fine.

                                                            1. 15

                                                              And now you are no longer one of those 0 points users.

                                                              1. 6

                                                                Likewise. I’ve also been visiting the site every day for probably around a year now. I’m not as experienced as most of those who post frequently, but I’m always learning and I don’t want to comment just for the sake of commenting if I have nothing valuable to add (I respect this community because there is so little noise). I only recently sought out an invite as there have been a few times that I’ve felt the need to correct something or offer a different viewpoint and I completely missed the boat on those opportunities to get involved so thought I’d better be prepared for next time! I understand the preference for active participation, but not everyone on zero points is necessarily inactive. Some of us are just sitting quietly in the corner listening and learning and will speak up when the time is right :).

                                                                1. 4

                                                                  Remember I do include votes as active participation so long as you read the article or comments.That it’s a site collecting links on a page means more people would be readers than writers. Only natural.

                                                              2. 5

                                                                Maybe a first cut approach would be to filter all the karma 0 users?

                                                                1. 3

                                                                  That equivalent to the first pass in my proposal. Just more specific. I’d like to have a score for how they participate by voting. iamnearlythere has just 10 karma but claims to be reading and voting a lot. Karma alone would’ve had be thinking it a was an inactive account. This change would be a counter that just tracks action to know they’re still with us to a degree. Then we can for other reasons make a distinction between passive users of the site & active ones who post stories/comments. Keep it invisible (API-only), in user tree, or in profile. Idea being two scores with one ridiculously large might be confusing vs just a karma one. So, hide it somewhere where it’s there to use but regular karma remains the prominent one.

                                                                2. 5

                                                                  There’s an open feature request on GitHub for something like that fwiw.