Threads for rod

  1. 11

    The section on granularity alludes to another principle: “reduce the scope and size of your state”. That is, push state to the leaves of your system, so your higher level processes have less opportunity to fall into a broken state.

    Containerisation is a form of the this.

    https://grahamc.com/blog/erase-your-darlings is a realisation of this for a Linux distribution: each boot is from the exact same state, with optional allowlisted state to persist between reboots.

    1. 1

      The whole arm board/SoC thing is a mess now. I guess we need a powerhouse of the IBM in the early PC era.

      1. 5

        https://tow-boot.org/ does that. The recommended installation approach is to install that to your board’s SPI, which can then boot generic arm64 images.

        It’s used by ~most Linux mobile distributions, and allows the distributions to publish a single arm64 image.

        1. 2

          Oh tow-boot looks really good thank you for the link!

          1. 2

            It doesn’t support Rock64 yet, but the developer recently suggested it wouldn’t be much work, and they’d give it go, if others could help verify it: https://github.com/Tow-Boot/Tow-Boot/issues/151

            Consider offering to verify, if you’re keen!

      1. 4

        As an alternative to needing containers for sites, folks might want to consider using about:config’s privacy.firstparty.isolate which runs each ~site in its own isolated container.

        This provides the same isolation benefits as FF containers, but saves you having to create/manage/open containers. As an example, if you log into facebook.com, then visit messenger.com, messenger.com won’t be aware of your facebook.com session.

        See also: privacy.resistFingerprinting.

        1. 1

          Thanks - privacy.firstpary.isolate looks interesting - I’ll try that out, I’m actually already using privacy.resistFingerprinting, I’ve added that to the post.

        1. 12

          As mentioned, git ls-files is a good solution to this.

          For a VC-agnostic tool, consider ripgrep: rg --files.

          1. 3

            The detail on this is decent, and the lessons look useful.

            Though this only impacted a small number of customers, total unavailability for 14 days is a big deal.

            This sounds like another example of “Your nines are not my nines”: https://rachelbythebay.com/w/2019/07/15/giant/

            1. 3

              It’s great to see more companies contributing to improving free Bluetooth stacks. Bravo Valve and Collabora!

              1. 1

                Guix does this well too. The info manual is detailed/searchable, and available offline during install and when installed.

                1. 1

                  It feels to me that the major feature is rollbacks. Now while I can understand that, after all it’s why people like having backups and ZFS snapshots (and but environments by extension) I just don’t really see how this really matters being integrated into the package manager really matters. If you have software installing it uninstalling software seems to have worked fine over the past few decades.

                  Is state/data/configuration somehow managed in a special way?

                  When I think of such scenarios the burden is on getting stuff that is not managed by package managers back into order.

                  On the topic of running multiple versions. While I very rarely have wanted to do that, mostly for Debugging or bad upgrade paths of software while not having backups from the article I understand that certain services are being run multiple times with different versions. If that is correct I’m very curious how that is done in relation to sockets (Unix, TCP, etc ) how does other software decide where to connect to? Just the address or is there something else in how packages are built handled that helps with deciding?

                  1. 7

                    It feels to me that the major feature is rollbacks.

                    I’m a keen NixOS and Guix user, and I don’t consider this to be directly important, though I can see why it’s seen that way.

                    On the topic of running multiple versions.

                    I also don’t run multiple versions of things.

                    The biggest benefit that NixOS/Guix introduces for me is that I can treat my machine as code, with high fidelity and efficiently.

                    In Linux distributions such as Debian/Arch/CentOS, I consider my machine to be a 30GB mutable blob (the contents of /etc, /usr, etc.). Updates are mutations to this 30GB blob, and I have low confidence how it’s going to behave. Cfgmgmt /automates/ this, automating low-confidence steps still results in low confidence.

                    For NixOS/Guix, I consider my machine to be about 100kB (my Guix config). I can understand this 100kB, and when I change the system, I change this 100kB, and accurately know what state my machine will in after its changed: the update is actually a replace.

                    Is state/data/configuration somehow managed in a special way?

                    ~Everything in /etc is managed via NixOS/Guix.

                    ~nothing in /var/ is managed via NixOS/Guix. NixOS/Guix reduces my state space, but /var is still state I have to care about. (Actually, on my desktop no state is preserved on /var between boots: every boot has a fresh computer smell.)

                    1. 1

                      Hey, thanks a lot for your response. I have some naive questions then.

                      In your desktop system, in many cases that blob, the state one cares about would probably be in $HOME. What about that?

                      What do you mean sheet “/etc is managed”? Say I have a configuration that would usually lie there. Where is it now? Say I want to customize it what would I do? Say configuration syntax changes, what would I do?

                      I understand your comparison with mutable blob vs declared state, after all that’s the same approach that other kinds of software often use, be it configuration management, some cloud/service orchestration tools and honestly a lot of software that has the word declarative in the first few sentences.

                      In practical use I see these systems fall apart very quickly, because a lot of the time it’s more a changing state in the way one would define a sweet of database migrations.

                      So for a simple example. Let’s take /etc. That’s the configuration. You in many situations can copy that to a new system and it’s fresh and the way you want. Various package managers also can output a list of which packages are installed in a format that can be read so you usually have /usr covered as well. Because of that I don’t usually see this part as a big issue. After all that’s in a way how many distro installers look at things. /boot is similar. /usr should not be touched, though sometimes it can be an emergency hack, but I prefer to have it read only other than on changes by the package manager.

                      That leaves /var and /home, which sounds at least somewhat similar to what you are saying (correct me if I’m wrong. So in my understanding what is done is more that the system makes sure that what should be actually is? Talking about upgrades, removals, etc. not leaving stuff behind? I guess that makes quick hacks hard or impossible? Don’t get me wrong I’d actually consider that a good thing.

                      /var on desktop might not have much needed state, but in many situations that state would be in /home.

                      Anyways, thank you again for your response. I guess at this point it might make sense if I took a closer look at it myself. I just am curious about practical experiences, because I completely understand that declaratively describing a system tends to look very nice on paper, but in many situations (also because of badly designed software) is more like simply writing a setup shell script and maybe running it each boot, just that shell scripts tend to be more flexible for better and for worse.

                      Of course having a solution that does that fit you with a good abstraction is interesting.

                      That’s why lately I’ve been thinking about where we handle big blobs that we sometimes want to modify in a predictive manner and had to think about database schemas and migrations.

                      Thanks again, have a nice day. :)

                      1. 4

                        What do you mean sheet “/etc is managed”? Say I have a configuration that would usually lie there. Where is it now? Say I want to customize it what would I do? Say configuration syntax changes, what would I do?

                        The contents live as part of your nix configuration. It’s both awesome and frustrating at times. Nix tries to overlay it’s view of reality onto the config file format, so say it’s nginx and instead of trying to write nginx confg like:

                        http {
                                sendfile on;
                        }
                        

                        in nix you would write something like:

                        services.nginx.http.sendfile = true;
                        
                        

                        and then when nix goes to build the system, it will generate a nginx config file for you. This allows for some nice things, like services.nginx.recommendedOptimisation = true; and it will fill in a lot of boilerplate for you.

                        Not all of nix is this integrated with the config file(s), so sometimes you get some oddities, or sometimes the magic that nix adds isn’t very clear and you have to go dig around to see what it’s actually doing.

                        Another downside, is it means a re-build every time you want to change a minor thing in 1 application. The upside, nix is usually really good about not restarting the entire world and will try to just restart that 1 application that changed. This is just an off-shoot of the declarative process, and some will call it a feature, especially in production, but it can be annoying when in development.

                        You can turn all of that off and just say this app will read from /var/etc/nginx/nginx.conf and leave nginx.conf entirely in your control. This is handy when moving to nix, or maybe in development of a new service or something.

                        As far as the mutable state of applications, nix mostly punts on this, and makes it YOUR problem. There is some nix config options that packagers of apps can take advantage of, so say on upgrades, if you installed PG11 originally, it won’t willy nilly upgrade you to PG12. It makes you do that yourself. So you get all the new bits except PG11 will still run.

                        All that said this stuff isn’t perfect, so testing is your friend.

                        1. 1

                          You can turn all of that off and just say this app will read from /var/etc/nginx/nginx.conf and leave nginx.conf entirely in your control.

                          My goodness, do you have a guide or blogpost or something for this way of going about it? That’d be super helpful. I’ve tried Nix a few times and this is exactly where I go crazy. I can store real config files in git too; just let me do that! (and avoid the Nix language!)

                          1. 2

                            https://search.nixos.org/options?channel=21.05&from=0&size=50&sort=relevance&type=packages&query=services.nginx is where I’d start to look for how to disable the config management part of the nginx service. If that wasn’t enough, I’d go to the corresponding file in nixpkgs.

                            If a module doesn’t meet my needs, and I can’t easily make it do so, sometimes I will write my own module, to have fully control over it. I still reuse the nginx package, and would typically start my module by copy-pasting and trimming the existing one.

                            95% of the time, the provided modules do exactly what I want.

                            https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md aims to make the “please let me take control over the service” usecase easier.

                            1. 1

                              Well, doing this in some ways defeats one of the big reasons for Nix, but there are valid use-cases:

                              For nginx, this is what I do:

                              # this sets the config file to /etc/nginx/nginx.conf, perfect for us using consul-template.
                                services.nginx.enableReload = true;
                                services.nginx.config = "#this should be replaced.";
                              

                              Now it’s on you to maintain /etc/nginx/nginx.conf

                              For us, we use consul-template(ran via systemd as more nix configuration) and it generates the config. But you are free to replace it (after deploy) manually. i.e. nix will over-write the /etc/nginx/nginx.conf file every nixos-rebuild build with the contents: #this should be replaced.

                              Otherwise, what nixos tends to do is symlink the /etc//configfile -> to somewhere in /nix which is read-only for you. so it’s up to you to erase the symlink and put a real file there. One could automate this with a systemd service that runs on startup.

                              Another way to do this, is hack up the systemd service, assuming the service will accept the location of the config file as a cmd line argument. This is non-standard and can be fiddly in nix.

                              I don’t know of a better way.

                          2. 2

                            On Guix system it is recommended to manage anything in etc via a “service” which will be written/configuerd in scheme and has deploy and rollback semantics.

                            For config that lives in $HOME there is guix home and guix home services, which parallel the system for /etc and system services, and even work on other operating systems.

                            1. 2

                              In your desktop system, in many cases that blob, the state one cares about would probably be in $HOME. What about that?

                              I use https://github.com/nix-community/home-manager to manage $HOME. I use that to manage my git config and bashrc. I also use it to declare which directories should survive a reboot. e.g. I persist ~/.steam and ~/.thunderbird, “Documents/”, a few others. But everything else, e.g. ~/.vim (which I only use in an ad-hoc manner) is wiped.

                              Even that leaves some blob-like state: I persist “.config/dconf”. Ideally that could be managed declaratively, but I haven’t seen a workable solution.

                              Let’s take /etc. That’s the configuration. You in many situations can copy that to a new system and it’s fresh and the way you want. Various package managers also can output a list of which packages are installed in a format that can be read so you usually have /usr covered as well.

                              That works fine for building new machines well, but a typical Linux machine is built far less frequently than its updated. For example, I’ve managed machines with packages/Puppet/Ansible in the past, and occasionally run into situations where the machine state according to packages/Puppet/Ansible no longer matches the actual machine state:

                              • postinst scripts that worked well during install, but get updated such that upgrades work and installs are broken.
                              • cases where apt-get install $x followed by apt-get purge $x leaves live config (e.g. files in /etc/pam.d)
                              • cases where the underlying packages are changed in ways incompatible with the Puppet config: after all, the underlying packages typically don’t attempt to QA against Puppet config.

                              The result is that even just covering /etc and /usr, machines are brittle, and occasionally need to be rebuilt to have confidence.

                              Talking about upgrades, removals, etc. not leaving stuff behind? I guess that makes quick hacks hard or impossible? Don’t get me wrong I’d actually consider that a good thing.

                              Yes, it does make quick hacks hard/impossible. It is possible to do some quick hacks on the box (systemctl stop foo, for example), and nixpkgs is designed so that various parts can be overridden if needed.

                              When we climb the ladder of abstraction, and lose access to easily change the inner workings of lower levels, it looks like (and is!) restrictive. In the same way I wouldn’t modify a binary in a hex editor to perform deployments, nor would I make live changes to a Docker image, I aim to not SSH to a machine to mutate it either. I prefer my interactions with lower-level abstractions to be mediated via tooling that applies checks-and-balances.

                              Anyways, thank you again for your response. I guess at this point it might make sense if I took a closer look at it myself.

                              I don’t make recommendations without understanding requirements, but NixOS/Guix is at least a novel approach to distributions, which might be interesting to OS folks.

                              NixOS/Guix might have come too late for industry: containers also aim to manage system complexity, and do a good job of it. I think NixOS/Guix offers good solutions for low-medium scale, and as a way to build container images.

                              I just am curious about practical experiences, because I completely understand that declaratively describing a system tends to look very nice on paper, but in many situations (also because of badly designed software) is more like simply writing a setup shell script and maybe running it each boot, just that shell scripts tend to be more flexible for better and for worse.

                              I only use NixOS/Guix for my personal infra, and manage all those machines in a declarative manner (other than out-of-scope things such as databases like ~/.config/dconf and postgres).

                              That’s why lately I’ve been thinking about where we handle big blobs that we sometimes want to modify in a predictive manner and had to think about database schemas and migrations.

                              Yes, DB schema migrations is an interesting case where a declarative approach would be nice to have: it’s much easier to reason about a single SQL DDL than a sequence of updates.

                              A similar problem I have is the desire for declarative disk partitions: ideally I could declare my partition scheme, and apply a diff-patch of mutations to make the declaration reality. It would only proceed if it was safe and preserved the underlying files. It’d likely only be possible under particular constraints (lvm/btrfs/zfs ?). Even then that’s hard to get right!

                              Thanks again, have a nice day. :)

                              You too!

                        1. 19

                          This also got posted to HN, there’s a noteworthy discussion on some security concerns wrt. Linux-libre:

                          Guix System is an advanced distribution of the GNU operating system. It uses the Linux-libre kernel

                          It’s worth pointing out that the linux-libre kernel is developed under the FSF doctrine that “binary blobs are bad unless you can’t see them”. This has been taken to its logical extreme here, where this Linux fork actively removes security warnings informing users that they need to update their CPU microcode, because microcode in ROM is fine but dynamically loaded microcode updates are not, in this school of thought.

                          https://lists.gnu.org/archive/html/info-gnu/2018-04/msg00002

                          I have no interest in software that uses arbitrary religious dogma (that doesn’t help users’ freedom, as those users’ CPUs have proprietary microcode whether they know about it or not) to justify censoring critical security vulnerability notifications for users. I regard this as actively evil anti-user behavior.

                          1. 6

                            I’m a Guix user, and acknowledge that linux-libre’s microcode security/freedom decision is not what I’d pick. Many Guix users choose to use the vanilla kernel from https://gitlab.com/nonguix/nonguix#CPU%20Microcode for similar reasons.

                            However, it’s not surprising that software freedom groups value freedom and will sacrifice security for it. That’s the nature of prioritizing values: some other important values will be second place.

                            Values aren’t prioritized for no reason, however. Picking freedom over security in this way might have some positive effects (as well as the obvious negative effects):

                            • microcode updates could downgrade the experience in ways the user doesn’t agree to.
                            • it might encourage Intel to open their microcode specs/updates. This sounds far fetched, but we’ve seen large openness wins in the WiFi and GPU space over the past few years.
                            • it might encourage users to purchase hardware that has open microcode updates.

                            FWIW, I prefer free software, but am willing to sacrifice it for security at times.

                            1. 5

                              That’s a weird position for the guix though because you still run the microcode, you still get partial updates for it (bios), and they spent time to hide warnings. Trying to hide the fact you have the freedom to get the updates, because their idea of freedom doesn’t like the updates is serious mental gymnastics.

                              1. 3

                                I agree. I see this as a too large a security compromise for such a small win of “Freedom”.

                                That said, drawing these lines makes it really clear what FSF/free software movement’s priorities are, which is important for public sentiment/recognition/visibility.

                                When I think of FSF/free software movement I think “their goals align with mine, but I don’t always agree with their methods”. That’s a decent position for an advocacy group/movement.

                                1. 1

                                  It doesn’t really matter the minutiae of the decision. Once these people have decided they know better than you what you should be allowed to know because they want to guide your behaviour for their religious crusade, they are not to be trusted with your OS.

                                2. 1

                                  we’ve seen large openness wins in the WiFi and GPU space over the past few years

                                  We have? I thought free+open WiFi ended with the 802.11n Atheros chipset, and Nvidia is as closed as ever.

                                  1. 3

                                    NVIDIA is closed, but both AMD and Intel are doing pretty well with upstream Linux drivers as far as I know.

                              1. 3

                                Yes, I’m glad to see XMPP being revitalized!

                                I have more confidence in the longevity in messaging approaches that have ecosystems with multiple participants, rather than messaging approaches that rely on BDFL.

                                I use XMPP clients Quicksy on Android and Dino on Linux to chat to my partner and some friends.

                                1. 4

                                  +1 to all this.

                                  I’m migrating from NixOS to Guix System at the moment.

                                  NixOS is great, but after 4 years, I’ve had difficulties understanding some of Nix’s foundations:

                                  • I don’t write intermediate Nix often enough for it to be a smooth process.
                                  • I have difficulty remembering/tracking use of nixpkgs’s lib/ and nixos/lib directories.
                                  • I cannot easily determine what the Nix tooling does. I’m not strong in C++.

                                  Guix fixes those foundational issues:

                                  • Guile was easy for me to pickup.
                                  • Guile has language-level modules/imports, and good libraries/editor support, making it easier to understand the relevant libraries.
                                  • Guix’s tooling is written in Guile

                                  Guix has issues, but they are more easily fixable:

                                  • Fewer packages than NixOS. Guix has most of the packages I care about. I can contribute the rest.
                                  • Strict focus on free software. I support free software, but would like support for non-free software at times too. I’m happy to use nonguix. This approach worked well enough for me when I used Debian nonfree in the past.

                                  NixOS and Guix System are both great choices, IMO.

                                  1. 3

                                    One of the things that trip me when I first tried emacs is what I wanted it to do too much without having any clue what was happening. I wanted it to replace by go IDE (GoLand), my JS and Rust tools (VS Code) and work perfectly with the language I was just learning (Elixir).

                                    After a bunch of copy/paste and like 500 lines in init.el that I didn’t know what they where doing I obviously felt flat on my face.

                                    Nowadays I’ve come back to it with more curiosity and less expectations. Got it working perfectly for beancount, got rid of the awkward (for me) hotkeys with evil, got the fonts and colors that I like, and now I’m free to explore and grow from here :-)

                                    The rest of the tools are fine and I’m sure that emacs could replace them, but Idk how nor I need to know right now… I can just enjoy the experience!

                                    1. 4

                                      Glad to hear it’s now working for you.

                                      For those wanting to start, I suggest either:

                                      The only config I’d recommend from the beginning is (fido-mode), https://www.gnu.org/software/emacs/manual/html_node/emacs/Icomplete.html .

                                      1. 2

                                        If you are looking for something between Doom and vanilla, the author’s prelude starter kit is great. And if you use org-mode, the sci-max kit has some really helpful utilities.

                                        I personally can’t seem to feel at home adding on to any of the starter kits, but I’ve pulled in some things from both of these to my hot garbage of an ancient config mess. :)

                                        1. 2

                                          I think doom is an incredible mix of out of box power, emacs customizability, and evil mode. Even as a power user, distributions ensure I don’t miss out on new features, as my dot files over time are likely to lag behind without a lot of attention.

                                          1. 3

                                            Also recommending Doom Emacs. After 20 (25ish?) years of Emacs I switched to Doom Emacs so I could throw away a large part of my homegrown config and not having to maintain it anymore.

                                            For me it does the same as the i3 window manager and the Fish shell: it has sensible defaults that I am mostly willing to accept to save time on config maintenance. (Ofcourse I still have my own customizations to these tools.)

                                            1. 1

                                              Same, as a long-time Emacs greybeard. I switched to Doom and got rid of a lot of garbage that I had accumulated over the years.

                                      1. 3

                                        Very nice! I’m a NixOS and Guix user/contributor, but recognize they both have approachability/learning curve issues.

                                        I think Skylark is a great choice: its similarity to Python will make it much more approachable.

                                        Would using Bazel directly have been possible? That is, have a Bazel rule that returns a derivation. This would have benefitted from the existing Bazel features:

                                        • remote caching
                                        • external dependencies
                                        • sandboxing
                                        • Bazel’s build rules, e.g. https://github.com/bazelbuild/rules_go . I hear Nix plans to support source-file level building, and this would allow you to jump to that in one go.

                                        Perhaps there’s some downsides I’m missing?

                                        1. 1

                                          Interesting, possibly. I did want this use case “bramble run github.com/maxmcd/busybox ash”. Running arbitrary programs from remote sources. Not sure if I could have gotten that with Bazel, or how I would have made it work.

                                          I will say that to my understanding (which could be wrong) in Bazel:

                                          • Builders can touch anything in the operating system.
                                          • Dynamic dependencies are not supported (more here and elsewhere).

                                          I wanted Bramble to be able to support custom builders without having to trust them to not mess with things on the host machine. I don’t know a ton about Bazel custom builders, but I think they require some level of trust. When similar language-specific builders are added to Bramble they will be able to define custom build logic without having access to the host machine. This was part of the goal, lower barrier to entry to contribute custom builders, both in terms of complexity and trust. Would be interested in your thoughts on this one, might have it wrong.

                                          I’d also really like to support dynamic dependencies with Bramble, some references to that here. This work is ongoing, but I think there’s a path.

                                          On a more honest note, I wanted to build a thing, and didn’t spend as much time auditing nix/guix/bazel, so could have missed a more harmonious opportunity. I do also wonder if building a Starlark frontend for Nix was something else I should have explored.

                                          1. 2

                                            I will say that to my understanding (which could be wrong) in Bazel:

                                            • Builders can touch anything in the operating system.
                                            • Dynamic dependencies are not supported (more here and elsewhere).

                                            https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf compares build systems and finds (see Table 1) that Bazel internally supports dynamic dependencies:

                                            ``As of writing, it is not possible to express dynamic dependencies in user-defined build rules; however some of the pre-defined build rules require dynamic dependencies and the internal build engine can cope with them''
                                            

                                            Though as you point out, the Bazel developers don’t think that’s possible: https://github.com/bazelbuild/bazel/issues/2362#issuecomment-273508287.

                                            I wanted Bramble to be able to support custom builders without having to trust them to not mess with things on the host machine. I don’t know a ton about Bazel custom builders, but I think they require some level of trust. When similar language-specific builders are added to Bramble they will be able to define custom build logic without having access to the host machine. This was part of the goal, lower barrier to entry to contribute custom builders, both in terms of complexity and trust. Would be interested in your thoughts on this one, might have it wrong.

                                            I don’t know much about Bazel’s sandboxing internals.

                                            On a more honest note, I wanted to build a thing, and didn’t spend as much time auditing nix/guix/bazel, so could have missed a more harmonious opportunity. I do also wonder if building a Starlark frontend for Nix was something else I should have explored.

                                            That’s fair enough. It’s great that you’ve created this and shared it. The ecosystem benefits from different approaches.