1. 4

    Driving up the country to fetch some new (heated!) seats for the Golf. Also picking up a cheap upgraded dashboard whilst I’m there, because why not. Then driving to friends’ boat to help him do some work on it for a couple of days.

    No running this weekend, but the boat is on the hard so you have to climb ~4 meters up/down every time you get on/off it. Should be a fairly active couple of days, possibly without any computing which will be a change of pace.

    1. 7

      god I love not computing

    1. 2

      Interesting, what do you mean by this is a better compromise for scripts? I’m not sure I see where this would be much different in that context.

      1. 2

        I’m working on a deployment tool https://deployer.org/ and for example, if you want to use git and clone repo for the first time (from example from CI) you need to manually login into the server and run the ssh command to github.com to update knopwn_hosts.

        With accept-new this workflow is automated and no manual setup is needed.

        1. 1

          I imagine it’ll be better for scripts that issue multiple SSH commands. You can verify the remote end hasn’t changed host keys between the two (or more) invocations of SSH; whereas with no you just accept whatever the host key is whether it changes or not.

          You can’t tell if the host changes between script runs but you can be sure the host hasn’t changed during the current run.

          1. 4

            I solve this in CI by putting the host’s fingerprint in a variable and writing that to known_hosts. I would think the odds of a host key changing in between commands of a job would be tiny, and the damage could already be done.

            It’s still “trust on first use”, but that first use is when I set up CI and set the variable, not at the start of every job.

            1. 3

              I think this is the correct way to do it, I do this as well for CI jobs SSH-ing to longer-lived systems.

              If the thing I’m SSHing into is ephemeral, I’ll make it upload its ssh host public keys to an object storage bucket when it boots via its cloud-init or “Userdata” script. That way the CI job can simply look up the appropriate host keys in the object storage bucket.

              IMO any sort of system that creates and destroys servers regularly, like virtual machines or VPSes, should make it easy to query or grab the machine’s ssh public keys over something like HTTPS, like my object storage bucket solution.

              I guess this is a sort of pet peeve of mine. I was always bugged by the way that Terraform’s remote-exec provisioner turns off host key checking by default, and doesn’t warn the user about that. I told them this is a security issue and they told me to buzz off. Ugh. I know its a bit pedantic, but I always want to make sure I have the correct host key before I connect!!! Similar to TLS, the entire security model of the connection can fall apart if the host key is not known to be authentic.

            2. 2

              Unless you’re clearing the known_hosts file (and if so, WTF), I don’t see why there would be a difference between consecutive connections within a script and consecutive connections between script runs.

              1. 4

                Jobs/tasks running under CI pipelines often don’t start with a populated known_hosts. Ephemeral containers too. Knowing you’re still talking to the same remote end (or someone with control of the same private key at least) is better than just accepting any remote candidate in that case.

                Less “clearing known_hosts” file, more “starting without a known_hosts” file.

          1. 2

            Work:

            • Deploying a couple of new services and trying slight adjustments to our deploy processes as part of them
            • Starting to document thoughts in my head, been here/using our processes long enough to think I know improvements for us to make. Time to write them down and get feedback on them.

            Home:

            • Started ripping apart a fake wall to see what’s inside it. Medium term goal is to extend it (it’s more of an archway than a wall) down into a proper doorway with a sliding door in it. Need to research sliding doors and how to fit them.
            • Continue getting going with nix (flakes) managing my laptops, and incorporate deploy-rs to start managing servers. Hit the point of configuring stuff that I want all the configuration in one place (git repo) and easily pushable to each node.
            1. 2

              This is a really accessible write up I wish I could have read years ago

              1. 2
                • 5km Parkrun with a friend, because it’s 5 years since we went to a specific one we enjoyed and we should go back
                • Oil changes on Golf and Z4, because I have the parts and the golf is about 3k miles overdue. Need to stop being lazy and jfdi.
                • More experimentation with NixOS to get my shit running on it, in the hope of deprecating a couple of other machines which are long overdue replacement with something more up to date
                • Possibly pick up my Z4 RPi project again
                1. 1

                  My work IT is sort of sucky and disorganized, so sure. Ok you can see I use svn in 2021. Oh well. Also, really? Super secret partnership or merger domain names? I mean I guess that could happen, but it’s not like transparency logs are the only way this would leak.

                  I can’t imagine why a host name being public would be so bad. I still find it annoying having to do dns based discovery queries or just scanning a whole subnet. I helped my mom find a work service because their new dns setup didn’t (at the time) block zone transfers so it seems pretty useful to have dns be for, well looking up domains.

                  1. 3

                    Finding new product names perhaps? https://caiustheory.com/lets-peek/ 😂

                  1. 2

                    Honestly? Staring into the abyss after a long weekend away from work, hoping whatever stares back doesn’t demand too much of my attention span today.

                    I’ve been hoarding my vacating time until I can actually enjoy it, and only taking, like, a Friday off here and there so I don’t go over the vacation limit and lose time off. Which means I worked through December with only one day off outside of the paid holidays and weekends.

                    My reasoning is that, because it’s a pandemic, I can’t really go anywhere or do anything, so I’ll save it for later. But “later” isn’t coming anytime soon, so I’m hotly debating whether or not to say “fuck it” and take a week off in January just to play video games and recharge. But that would disrupt a lot of meetings that are already scheduled🙃

                    1. 2

                      Hitting the point where the meetings aren’t achieving anything because you’re mentally checked out is also disruptive and counterproductive for the business. I did similar in 2020 with not taking holiday, and found that taking the odd day or two every couple of weeks to use up my allowance did wonders for my attitude towards work (and reminding me that there’s things to do outside work.)

                      I’d totally suggest a week off for video games, sounds amazing 🙃

                      1. 2

                        For what it’s worth, the second week of my time off over Christmas/New Year was spent at home by myself, and I felt so recharged from it.

                        I ended up playing video games, hanging out in video calls with my (long-distance) partner, and honestly just getting the kind of rest you only get from pure downtime. By the last day or two, I gravitated to a bit of (non-work) programming, which I think speaks for how refreshing it was!

                      1. 1

                        Loading $work state back into cache from disk.

                        Couple of cars need oil/filter changes (one also needs the rest of a major service now I’ve had a closer look, definitely not been serviced recently before I bought it.) Want to reactivate the 3d printer—which I’m pretty sure was working last time I turned it off—to print a phone holder for the Z4.

                        1. 2

                          Visiting friends, going away with the other half for our tenth anniversary and not doing a lot is my current plan.

                          1. 2

                            Happy anniversary!

                          1. 1

                            Start of a long week off work, so catching up on jobs I’ve been meaning to do but haven’t gotten around to. Things like polishing the {Z4,Golf} headlights to remove the hazing, change the rocker cover gasket on the Z4.

                            Then there’s the tech things I want to do, homebridge running so I can turn the garage light on/off reliably, get a reproducible setup for NixOS on Raspberry Pis and the Microservers. And then exploratory things like putting an RPi in the Z4 and wiring it into the kbus.

                            1. 4

                              Been thinking about standardizing on asdf+direnv. Could anyone offer a quick comparison?

                              It sounds like Nix can also build your containers for you based on your project definition?

                              1. 6

                                asdf works fine for pinning runtimes until you have system libraries, etc that extensions link against which aren’t versioned with asdf. Then you’re back in the same boat as you are with brew, etc where upgrading might break existing workdirs.

                                1. 3

                                  It sounds like Nix can also build your containers for you based on your project definition?

                                  Yep basically just something like this, lots of assumptions given with this and that you “want” to containerize the hello world program gnu ships but eh its an example:

                                  $ cat default.nix
                                  { pkgs? import <nixpkgs> { system = "x86_64-linux"; } }:
                                  pkgs.dockerTools.buildImage {
                                    name = "containername";
                                    config = {
                                      cmd = [ "${pkgs.hello}/bin/hello" ];
                                    };
                                  }
                                  $ nix-build default.nix
                                  # lengthy output omitted
                                  $ docker load < result  
                                  259994eca12e: Loading layer [==================================================>]  34.04MB/34.04MB
                                  Loaded image: containername:zvrzzl5vlbjdbjz8wmy8w4dv905zra1j
                                  $ docker run containername:zvrzzl5vlbjdbjz8wmy8w4dv905zra1j     
                                  Hello, world!
                                  

                                  There are caveats to using the docker builds (can’t build on macos) and you’ll need to learn the nix programming language at some point but its a rather droll affair IME once you get that its all just data and functions. And before you ask why is it so big, the short answer is everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason. But its basically lifted straight out of the nix store verbatim.

                                  1. 1

                                    everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason

                                    I recognise this list. These are the dependencies used in the shell scripts which build the Docker image. They shouldn’t be included in the image itself.

                                    1. 2

                                      They won’t be included in the image if unused.

                                      1. 2

                                        Have I just been building docker images wrong then this whole time?

                                        1. 2

                                          Yup. Nix is a fantastic way to build docker images. For example https://gitlab.com/kevincox/dontsayit-api/-/blob/46cbc50038dfd3d76fee2e458a4503c646b8ff2c/default.nix#L23-35 (nd older project but good example because it has more than just a single binary) creates an image with:

                                          563528481rvhc5kxwipjmg6rqrl95mdx-glibc-2.33-56
                                          7hq7ls1nqdn0ksy059y49vnfn6m9p8hm-dontsayit-api
                                          qabnj48kj88r1zkz17hcfzzw3z8k5rmv-words.csv
                                          qbdsd82q5fyr0v31cvfxda0n0h7jh03g-libunistring-0.9.10
                                          scz4zbxirykss3hh5iahgl39wk9wpaps-libidn2-2.3.2
                                          

                                          Of course if I used musl libc than glibc and its dependencies would go away automatically.

                                          What’s better is that if you use buildLayeredImage each of these is a separate layer so that rebuilding for example the word list, or the binary doesn’t require rebuilding other layers. (This is actually better than docker itself because docker only supports linear layering, so you would have to decide if the word list or the binary is the top layer and rebuilding the lower would force a rebuild of the higher one.)

                                  2. 2

                                    It sounds like Nix can also build your containers for you based on your project definition?

                                    There is also https://nixery.dev/ which allows you to build a container with the necessary tools as easy as just properly naming them. For example:

                                    docker run -ti nixery.dev/shell/git/htop bash
                                    

                                    Will bring you in a container that has a shell, git, and htop.

                                    1. 2

                                      Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure… and sometimes sell it back to you as a service.

                                      YES!! That’s it. Thank you, amazing :) This is an obscure comment but I guess I remembered it because I think it’s very true.

                                      Top level story: The sad state of sysadmin in the age of containers (2015) (vitavonni.de)

                                      FWIW I thought of this because I’m applying to fund a “compiler engineer” position for https://www.oilshell.org. One of the funding agencies has a “User-operated Internet” theme.

                                      So I think it is relevant to point out how the cloud has moved us further and further away from a user-operated Internet, and essential tools like shell have languished. Whenever I try to host my own Internet services, I invariably run into a mess of shell scripts (at both build time and runtime).

                                      https://news.ycombinator.com/item?id=29605304

                                      https://nlnet.nl/useroperated/

                                      1. 1

                                        I don’t disagree with it either, also comes to mind occasionally when I’m lamenting about the state of the world in the pub. 😬

                                    1. 1

                                      Got a garage to clear out so I can fit a sofa in for storage next week, which will likely involve a run to the tip. Parts for the Z4 have arrived as well (including a repaired GM5 module to make the central locking functional 🤞🏻) so that needs undressing and bits throwing on it. Probably give it a run out somewhere as well, maybe across to the sailing club to look at the boats.

                                      1. 1

                                        Flying home from Lanzarote, going into hiding until my arrival test results come back then heading to Wales to help deliver my friend’s boat to the port it’s wintering in. Going from 24°C and sunshine to 4°C and wet in 36 hours is going to be quite the shock.

                                        Rest of week is getting back into the swing of things at home, everyone out of isolation and back to normal. Still have things in the homelab to tidy up and experiment with, my long running battle with the “media server” apps getting wedged occasionally and having to respond to a family P1 (which are really P3s, admittedly) is ongoing currently.

                                        1. 1

                                          Heading on my first plane out of Manchester since nov 2019 to Lanzarote for the weekend. Technically have been before but was too young to remember, looking forward to exploring. Also the final F1 race of the year, which could be amazing or a farce, too difficult to call.

                                          1. 6

                                            There’s one place where a few years ago ISOs were required and they may still be. HP’s iLO could only use ISOs for remote installation from virtual media. I haven’t used it in a while, so maybe they improved it. If not, it’s either the virtual CD or PXE boot if you want to install a server.

                                            1. 3

                                              I think my servers have a previous major version (or two?) of iLO, but that only takes ISO images yes. I usually boot https://netboot.xyz then install from their livecd choices rather than load the ISOs directly. I guess I wouldn’t be able to boot puppy via netboot using the usb image though.

                                              1. 1

                                                This has also been the case for most BMCs I know of that support virtual media. And while in most cases I prefer to use PXE to image machines, I do occasionally need to do a “completely fresh” install of Linux on some machine in a datacenter…

                                              1. 1
                                                • Attempting to stick with Advent of code longer than I have in previous years. (I think I’ve managed to day 5 before, currently solving day 3 this year)
                                                • Trying to get the Z4 through it’s MOT. Fixed the airbag light at the weekend, chucking it in with a busted door lock in the hope it passes. (It’s only bust from the outside, so there’s less of a safety concern in a crash for folks getting out.)
                                                • Attempting to run the household solo whilst working full time as half the house is in self-isolation. (Basically it’s impossible to do everything successfully, thankfully work are super understanding which is a great help.)
                                                • Flying to Lanzarote on Friday for a long weekend away; last of the winter sun, cannot wait.
                                                1. 5

                                                  Why do you say Nomad is not free?

                                                  1. 4

                                                    That paragraph seems like the author is justifying to themselves why they want to play with k8s. I think they’re super weak reasons, could delete the entire paragraph and improve the post.

                                                    eg, not sure I’d claim Mesos hasn’t gained critical mass given it underpins Netflix’s entire stack (Titus depends on Mesos & Zookeeper.) Nomad has equivalent large deployments, Cloudflare, Roblox and as you point out, is free (with paid-for extra features.)

                                                    1. 4

                                                      Author here, thanks for the feedback. I meant Nomad is not free because some features are behind an “Enterprise” paywall.

                                                      1. 1

                                                        Isn’t that also technically true of k8s? In the sense that cloud providers (google, amazon) have a special sauce that they don’t share with mere mortals?

                                                        1. 3

                                                          You don’t bump into “Error - Enterprise only feature” messages when working with K8s. I’m sure Amazon and Google have their own tools for working with it, but their use case is very specific.

                                                          1. 1

                                                            Thankfully no, that is not the case.

                                                        2. 2

                                                          Mesos was a contender for a hot minute but it’s definitely donezo at this point, isn’t it?

                                                          1. 2
                                                            1. 2

                                                              Goodnight, sweet prince.

                                                              edit: I actually had no idea it had gone to Apache, hah.

                                                              1. 2

                                                                Apparently it was originally an Apache jam? Wow. I know nothing.

                                                      1. 5

                                                        Putting some time into the Z4 to get it back on the road (MOT expired in October). Airbag light is permanently on, but there’s broken wires under the passenger seat which are likely related so trying to figure out where they connect is the first job. Also need to rip the GM5 module out from under the dash and send it off to have relays replaced, to make the passenger door unlock with central locking again. Quicker I start on the jobs, quicker it’ll be done.

                                                        Also running 5k organised by the local brewery, because there’s a free pint at the finish line with my name on it. Haven’t run since I did a 10k in September, so that’s going to be slow I suspect.

                                                        1. 8

                                                          This is a really nicely laid out blog post, excellent use of “show more” in footnotes for “totally useful, but not relevant to the main post” information, as well as explaining why each of the settings are there with visual aids.