1. 4

    I’ve been quietly working on some stuff combining Nix and Kubernetes over the last year or so. In fact I recently - just in the last week - had a rather elegant idea (I’m biased, of course) which I’ve implemented and plan to write about.

    Good to see others considering potential crossovers!

    1. 1

      Awesome :) I look forward to seeing the post!

    1. 6

      This is a really neat write-up!

      I’ll admit I’ve been rather avoiding Kubernetes and am just barely beginning to get cozy with things like docker-compose and the like, and this article is making me think I should reconsider that choice!

      1. 6

        I recommend looking into hashicorp’s nomad

        1. 1

          I adore Hashicorp software, but it would depend upon the goal of working with k8s, wouldn’t it?

          If the goal is to deploy a technology as a learning experience because it’s becoming an industry standard, as awesome as I’m sure nomad is, it’s not going to fit the bill I’d think.

          I’m still blown away all these years later by Terraform and Consul :) Those tools are just amazing. True infrastructure idempotence, the goal that so many systems have just given up on entirely.

          1. 4

            To be clear: if your goal is to learn k8s–which is fine; it’s a very marketable skill right now, and I’m 100% empathetic with wanting to learn it for that reason–then I think it makes sense. But for personal use, Nomad’s dramatically simpler architecture and clean integration with other HashiCorp projects is really hard to beat. I honestly even use it as a single-node instance on most of my boxes simply because it gives me a cross-platform cron/service worker that works identically on macOS, Windows, and Linux, so I don’t need to keep track of systemd v. launchd v. Services Manager.

        2. 4

          Don’t, just don’t… I am trying to avoid k8s in in homelab to reduce the overhead. Since I don’t have a cluster or any feature in k8s that’s missing in a simple docker (-compose) setup

          1. 5

            It depends on what you call your “lab”. A couple of years ago I realized that there’s only one way I master things: practice. If I don’t run something, I forget 90% about it in 6 months.

            My take on the homelab is to use as much overhead as possible. I run a bunch of static sites, an S3-like server, dynamic DNS and not much else, yet I use more stuff/overhead to run it than obviously necessary.

            The thing is, I’ve reached a point where more often than not, I’m using the knowledge from the lab at $WORK, even recycling some stuff such as Ansible roles or Kubernetes manifests.

            1. 6

              I believe this to be the differentiation between a homelab and “selfhosted services”. The purpose of a homelab is to learn how to do things. The purpose of selfhosted services is to host useful services outside of learning time. That is not to say that the two cannot intersect, but a homelab, in my opinion, is primarily for learning and breaking things when it doesn’t affect anything.

              1. 2

                Yup I think this is the key.

                I’m already using docker-compose for my actual self hosted services because it’s simple and easy for me to reason about, back up the configuration of, etc etc.

              2. 3

                Agreed, it certainly comes with a rather large overhead. I use Kubernetes at work and rather enjoy it. So, it’s great having a lab environment to try things out in and learn, so that’s why I bother hosting my own cluster.

              3. 3

                I started with docker-compose as I began to learn containerized tech, but transitioned to Kubernetes because the company wanted to use it for prod infrastructure. I actually found that K8s is more consistent and easier to reason about. There are a lot of concepts to learn, but they hang together.

                Except PersistentVolumeClaims.

                1. 2

                  Thank you for reading. I’m glad you enjoyed it :)

                  I’ll say, picking up Kubernetes at home is a good choice if it’s something you want to learn. It’s really useful to have a lab environment to try things out and build your knowledge with projects.

                1. 3

                  Okay, and here is what I don’t get about the ongoing nix-declarative enthusiasm:

                  How does this deal with your passwords, history, bookmarks, etc? And if it doesn’t what’s the big deal? What do I get out of this that wouldn’t be better with Firefox sync?

                  1. 4

                    I personally use both. I use Firefox Sync to handle syncing of data I’d consider owned by the browser (history, open tabs, bookmarks). I don’t, however, consider it my browser’s job to own & manage my passwords. I keep those elsewhere and manage them separately.

                    I wouldn’t say this is “a big deal”, nor a particularly strong purveyor of “the ongoing nix-declarative enthusiasm”. This is just a write up of a few behaviours I wanted from Firefox that could be expressed in a stable, repeatable, and fairly elegant manner using Nix. Firefox Sync will not give you the profile management approach I wrote about, nor will it sync your userChrome.css, or even all of your about:config settings.

                    1. 1

                      Two follow-up questions, if I may:

                      1. Can this be extended to set up the sync for the two profiles it is adding?
                      2. Does setting up extensions with Nix not collide with the extensions from the syncing?
                      1. 3


                        1. Potentially. Anything you can set in about:config (userprefs.js) can be expressed here. So if there’s a setting for that, then yes it could also do that
                        2. I don’t sync my extensions via Firefox Sync, so I’m not entirely sure
                  1. 3

                    I’m not sure if Nix is necessarily the best option as most of the time it’s not the language itself that is the point of the posts, rather the transactional package manager it makes possible (now that I’ve written that it sounds like a description of most other programming languages, but I maintain that the Nix situation feels different).

                    If a tag is created, perhaps something in that vein would be more useful? For example, Guix posts also occasionally crop up and are very likely to be of interest and not to the same people as those who want to see Nix posts. I also recall seeing another similar package manager mentioned on here recently, which would fit in the same category.

                    1. 1

                      You raise a very good point. I just wonder what short tag could succinctly express all that these technologies give us.

                    1. 3

                      I’d like the tag to be more general than nix, but I’m not sure how to boil-down functional declarative package management into a short label.

                      1. 1

                        Yeah, it’s not the easiest thing to put so succintly

                      1. 3

                        I was was first skeptical, but given that there are already bunch of non-Linux operating systems (except android) and programming languages as tags available and the narrow usage of nix specific content I’d support a nix or nixos tag.

                        Otherwise I’d either suggest to use the already existing category devops, which could be argued is applied package management, system automation, configuration management/orchestration (and more) in one tag or introduce something that covers system automation.

                        Edit: If you search for puppet|ansible|saltstack|docker|kubernetes you find a lot of content that is tagged under devops and linux.

                        1. 2

                          I certainly agree with the use of the devops tag for these sorts of posts.
                          I suppose I felt a nix tag would fit nicely with the other language tags.

                        1. 10

                          I’m in favor of a nix tag because it’s not just a language or a package manager or a distro, it’s all those things and more

                          1. 2

                            This is how I feel about it, but there are also some good points being raised by others around generalising the potential tag

                          1. 2

                            I’m a little late to the party, and there’s an overwhelming response to this post (which is great to see!), but I figure I’ll share my thoughts - keeping it short and sweet.

                            Personally, I’ve built my config up slowly over the years with quite a lot of additional functionality.
                            The way I keep this sane:

                            1. Use use-package - as many others have suggested here
                            2. Make your configuration literate. A common approach, and the one that I use, is to write it in org mode and tangle it when starting up (see my init.el and readme.org below)
                            3. This is perhaps a point that others might not have made, and one that I feel is vital: in your literate configuration, above the package expression, explain what the thing is, why you have it, and elaborate on any additional config you’ve got set there. Not only does this help other people stumbling across your configuration understand what they’re looking it, it also helps you reason about the worth of that config.

                            Here’s my configuration: https://github.com/cmacrae/.emacs.d

                            1. 11

                              Battlestation (with screenshot):



                              • Kinesis Advantage keyboard
                              • Lenovo X1 Extreme 2nd gen
                              1. 4

                                Wow! What a beautiful environment to hack in <3

                                1. 2


                                  You’re welcome to co-work anytime: https://zen-temple.net/lambda-zen-temple/introduction/

                                2. 2

                                  And you even have a real shack that you could paint or go to to run away from the internet :)

                                  1. 1

                                    There was quite some painting going on at the house - all the outside surfaces have been redone for example^^

                                    The shak is for storing 6.5 qm^3 of local wood for the fireplaces, though:

                                    In summer, when it’s empty and raining outside, I do tend to sit inside it from time to time enjoy life with a cup of coffee and a good book^^

                                  2. 2

                                    No MacBook anymore? ;D

                                    1. 1

                                      Unfortunately, the last subjectively ‘good’ MBP was the 2015 model which broke down on me hard close to two years ago.

                                      Objectively speaking - or at least to the best of my knowledge, the new MBPs are not capable of running Linux due to loads of proprietary hardware.

                                      If only Apple still sold the 2015 model with a RAM, SSD and CPU upgrade, I’d still have a MBP and would have saved weeks of research and tinkering on slightly less differently proprietary hardware(;

                                  1. 2



                                    1. 2

                                      Awesome! What led you to build your own Neovim client?

                                      1. 3

                                        I wanted something that started up faster, had a no nonsense UI and ligature support. The alternatives didn’t meet my needs so I started to build one and have been iterating on it since

                                      2. 1

                                        Yay \o/ A fellow Model 01 user!
                                        I have mine at work… but I’m awfully tempted to get another for home.

                                        1. 1

                                          doooo it. Its an amazing board. And having a symmetrical setup at home and work means customization carries over :)

                                        2. 1

                                          That’s the first post that makes the word “battlestation” sound “too modest” :P

                                        1. 19
                                          jtm@x1c ~ % ls
                                          TODO acme aux bin doc down git go mail mnt music rom www xen 
                                          • acme - additional scripts for acme.
                                          • aux - synonymous with “junk”.
                                          • doc - images, documents, slides, etc.
                                          • down - downloads
                                          • mail - emails in Maildir format.
                                          • www - source code for a static site.
                                          • xen - vm configurations.

                                          I’ve been meaning to add a cron job to delete anything in down that’s older than three days.. that folder tends to blow up.

                                          To keep junk from accumulating in $HOME, I use a shell function called “t” that will automatically cd to a temporary scratch space, so I don’t have to think about cleaning up junk that I don’t intend to keep. I use this almost every day.

                                          function t {
                                            cd $(mktemp -d /tmp/$1.XXXX)
                                          1. 15

                                            Instead of a t function, I have a ~/tmp folder.

                                            Because ~/tmp is persistent, I end up with stuff in there that I’m afraid to delete, so I made a ~/tmp/tmp for stuff that’s actually temporary.

                                            Because ~/tmp/tmp is persistent, I occasionally end up with stuff in there that I'm afraid to delete. I once just needed a temp folder quickly, but didn't feel like stuff in ~/tmp was safe to delete yet, so I made a ~/tmp/tmp/tmp`.

                                            I should add that t function to my rc.

                                            1. 3

                                              Ooo, I love the t function. I make temp dirs for stuff all the time. So simple, but really helpful - thanks for sharing!

                                              1. 2

                                                I have a similar j function which creates a named (default: junk) subdirectory in the current directory and cds to it. There is a corresponding jjj function which is essentially rm -fr junk. Because the new directory is under the current directory, I find it is easier to reference the original directory with .. rather than using "$OLDPWD".

                                              1. 28

                                                The thing I like about Iosevka is that you can build it with different sets of ligatures, and I’ve configured my Emacs to use the version of Ioseveka specific to the language in the buffer. That’s sort of cool. Of course, I’ve now officially spent more time setting Emacs up to write Haskell and OCaml in than I have actually writing Haskell or OCaml, but, you know, you gotta start somewhere.

                                                1. 48

                                                  I think you’re doing fine. I’ve spent more time reading about people setting up their Emacs than I ever used Emacs

                                                  1. 3

                                                    I always try to teach Haskell using nano otherwise everyone is like “ooo what are you using?” and “can I get something like that but for jEdit/Sublime/Atom/VS?”

                                                    1. 3

                                                      Hah! Nice. I can certainly empathise with spending more time configuring Emacs than actually using it to write things. Do you have your configuration published anywhere? I’d be interested in seeing the font set-up :)

                                                      In the interest of sharing, here’s mine: https://github.com/cmacrae/.emacs.d

                                                      1. 1

                                                        Here’s some horrible elisp, assuming “Iosevka Haskell P” and “Iosevka ML P” are proportionally spaced versions of Iosevka with the appropriate ligatures defined:

                                                        (defun jfb/frame-is-displaying-?-buffer (frame mode)
                                                          (eq mode (buffer-local-value 'major-mode (car (buffer-list frame))))) ;; this assumes that the first buffer in a frame's list of buffers is the visible one. Sigh.
                                                        (defun jfb/frames-that-? (predicate)
                                                          (require 'seq)
                                                          (seq-filter predicate (visible-frame-list)))
                                                        (defun jfb/assign-font-to-frame (predicate-to-pick face-to-assign)
                                                          (setq frames (jfb/frames-that-? predicate-to-pick))
                                                          (if frames   (set-frame-font face-to-assign
                                                        (defun jfb/define-haskell-font ()
                                                          (jfb/assign-font-to-frame (lambda (f) (jfb/frame-is-displaying-?-buffer f 'haskell-mode)) "Iosevka Haskell P:weight=ultra-light:slant=normal:width=normal:spacing=100:scalable=true"))
                                                        (defun jfb/define-ocaml-font ()
                                                          (jfb/assign-font-to-frame (lambda (f) (jfb/frame-is-displaying-?-buffer f 'tuareg-mode)) "Iosevka ML P:weight=ultra-light:slant=normal:width=normal:spacing=100:scalable=true"))
                                                        (defun jfb/fixup-fonts ()
                                                        (global-set-key [f9] 'jfb/fixup-fonts)
                                                      2. 2

                                                        How are you handling ligatures? AFAIU emacs doesn’t render opentype ligatures?

                                                        1. 3

                                                          On MacOS it does!

                                                          1. 1

                                                            It doesn’t out of the box, no - even if the new shaping engine might help on that front? - but you can tell it to use different glyphs for arbitrary character combinations. There’s apparently at least three different ways to tackle this, by using prettify-symbols-mode, a composition table, or font-lock. All of them, though, are specific to a single font, but there should be instructions for most of them nowadays.

                                                          2. 1

                                                            Ligatures are some of the finest things known to man. Are you familiar with Chartwell? Are you doing that sort of thing in emacs?


                                                          1. 9

                                                            illumos, and moreover Triton/SmartOS from Joyent, is excellent.

                                                            Although I’ve done a recent rebuild of my home infrastructure and have moved on, I spent years running Joyent’s cloud platform: Triton, on a cluster of Intel NUCs. It’s great that they offer it open source, and I highly recommend people check it out, if they’re unfamiliar. Although we’re living in an ephemeral container-centric world, with lots of cool constructs and patterns evolving, the notion of having a container that acted just like a HVM was always a pleasurable and exciting one (illumos Zones, check them out!).

                                                            And of course, Joyent really pushed their engineering with a great Docker API solution, too! So, I had a bunch of services running in zones, and a fair few containers too. All wrapped up with Terraform, Packer, and Ansible for provisioning. A lone KVM instance running OpenBSD for my OpenIKED VPN. I’m just rambling now, but I’m sure people can tell I loved that stack, and it demonstrates how flexible it is for something you can set up at home/in a private DC.

                                                            TL;DR - if you’re not familiar with illumos, SmartOS, Triton (and Joyent in general), definitely check out their stuff. It’s all open source, and is really cool!

                                                            1. 4

                                                              Illumos is in my top two companies I would trust to run a docker container in production along with Google. I trust them because

                                                              1. They have really solid systems engineers.

                                                              2. Neither of them actually run the docker engine in production.

                                                              1. 2

                                                                You also gain dtrace and the best ZFS implementation. I’ve never had to run Docker in prod but this has been my planned solution since this became possible.

                                                                1. 2

                                                                  Absolutely! There are some fantastic technologies that you get at your fingertips. I also forgot to mention in my post how exciting the Linux syscall translation was when it hit. OS level virtualization (containers) of the Linux kernel… on an illumos host. Mindblowing stuff. There are some excellent talks out there from @bcantrill (that are always very entertaining) on many of the things I’ve noted. I’d urge anyone reading, who’s curious about any of this, go watch some of them :)

                                                                  1. 2

                                                                    I was jealous for a long time because Zones were a bit more “complete” than FreeBSD jails and then their Linux syscall translation was also more complete than FreeBSD’s…

                                                                    Things are better now in FreeBSD land but Illumos still has a more polished solution…. and a damn fine network stack… and a damn fine CPU scheduler… and a damn fine memory management…

                                                                    If Solaris was open sourced sooner I don’t know what the world would look like

                                                                  2. 1

                                                                    I’m a big fan of both of those technologies. So it only sweetens the deal for me.

                                                                  3. 1

                                                                    Doesn’t Google use Docker in production? That was surprising, to me.

                                                                    1. 4

                                                                      Nope, They use their own container technology which predates docker by over a decade. They just wrap it in a docker api facade for you to make it easier for you to interact with it.

                                                                  4. 1

                                                                    What are the reasons for moving?

                                                                    1. 2

                                                                      Good question! To be honest, although I loved the stack, it had gathered dust for a while. Certainly in a sense of the methods I was using to define my infrastructure. The landscape changed pretty drastically in a short period of time, in the Ops world. I was doing all this stuff with kubernetes and GitOps at work, and still deploying with Terraform and Ansible at home.

                                                                      A large part of why I have my home setup is to learn things, try things, develop things. I felt I wanted a stack that closer represented the things I was currently enjoying.

                                                                      I could have tried out running k8s on top of Triton, but to be honest, the implementation Joyent have blogged about looks a little hefty for my liking (and my resources). It leverages KVM instances to run various k8s components.

                                                                      I’ve been thoroughly enjoying Nix (and NIxOS) for quite some time, so I decided I’d redesign my home cluster:

                                                                      • NixOS on the metal
                                                                      • All system expressions deployed to the servers via NixOps
                                                                      • Declarative setup of k8s and some accompanying ‘core’ services
                                                                      • k8s services defined with YAML/kustomize, slurped in and deployed via GitOps with ArgoCD

                                                                      I’ve been a massive nerd about it all and captured everything in a GitHub project, with a roadmap and issues for everything I plan to implement.

                                                                      Whilst I’m excited about it, it’s largely blocked at the moment by the state of k8s deployments on NixOS. The modules provided to bootstrap a k8s cluster are a bit wonky in their current state. I believe ‘offline hacker’ is doing a complete rework of it all in the background. So I’m very much looking forward to his work.

                                                                      1. 1

                                                                        Out of curiosity, is that GitHub project public?

                                                                  1. 12

                                                                    Another article about macOS package managers which fails to mention Joyent’s binary sets for pkgsrc on macOS… I’m beginning to think I’m the only one using them.

                                                                    1. 8

                                                                      I used to maintain a bootstrapper script for getting people going with Joyent’s pkgsrc builds on macOS a few years back: https://github.com/cmacrae/savemacos

                                                                      I have since moved to Nix, as I use it for more than just package management. But, I can assure you; you won’t be the only one using those packages!

                                                                      1. 3

                                                                        I used to use savemacos back then, thank you a lot for making something useful and sharing it.

                                                                        1. 2

                                                                          Ah well that’s nice to hear! You’re very welcome

                                                                      2. 3

                                                                        You are not

                                                                        1. 3

                                                                          I’m not using /u/jperkin’s binary packages but I’ve been using pkgsrc on macOS since ~2007 (I’m a former heavy NetBSD user, so rather like pkgsrc). I’m pretty happy with it although I’ve never explored any of the other options (MacPorts, Homebrew, etc).

                                                                        1. 7

                                                                          For those of us more BSD inclined, I’ve been using a modified version of this OpenBSD/iked setup to VPN into my community WAN (it uses internally) from my office. (The only problem I seem to be having is with Debian clients not working with PSK).

                                                                          Rather than performing a bunch of manual VPN provisioning, I prefer algo which is a quick way to stand up a vpn + generate Apple .mobileprofiles on a bunch of cloud providers.

                                                                          1. 2

                                                                            Nice I am a big OpenBSD fan and used to use it for my home firewall… now I don’t have a home ;-)

                                                                            1. 1

                                                                              Glad to see people found this write up useful :) Thanks for sharing with others

                                                                            1. 4

                                                                              Nice! Gunna chime in here and tell you to check out my collection of Nix expressions for my fully declarative, functional, atomic configuration I use across my macOS systems:

                                                                              Start by digging through the darwin-configuration.nix if you’re curious, it’ll lead onto the other expressions with include statements.

                                                                              Of course, this is all achievable on NixOS (and a lot more), and any system that supports Nix.
                                                                              Hope someone finds this helpful/inspirational!

                                                                              1. 6

                                                                                Woo! Excited to see some Nix stuff being mentioned.

                                                                                I had been running NixOS on my work Macbook for a while, but due to issues with a mix of HiDPI & non HiDPI displays on Linux - I went back to macOS.

                                                                                With a new found love though, I was delighted to find out about nix-darwin - LnL has always been really friendly and helpful on #nixos, when I had questions about achieving something with it.

                                                                                As such, here’s my collection of expressions that declare the system configuration I use across my macOS machines: https://github.com/cmacrae/.nixpkgs

                                                                                I’d certainly consider myself an absolute novice, but as you can see - even with little experience you can cobble something fairly comprehensive together.

                                                                                My next plans are formed around my home infrastructure. Currently, I have a little rack - with one shelf occupied by a little Joyent Triton cluster made up of 3 intel NUCs. On top of Triton I run a number of home media services which are in lx-branded OS containers. Right now, I’ve formed a workflow around Packer, Ansible, and Terraform for creating images and deploying services.

                                                                                I’m planning to introduce NixOS as a base lx-branded OS image, which you could then “inject” Nix system expressions into for declarative, reproducible images for varying deployments and services.

                                                                                1. 2

                                                                                  I find nix-darwin and your nixPGS for macOS really interesting. I want to try it out, but am pretty much non-nix smart.

                                                                                  I’ve just ordered a new MacBook, and will need to move everything over.
                                                                                  Is there a way to take your existing configuration and put it into nix-darwin?

                                                                                  Is there an idiot’s guide to getting started and making this all work somewhere?

                                                                                  How do I know the name of the variables I can set?


                                                                                  system.keyboard = { enableKeyMapping = true; remapCapsLockToControl = true; };

                                                                                  this is awesome, but how could I have figured it out except seeing it in your config?

                                                                                  I’m not exactly an idiot, but around nix, I definitely am :)

                                                                                  1. 4

                                                                                    I’m afraid there’s no good answer to that at the moment, I should probably look into how nixos builds the configuration.nix manpage. Currently you’ll need to use the darwin-option command or look at the sources.

                                                                                    1. 3

                                                                                      Hey, I’m pretty much non-nix smart too!

                                                                                      The only means of taking your existing config and putting it into nix-darwin is to express it yourself.

                                                                                      The NixOS manual and Nixpkgs manual are the best resources for learning.

                                                                                      Regarding knowing the expressions you can use, I’ve found it’s a combination of what you learn from the NixOS manual and digging through the available modules in nix-darwin. It’s pretty easy to understand what’s possible with these when reading them. Nix is pretty verbose, so it should be clear what values you can/cannot pass to module parameters.

                                                                                      On top of that, you also have my config to go on! And LnL has an ‘examples’ directory in the ‘modules’ part of nix-darwin that you may also be able to learn a bit from.

                                                                                      1. 2

                                                                                        Poking me on irc is also an option, I’m happy to answer any questions you have. As for things that are not supported yet feature requests or contributions are very welcome.