I enjoyed reading this. NixOS is still surprising to me, I don’t know how to summarize my experience but I think I still need to internalize NixOS a bit. I’ve chunked it as “grub labeling” which is reductive. My distracting thought while learning about NixOS was that rolling back a database with all of its state would require downtime because “select a grub label”. Is zero downtime with a production database possible with NixOS? How can you rollback something like that? If some external volume is exempt from Nix somehow, this still breaks the use case where a postgres major version bump horks everything.
Anyway, my own Nix learning aside, my only distracting thought while reading this article was similar: “another example of the database killing all good ideas”. How many ideas have a footnote about the database?
‡ Does not apply to database - abandon all hope
Random good ideas that are killed:
Containers - volumes are too tricky for a business outage
Backups - you didn’t do a full realistic restore / flip because your hardware is too big
Configuration Management - it’s the same as backups, you rarely exercise it fully
Load Balancing - the middle tier scales and deploys with grace
Of course I mean a relational database like mysql/postgres/etc. I know there are exceptions and shops that mitigate these things that resist. I’m not trying to enumerate the things that don’t match. My point is, the database kills good ideas. I don’t like state, this post is about not having darling state. But then we can’t easily apply it to a state server - which is what a database is.
If I understand the article correctly, then you can just use /persist directory to store your database data and it will be safe between restarts. About updates, as you can have multiple versions of the same software running at ease in the same OS (and all managed by the package manager) then I do not see much problem there. You spawn new server, copy data from one to another and then swap the listening daemon.
Is zero downtime with a production database possible with NixOS?
Absolutely. In fact, it’s probably easier (based on my anecdotal experience) to deploy NixOS systems that maintain zero downtime than many other distros.
If NixOS made statefulness impossible, it’d be unusable for hosting most things that required a database. Instead, NixOS removes the direct ability to mutate system configuration - as opposed to user and application data. So, on a NixOS deployment, you still have your runbooks for things like database migrations and such, but those would be just baked into your system or package config as systemd units or what have you.
A good example of this is how NixOS handles certbot/Let’s Encrypt - it deploys a systemd timer that takes care of managing private key generation and getting certificates - i.e. that service’s whole state in /var/lib - for you. If you change the configuration, then the deployed systemd timer changes too, but the certs and privkeys remain. Since the difference between config and data is enforced at a filesystem level, this lets you, as a sysadmin, make useful assumptions about what’s actually mutable data and what’s not.
I enjoyed reading this. NixOS is still surprising to me, I don’t know how to summarize my experience but I think I still need to internalize NixOS a bit. I’ve chunked it as “grub labeling” which is reductive. My distracting thought while learning about NixOS was that rolling back a database with all of its state would require downtime because “select a grub label”. Is zero downtime with a production database possible with NixOS? How can you rollback something like that? If some external volume is exempt from Nix somehow, this still breaks the use case where a postgres major version bump horks everything.
Anyway, my own Nix learning aside, my only distracting thought while reading this article was similar: “another example of the database killing all good ideas”. How many ideas have a footnote about the database?
Random good ideas that are killed:
Of course I mean a relational database like mysql/postgres/etc. I know there are exceptions and shops that mitigate these things that resist. I’m not trying to enumerate the things that don’t match. My point is, the database kills good ideas. I don’t like state, this post is about not having darling state. But then we can’t easily apply it to a state server - which is what a database is.
If I understand the article correctly, then you can just use
/persistdirectory to store your database data and it will be safe between restarts. About updates, as you can have multiple versions of the same software running at ease in the same OS (and all managed by the package manager) then I do not see much problem there. You spawn new server, copy data from one to another and then swap the listening daemon.Absolutely. In fact, it’s probably easier (based on my anecdotal experience) to deploy NixOS systems that maintain zero downtime than many other distros.
If NixOS made statefulness impossible, it’d be unusable for hosting most things that required a database. Instead, NixOS removes the direct ability to mutate system configuration - as opposed to user and application data. So, on a NixOS deployment, you still have your runbooks for things like database migrations and such, but those would be just baked into your system or package config as systemd units or what have you.
A good example of this is how NixOS handles certbot/Let’s Encrypt - it deploys a systemd timer that takes care of managing private key generation and getting certificates - i.e. that service’s whole state in /var/lib - for you. If you change the configuration, then the deployed systemd timer changes too, but the certs and privkeys remain. Since the difference between config and data is enforced at a filesystem level, this lets you, as a sysadmin, make useful assumptions about what’s actually mutable data and what’s not.