NixOS ISOs from their website do not enable the serial console by default, but building a custom ISO which does is easy enough. I did so a few days ago on Debian using nix to create an NixOS installer for my APU2:

git clone --branch 18.03 --depth=1 https://github.com/NixOS/nixpkgs.git nixpgs
cat > serial-iso.nix <<EOF
{config, pkgs, ...}:
{
  imports = [
    <nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix>
    <nixpkgs/nixos/modules/installer/cd-dvd/channel.nix>
  ];
  boot.kernelParams = [ "console=ttyS0,115200n8" ];
}
EOF
nix-build -A config.system.build.isoImage -I nixos-config=serial-iso.nix nixpkgs/nixos/default.nix

Some years ago, a friend taught me a simple trick which I used twice to install OpenBSD at providers where neither OpenBSD nor custom ISOs were directly supported: We would build or download a statically linked build of Qemu, boot the VPS into its rescue image and start Qemu with the actual hard disk of the VPS as disk and an ISO to boot from. Thats not too hard and works for pretty much everything where you got a rescue system with internet access. I guess it should work for NixOS too and maybe nix could even be used for the qemu build ;)

If you want to give it a go and get stuck write support@prgmr.com and we’ll help you debug.

We do have some single points of failure in our network. We’ve been eliminating them as time permits but it is a work in progress.

We peer with HE and use NTT as a backup. If the event of an outage with HE we can get to the Internet with NTT instead. This switch that failed also had a ‘backup’ but not quite as warm as our peering connection. As a result of this failure we’ll be experimenting with Multi-Chasis Link Aggregation (MLAG).

Do you have any experience with MLAG or other switch-level redundancy?

I see.. Presumably not installed via a package (e.g. deb, rpm, etc) then?

By default certbot will try to renew certificates that expire < 30 days from “now”.

But the other reply solves the mystery (sort of): it was never called to renew.

I use acme-client since certbot causes Python to create memory mappings that are both writable and executable, which is a huge no-no in my book.

Oof. That sounds awful. Glad you got it sorted.

Internally we refer to that as a “label.” You’re correct, it doesn’t match the limitations in DNS. That label is used to not only set a hostname in .xen.prgmr.com, but also shows up in other systems. Most of our customers ignore this label in favor of setting up their own domain. In DNS you can set any valid A record for the IPv4 associated with your VPS. If you set an rDNS record for us you’ve got identical flexibility.

All that said, that limitation may also exist for legacy reasons that are no longer valid. Did you try to enter a hostname it would not accept or is it that you noticed it was more restrictive than it might need to be? I’m happy to look at changing the constraints here.

I found the comment about doing the renew manually. I think if anything, this makes your example even stronger. Meaning there is no single failure here, and it took a combination of all of them to create the long downtime you saw. In other words, there is no root cause.

I think you guys did a great job and I even learned about a new (to me) old service, prgmr!

In my case it feeds the new cert into the configuration management tool and it then pushes out the new config like normal. acme.sh itself is decoupled from the web server.