Our host prgmr had a networking outage starting around 6:45 Central time (GMT-5) this morning. A switch rebooted and, after 30m, entered a reboot loop, taking their hosting entirely offline. They dispatched an employee to the data center to configure and install a backup switch.
When that was completed we came back online with everyone else. Unfortunately, our Let’s Encrypt cert was due to renew during the networking outage. So that didn’t happen and we came back up with an expired security certificate. @alynpost reissued the cert and we bounced nginx to get it working.
Between networking and the certificate were offline for about three hours. No data was lost, our vps did not go down. I don’t believe any part of this outage was malicious or has any security concerns.
Our hosting is donated by @alynpost, owner of prgmr and longtime lobster. During the networking outage I asked not to receive any updates unless all their customers were back up and something extra was wrong with Lobsters. I didn’t want to distract from their paying customers on what must be a very busy morning. We have no plans to change anything about our hosting.
And as long as I’m writing an
announce post many people will see, yesterday I fixed the bug that broke replies via mailing list mode.