1. 3
  1.  

  2. 1

    This is a pretty good strategy. I’d recommend using something like Dead Man’s Snitch to warn you if the restore isn’t successful, rather than relying on you noticing the email not coming through. Those kinds of emails are easily tuned out.

    It’s probably also good to add some sanity checks to the restore script to make sure that there are roughly the correct number of rows. Even better would be to run some unit/integration tests against the restored DB.

    1. 1

      This is clever, nicely self-contained, and an excellent use for spot instances.

      That said, I clicked through because of the “Huge” in the title, and was a bit disappointed that we were still talking about the “few hundred GB” range of database size.

      1. 1

        Sorry if this is off-topic but I immediately thought of libre fm. It is not “huge” by any stretch of imagination but since we are talking about backups

        Next, I made a complete disk backup of all of the PostgreSQL files, and went to look at our backups on rsync.net. And they were broken — because of our disk space issues, we’d been failing to make a backup file, and our backup scripts had been backing up a 0 byte file. Not good.

        https://librefm.wordpress.com/2016/06/30/libre-fm-june-2016-downtime-what-happened/