1. 26
  1. 14

    It’s really tempting to wonder why the author didn’t do X or Y. But I really, really like that they wrote about the mistakes they made. We all make mistakes but most of us are very shy about admitting it.

    1. 11

      Hetzner offers volumes (see “block storage modules”) that can be between 10GB and 10TB in size and easily mounted on a server. They are billed by the hour, so even though the 10TB volume costs 487,42€/month, if you rent a volume for only 3-4 days it will just be around 40-50€, and that is an extreme example.

      Anyway, the huge advantage is that these volumes are within the same bloody data centre (instead of some Tarsnap-data-centre halfway around the world) and you can attach them to multiple servers. It is very easy to do data-transfers this way.

      Even easier is to set up a private network (Hetzner offers that) and just transfer the files directly. Doing such things in-situ is always risky, as many others pointed out, and I’d always recommend to set up a new machine, do the transfer, and then switch. Hetzner offers floating IPs that can make such a process really simple.

      1. 8

        This is quite inspiring. I’d like to suggest another lesson: practice restoring from backups. Everybody’s backups are unique, because they’re a reified portion of your life. There’s really no way to have a restore go smoothly without rehearsing it a few times.

        Eventually it’ll suffice to do a test restore once or twice a year. But initially try it a few times. It’s kinda fun to see your life reflected back at you.

        1. 6

          You can attach strace to a process to see what it is doing. Grep for “open” to see what file it is at.

          1. 4

            Someone reached out to me via email to say the same thing! Got sent this article.

            I was using bottom to monitor the entire system, but strace will definitely be a good tool to add to my kit.

          2. 2

            I’m sorry you had a hard time with this. Did you not keep the old machine around while you were setting up the new one? Whenever I upgrade the OS on my VPS I do that. I also tend to tarball the whole disk rather than try to select files and then I keep the tarball around for a few months after I case I had forgotten something.

            1. 3

              I was thinking the same, but it seems that OP is trying to change the OS in place. I imagine it is a vps with longer term commitment, not a hetzner cloud instance.

              1. 2

                I think it’s a Hetzner dedicated server, with monthly rates and a setup fee.

                The following reads sorta like nitpicks, I know. I’m thinking aloud.

                I would have considered rsynced tarballs elsewhere; tarsnap can be slow in cases with latency to an East Coast AWS region, like, say, Germany. GPG tarballs to s3 or to Hetzner storage sound ok.

                It’s nice that it’s all in Docker. My last Alpine experience was unpleasant; any space savings were countered by lack of operational context and image caching anyway.

                If I was worried about downtime I’d maybe doing a prototype / trial migration to a vps with hourly or daily rates.

                A benefit of sticking to tarsnap is that it would show any gaps in backup coverage, such as the .env file here. Unfortunately, by doing this in place, it only surfaces any issues too late.

                The self-hosted Bitwarden blip feels … very close to a very serious outage, too.

                Thanks for the honest recap!

                1. 1

                  I ran them all with docker-compose from a Debian VPS.

                  Doesn’t sound like it was a dedicated server.

            2. 2

              Your plan to have decent backups is good; your decision to have only three is baffling.

              1. 1

                I figure: what would I ever need to do with a fourth backup? Just seems paranoid. I’ll almost certainly always restore from the most recent anyways. Plus, I can always increase it in the future (and am open to changing it now, just can’t think of why I would).

                1. 4

                  One real benefit to using tarsnap, specifically, is that tarsnap will deduplicate backed-up data across all backups. Deduplication doesn’t really help you if your server stores a rotating set of huge movies, but if you have a slowly-growing set of data, keeping old backups around is pretty much free.

                  1. 2

                    My backup scheme is:

                    • daily-01 to daily-31, these get overwritten as days progress.
                    • YYYY-MM, which never get overwritten.

                    The monthly snapshots are done on the first of every month.

                2. 2

                  My terminal sat empty for hours. There were no changes – the process was running, but there was no feedback. I was nervous.

                  What if it failed silently?
                  How can I check?
                  What should I do?

                  On a BSD, hit Ctrl-T.

                  On Linux, check to see if there’s a SIGUSR1 handler (there is), and send it that. Don’t do this blind, as the default handler will terminate the process.

                  To my horror, stats printed to the screen: the backup had been 90% complete, and I had stopped it. Convinced I had ruined the backup completely, I deleted the partial backup from Tarsnap and started again from scratch.

                  Oof. If you’d just have left the partial, tarsnap would have skipped uploading 90% of the next archive using timestamps and dedupe - you’d have lost very little in interrupting it.

                  1. 1

                    This is bemusing me somewhat. I’m planning an in-place OS cloud VPS upgrade, don’t need zero-downtime, so I’m thinking I’ll stop it, take a snapshot, run the upgrade, and if it doesn’t work out how I plan, I can restore the snapshot. Seems a lot simpler, am I missing something obvious?

                    1. 3

                      Sounds like a good plan to me

                      1. 1

                        Sounds a lot like my plan! I just made a whole bunch of mistakes.

                        1. 3

                          Oh, you should hear some of the mistakes I’m too embarrassed to tell you about …

                          1. 1

                            Oh, but no, you wiped your VPS. You had no place to restore it on a single click like with a snapshot.

                            A suggestion - you have all these fine things running as docker containers, why not trying to just raise them on your laptop? You don’t even need the docker daemon, just something like podman. Do that once a month or something to see if your backups are still good?

                        2. 1

                          This tale is exactly why I love ZFS. I’m able to completely migrate systems with zfs send | zfs recv. I’ve restored may backups created via zfs send. I’ve even migrated from physical to virtual and vice-versa too many times to count. No matter the scenario, it’s just zfs recv to restore or replicate the backup.

                          1. 2

                            To what media do you ZFS send your backups?

                            1. 1

                              Truly depends on the use case. I’ve backed up to a client-side encrypted iscsi volume served by my NAS using ZFS. I’ve backed up to USB drives. I’ve backed up to Google Drive. I’ve not backed up to Tarsnap, but that would be easily possible.

                              edit[0]: I’ve also backed up over ssh and sftp.

                              1. 1

                                So in all cases it’s to online media? Have you ever backed up to offline media, say optical discs?

                                1. 1

                                  Not for a couple decades. But creating an ISO (and burning it) with the data wouldn’t be too difficult.

                                  1. 1

                                    The real question is how it handles degrading media, or how realistic a threat that is on modern offline media

                                    1. 1

                                      Dunno. I haven’t burned a DVD in around two decades. ZFS provides checksums for every single block. For “modern offline media”, I just use a USB drive. The current one I use is 8TB. Works great.