1. 4

    I have ten years worth of files at ~/doc/archive/yyyy/mm/dd, all collected by running a very simple archive command. I use find and grep to search these, which work well enough to keep me from firing up elasticsearch. About 5 years ago, I moved ~/doc/archive from from local disk to a ZFS NAS with a quick fstab entry and a symlink.

    I rotate encrypted drives offsite for the entire NAS, because sneakernet still has the best bandwidth. But given the archive is append-only, it readily syncs nightly to a bucket which is encrypted (client-side!) with a script using duplicacy.

    Photos get their own yyyy/mm/dd directories, to avoid having to search/filter with the rest of the archive, which turns out to be the organization strategy used by shotwell, so I use it to import and index my photos. I loaded Resilio Sync (for free) on all mobile devices to transfer latest photos over wifi, and use a script to gather all photos from the last N days whenever I fire up shotwell to look at photos.

    I have a separate timestamped file which collects “journal” thoughts and snippets from books/code/chats (and I even have my screensaver write to it when it my screen blanks/unblanks, which happens to be useful when billing hours as a consultant), which ultimately is just stdin appended to a file with a lock to avoid corruption. grep is natural for this, but I most often use less with a keyword search to jump around. If you’re wondering whether that scales, consider the mbox format for email.

    No need for Blu-ray discs, unless you are looking forward to labeling them with your favorite album titles to obfuscate your archive. Not to mention the manual work needed to transfer these to your next epoch of storage. Even if you don’t go for the NAS, my Backblaze B2 bill is just over a dollar per month ($1.35 for about 300GB and the append traffic) and a few thumb drives at $20-50 each will get you covered with redundancy and nearline offsite backup (keep an encrypted thumb drive in your car or at a friend’s home).

    1. 2

      Neat! That’s very similar.

      re:bluray – I don’t find that’s something for obfuscation, it’s just the cheapest and least-maintenance way to store some data for 10-15 years (any other technology would probably need to be migrated in that time-frame). ~75 cents to burn a 25GB disc that will still be readable in 2030 is sort of attractive to me. I do use AWS Glacier as well.

      1. 2


        re:bluray – I don’t find that’s something for obfuscation

        I was building on your Mr. Robot reference. :-)

    1. 2

      A comment fit for the “What are you working on this week?” thread, I am developing qwerty.sh as a curl-ready POSIX-portable shell program to download, verify, and unpack files with a single command (on Unix systems). The README includes examples.

      This provides an alternative to the ever-common curl ... | sh copy/paste installation instructions for developer tools, and I use it regularly as part of a workflow to have all projects go from a fresh clone to a fully integrated environment in a single-command repeatable build (e.g. with one-liners in a Makefile to download/verify/unpack external resources).

      I test qwerty.sh on Ubuntu GNU/Linux, Mac OS X, and FreeBSD.