1. 12

  2. 12

    Apropos of nothing, I still always do gzip -dc foo.tar | tar xf - because whatever tar I used in circa 1996 didn’t support gzip and I’m too old to change now.

    1. 8

      FreeBSD’s tar supports --zstd and --lz4[1], both of which are great for when you want fast compression and decompression. 55% compression seems pretty low from gzip, that’s about what I get on random data with zstd and lz4 with settings where I/O remains the bottleneck.

      I haven’t used gzip for a long time. Where I care about compression ratio at the expense of everything else, bzip2 and then xz displaced gzip. Where I want compression without impacting throughput, lz4 took over. These days, zstd has a great range of options and gives a better compression-ratio:speed tradeoff than gzip for everything I’ve tested.

      [1] The article mentions that GNU tar supports external tools, the nice thing about the built-in support is that tar x on FreeBSD detects all of these compression headers and does the right thing, without needing to pass z, Z, J, and so on when decompressing.

      1. 10

        55% compression […] what I get on random data

        Your random number generator is very broken!


        1. 4

          Shouldn’t be, I got it from a reputable source.

        2. 2

          Really the only advantage of tgz is compatibility, only bz2 comes close and it’s slow. So tgz exists only for slinging, especially many small files as the “solid” compression will be advantageous compared to zip files (the other ubiquitous format, even more so as AFAIK Windows still does not have built-in explorer support for tgz).

          1. 4

            it’d be cool if more gnu userland tools started supporting libarchive. it’s made things so much easier to work with on the bsd side: https://libarchive.org/

            1. 2

              Or if Linux distros would just ship bsdtar instead of GNU tar by default. It’s far more pleasant to use and one of the things that keeps annoying me when I’m in a Linux terminal.

              1. 1

                bsdtar (libarchive) is a dependency of pacman, so we ship it by default in Arch. I agree with you that it is far more pleasant to use!

            2. 3

              Funny enough, Microsoft announced today that they are adding support for tar, gz, etc. through libarchive. Wouldn’t have believed this was ever going to happen 10 years ago!

              1. 1

                It’s really not clear from that article whether this is support built into Explorer (the UI). They’ve been shipping some sort of tar for a while in the terminal.

                1. 1

                  Who wouldn’t have believed this? Anybody who didn’t read the Halloween Documents in the 1990’s.

                  Embrace, Extend, Extinguish.

            3. 3

              This looks like a use case for filesystem snapshots or a backup tool like restic.

              For tar + compression I would recommend tar [...] | zstd -T0 > archive.tar.zst. zstd is usually a good choice in terms of speed and compression ratio and -T0 tries to detect the number of physical CPU cores and uses as many threads.

              1. 1

                zstd -T0 -1 should be even faster (for me, it’s even faster than lz4 -1 because the latter doesn’t support multi-threading out of the box). Depending on the speed of your drive (eg on HDDs), it may be faster to write a zstd -1-compressed tar than a raw tar due to the size reduction reducing the write time.

              2. 2

                I suppose comparison to stock tar z is fair, but it’d be interesting to compare with gzip -1 too.