1. 4

  2. 2

    Posting this now because it’s one of those insanely useful things that I only found out about recently. If you use bz2 archives, you can use tar‘s -I / --use-compress-program options to specify pbzip2 and of course it’ll use all your CPU cores to do the (de)compression work. I had cause to do this with a double-digit-GB archive on a 128-core VM recently and in keeping with the docs’ claim that it achieves a “near-linear speedup” with SMP, the difference in that case was pretty spectacular.

    1. 1

      pbzcat is single core. lbzcat can use multiple cores. bzip2 format can be decompressed in parallel because the compressed stream is made up of independent blocks.