Reminded me of this: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
When doing the same processing on a large number of files, I often just write a throwaway little 2-line makefile and use make -j…with a process substitution you don’t even need to save it to a file:
make -j
$ make -j`nproc` -f <(printf '%.bar: %.foo\n\tfoo2bar $< > $@\n') $(a=(*.foo); echo "${a[@]//.foo/.bar}")
Reminded me of this: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
When doing the same processing on a large number of files, I often just write a throwaway little 2-line makefile and use
make -j…with a process substitution you don’t even need to save it to a file: