I develop and use a lot of different scientific code in my work (I’m a theoretical chemist). I have pipelines with components written in C/C++, Python, AWK, Haskell, tcl, various shell dialects, and more. All of these programs and their associated languages aren’t going anywhere, not for a long time. When I add to them, I try to follow the “Unix philosophy” for my interfaces and build up single-purpose wrappers (sometimes via input redirection) where I can. To stave off reproducibility issues and to document what I’m doing, I aim for everything possible to be automated. Building initial “inputs” to a pipeline is fine, but I want to be able to run everything after that without fiddling by hand.
I end up writing lots of ad hoc shell scripts with pipes for orchestrating preparation, simulation, and analysis. Unfortunately, there are lot of ways to shoot one’s self in the foot with shell scripts even with “strict mode” and linters like shellcheck. (Crustaceans directed me to both; thank you!) Using Python’s subprocess helps in some ways, but I find it unwieldy for chaining a lot of processes together. Surely there’s a better way to do all of this that is straightforward to read, exploits the composability of programs, but doesn’t suffer from all the deficiencies of a BASH. What tool do/would folks here reach for in such a situation and why?
Any thoughts much appreciated.
tl;dr: What’s your preferred BASH replacement language for composing lots other programs and why?