The whole point of shell scripting is getting shit done and move on to focus on more important stuff.
Once the script takes the life of its own and try to grow up, just spend some time on this teenager and make it speak in a real adult language. Another option that is sometimes preferable would be keeping it young and little, while replacing its limbs with cybernetics made of the real grownup stuff.
As to the last point, “everything is a string”, I find it amusing that shell script is so lacking when it comes to readable string-handling operators and functions. Back when I really started learning to program on Unix, Perl was at the height of its popularity. I actively resisted learning shell in any depth for many years, and that was my reasoning. Why work in a stringly typed language that lacks nice string-handling, when I already have Perl?
Takes several lines of code and a lot more brainpower in many programming languages.
I completely agree with this. I often work making pipelines to process metagenomic data, which is often a lot of data with a lot of intermediate steps, all to reduce the raw data into a simple table that can be easily understood by ecologists. At first, I used to make my pipelines with Lua (great language). The script used to be ~1500L. Pretty small for programming standards, but too big for a pipeline that consisted of around 7-10 steps. Later I translated the everything that could be translated from Lua to shell script, and I ended up with a shell script <100L and a Lua script with <200L (the Lua part that I couldn’t translate to shell script).
Operations like iterating over files inside a folder is much easier in shell script, and was something that I had to do again and again on every step of that pipeline. That’s why a lot of lines were reduced.
I don’t think shell script is so intuitive, so I always kept mine small, executing existing tools or my Lua scripts. If working with shell, I think this hybrid approach prevents you from falling in this “trap”.
I agree with the overall message, but I find some of the particulars to be non-issues. Personally, I instinctively stick to the following, and it avoids a lot of issues without having to switch to a “big boy” language quite as soon/often:
Always quote variable substitutions, e.g. "$foo" never $foo. I find it disingenuous when someone (not this author, but it’s common enough) complains about e.g. filenames with spaces, says they don’t want to quote everything because it’s ugly/hassle/etc., then claims to prefer a language like Python/Go/etc. where all strings must be quoted. Why is it ugly/hassle in a shell script but not in those other languages? Don’t think about it as “adding quotes”, just default to treating shells like any other language.
Never use word splitting. If the shell has arrays then it’s basically an anti-feature, so I just pretend that it doesn’t exist so that I’m never even tempted to use it. Again, just quote everything ;)
Treat each shell as a different language. The difficulty of making scripts “portable” is a common complaint about shells, which is valid but again rather disingenuous if the “solution” is to enforce a particular dependency like Go (for compiling, at least) or Python (which is usually shorthand for something like “CPython > 2.7 && < 3” or “CPython 3.5+ with these packages…”). If you’re going to force a particular interpreter for Python/etc., then why not enforce a particular interpreter for your shell scripts? If it’s fine to write a Python script, then it’s fine to write a “bash 4+ script”. Who cares if it doesn’t run on zsh or dash? Python scripts don’t run on zsh or dash either.
These days I tend to go even further and use Nix to bake-in the versions of bash, grep, sed, etc. that I’m using, with a pinned nixpkgs version. I also do this for python scripts, etc. as well. If I have a collection of scripts, I’ll also have Nix run Shellcheck on all of them as part of their “build”; that can help catch things like undefined variables too. I don’t tweak the settings: just abort on everything, and add # shellcheck disable=SC123 comments as appropriate.
I find it disingenuous when someone (not this author, but it’s common enough) complains about e.g. filenames with spaces, says they don’t want to quote everything because it’s ugly/hassle/etc., then claims to prefer a language like Python/Go/etc. where all strings must be quoted. Why is it ugly/hassle in a shell script but not in those other languages?
I am the author (better late than never), and I always quote variables; I don’t mind quoting variables (I never claimed that it’s “ugly”).
The problems extend beyond just “quote everything”. This article gives a good overview. Admittedly, some of these scenarios are a bit rare, but they do exist.
Equally important, it’s easy to do it wrong if you’re not careful. I always have to run after other people’s script to add quotes and fix other small things like that. I much prefer a language/environment that makes it easy to do the right thing, and hard to do the wrong thing instead of the other way around.
I don’t mind when a shell script grows more lines. I do however have a couple rules for when it’s time to move.
If I have to do argument parsing. I really hate getopt in shell
If I need a data structure that I can remember how to use
If it needs a unit tests (and I usually decide it’s time for unit tests early on.
For me that still leaves room for things like backup scripts and many other types of automation. Also it does leave room for simple data stream processing.
You’d probably get more people if you wrote a list of why newLISP is a great shell language with examples instead of a rant. It certainly looked neat when you last mentioned it. Here’s some quotes from the FAQ for anyone interested:
“newLISP is a LISP-like scripting language for doing things you typically do with scripting languages: programming for the internet, system administration, text processing, gluing other programs together, etc. newLISP is a scripting LISP for people who are fascinated by LISP’s beauty and power of expression, but who need it stripped down to easy-to-learn essentials.
…pragmatic and casual, simple to learn without requiring you to know advanced computer science concepts. Like any good scripting language, newLISP is quick to get into and gets the job done without fuss… newLISP has a very fast startup time, is small on resources like disk space and memory and has a deep, practical API with functions for networking, statistics, machine learning, regular expressions, multiprocessing and distributed computing built right into it, not added as a second thought in external modules.”
BTW: I do not care if you use newLISP. I’m not interested in “getting more people”. Use it if you want. Or not (it’s your loss, not mine). I’m just sharing a tip.
I was, however, interested in writing a long rant, and I enjoyed the process thoroughly. It was very cathartic. :D
One of the cool ideas I’ve run across (I think from Paul Graham’s On Lisp) is petrification of a program - stabilizing and formalizing the program past the quick and dirty stage. I know that type hints/gradual typing are helping this, but would love to see more ideas (besides @andyc’s Oil) that can transition shell/quick scripts to something with more types, error handling, composability (besides pipes).
Dryly amusing: my .sig for years (a decade or so ago) used to contain a short example of just how zsh does handle NULs in strings.
Loosely speaking though, the moment that you’re using the beyond-POSIX features of bash or zsh for anything other than REPL control, that’s a sign that you’re entering technical debt territory and should be rewriting, now that the shell prototype has confirmed what needs to happen and what the general failure modes are.
The whole point of shell scripting is getting shit done and move on to focus on more important stuff.
Once the script takes the life of its own and try to grow up, just spend some time on this teenager and make it speak in a real adult language. Another option that is sometimes preferable would be keeping it young and little, while replacing its limbs with cybernetics made of the real grownup stuff.
I would say the point of shell is composing other programs well, not hacking stuff poorly, that is what python/perl is for :)
That being said, I agree with you, shell has so much potential, but is so shitty for some unknown reason.
For me the whole point of shell scripting is that if it breaks, I can easily decompose the program into parts, and execute each part independently.
For getting things done, I’d rather use any general-purpose dynamically-typed scripting language.
As to the last point, “everything is a string”, I find it amusing that shell script is so lacking when it comes to readable string-handling operators and functions. Back when I really started learning to program on Unix, Perl was at the height of its popularity. I actively resisted learning shell in any depth for many years, and that was my reasoning. Why work in a stringly typed language that lacks nice string-handling, when I already have Perl?
I completely agree with this. I often work making pipelines to process metagenomic data, which is often a lot of data with a lot of intermediate steps, all to reduce the raw data into a simple table that can be easily understood by ecologists. At first, I used to make my pipelines with Lua (great language). The script used to be ~1500L. Pretty small for programming standards, but too big for a pipeline that consisted of around 7-10 steps. Later I translated the everything that could be translated from Lua to shell script, and I ended up with a shell script <100L and a Lua script with <200L (the Lua part that I couldn’t translate to shell script).
Operations like iterating over files inside a folder is much easier in shell script, and was something that I had to do again and again on every step of that pipeline. That’s why a lot of lines were reduced.
I don’t think shell script is so intuitive, so I always kept mine small, executing existing tools or my Lua scripts. If working with shell, I think this hybrid approach prevents you from falling in this “trap”.
I agree with the overall message, but I find some of the particulars to be non-issues. Personally, I instinctively stick to the following, and it avoids a lot of issues without having to switch to a “big boy” language quite as soon/often:
"$foo"never$foo. I find it disingenuous when someone (not this author, but it’s common enough) complains about e.g. filenames with spaces, says they don’t want to quote everything because it’s ugly/hassle/etc., then claims to prefer a language like Python/Go/etc. where all strings must be quoted. Why is it ugly/hassle in a shell script but not in those other languages? Don’t think about it as “adding quotes”, just default to treating shells like any other language.# shellcheck disable=SC123comments as appropriate.I am the author (better late than never), and I always quote variables; I don’t mind quoting variables (I never claimed that it’s “ugly”).
The problems extend beyond just “quote everything”. This article gives a good overview. Admittedly, some of these scenarios are a bit rare, but they do exist.
Equally important, it’s easy to do it wrong if you’re not careful. I always have to run after other people’s script to add quotes and fix other small things like that. I much prefer a language/environment that makes it easy to do the right thing, and hard to do the wrong thing instead of the other way around.
I don’t mind when a shell script grows more lines. I do however have a couple rules for when it’s time to move.
For me that still leaves room for things like backup scripts and many other types of automation. Also it does leave room for simple data stream processing.
I wrote a long rant on this same subject and am using newLISP instead. It’s great.
You’d probably get more people if you wrote a list of why newLISP is a great shell language with examples instead of a rant. It certainly looked neat when you last mentioned it. Here’s some quotes from the FAQ for anyone interested:
“newLISP is a LISP-like scripting language for doing things you typically do with scripting languages: programming for the internet, system administration, text processing, gluing other programs together, etc. newLISP is a scripting LISP for people who are fascinated by LISP’s beauty and power of expression, but who need it stripped down to easy-to-learn essentials.
…pragmatic and casual, simple to learn without requiring you to know advanced computer science concepts. Like any good scripting language, newLISP is quick to get into and gets the job done without fuss… newLISP has a very fast startup time, is small on resources like disk space and memory and has a deep, practical API with functions for networking, statistics, machine learning, regular expressions, multiprocessing and distributed computing built right into it, not added as a second thought in external modules.”
You mean like this?
BTW: I do not care if you use newLISP. I’m not interested in “getting more people”. Use it if you want. Or not (it’s your loss, not mine). I’m just sharing a tip.
I was, however, interested in writing a long rant, and I enjoyed the process thoroughly. It was very cathartic. :D
One of the cool ideas I’ve run across (I think from Paul Graham’s On Lisp) is petrification of a program - stabilizing and formalizing the program past the quick and dirty stage. I know that type hints/gradual typing are helping this, but would love to see more ideas (besides @andyc’s Oil) that can transition shell/quick scripts to something with more types, error handling, composability (besides pipes).
There is the Oh shell: https://github.com/michaelmacinnis/oh
Excellent point. I finished watching the BSDCan video (from the lobsters discussion) , but haven’t dug into playing with it yet.
Dryly amusing: my
.sigfor years (a decade or so ago) used to contain a short example of just how zsh does handle NULs in strings.Loosely speaking though, the moment that you’re using the beyond-POSIX features of bash or zsh for anything other than REPL control, that’s a sign that you’re entering technical debt territory and should be rewriting, now that the shell prototype has confirmed what needs to happen and what the general failure modes are.