I also use set -e quite a lot. There is a very large and useful BashFAQ describing ways that set -e doesn’t do what authors might expect: https://mywiki.wooledge.org/BashFAQ/105
If you mean anything other than “accessibility”, could you please define what you’re talking about, or provide a link to a write-up of it? The abbreviation “a11y” is so universally used for “accessibility” (in the same way “i18n” is “internationalization”) that it’s effectively impossible to perform a search for other potential meanings.
Edit: Seeing as the automatic moderation event took place 5 minutes after my comment, it appears that my understanding of it being common syntax is true.
OK. I didn’t realize this was trying to suggest something about the metadata of the post; the context of the rest of the comment made it seem as if you were suggesting some strange set of bash flags that I’d never heard of.
This is exactly why I think these type of abbreviations are i5c [1]. It’s just a way to obfuscate words and confuse people. (Sorry for the overly salty reaction, but it’s a pet peeve.)
I wonder how many security issues are lurking underneath these inconsistent and weird semantics. This is the kind of muddled thinking that used to cause (and probably still causes) problems in PHP scripts.
I also use set -e quite a lot. There is a very large and useful BashFAQ describing ways that set -e doesn’t do what authors might expect: https://mywiki.wooledge.org/BashFAQ/105
Thanks. This is great learning for me. Fortunately, I switch to Python before such complications usually arise but you are right, one should not blindly add “set -e” to an existing BASH script they didn’t write.
I find all the admonishments to use POSIX shell rather than bash befuddling. Why would you not want to use Bash language features such as arrays and associative arrays? If your goal was portability across a wide swath of UNIX variants, then I understand that you might want to target POSIX shell, but that is a very unusual goal. In most instances why not leverage a shell language with very useful features?
Because arrays and associative arrays are two of the most broken features in bash. You can use them if you’re an expert, but your coworker who has to maintain the script will likely not become an expert. (I’ve been using shell for 15 years and implementing a shell for 4, and I still avoid them)
I implemented them more cleanly in Oil and some (not all) of the differences / problems are documented here.
${array} being equivalent to ${array[0]} is very confusing to people. Strings and arrays are confused.
associative arrays and arrays are confused. Try doing declare -A array and declare -a assoc_array and seeing what happens.
dynamic parsing of array initializers is confusing and buggy
in bash 4.3 and below (which is deployed on Ubuntu/Debian from 2016, not that old), empty arrays are confused with unset arrays (set -e). Not being able to use empty arrays makes them fundamentally broken
I could list more examples, but if you’ve used them even a little bit, you will run into such problems.
In Oil arrays and assoc arrays are usable. And you even have the repr builtin to tell you what the state of the shell is, which is hard in bash.
edit: I forgot that one of the first problems I wrote about is related to array subscripts:
A statement as simple as echo $(( a )) is in theory vulnerable to shell code injection.
Shells that do not have arrays do not have the security problem.
You can inject code into shells with arrays with a piece of data like a[$(echo 42; rm -rf /)].
All ksh derived shells including bash have this problem. Note that you don’t even have to use arrays in your program to be vulnerable to this. Simply using arithmetic in a shell that has arrays is enough! Because arithmetic is dynamically parsed an evaluated, and accepts $(command subs) in the array index expression.
bash is poorly implemented along many dimensions, but I would say arrays are one of the worse areas. But I learned that it got a lot of that bad behavior from ksh, i.e. “bash-isms” are really “ksh-isms”. So the blame for such a bad language to some extent goes back to AT&T and not GNU.
Using strings and implicit splitting for known/trusted arguments. If you’re building up flags to a compiler like -O3 or -fsanitize=address, this is a reasonable option, and the one I use.
Using arrays and taking care not to use empty arrays (which is annoying).
The fact that it’s awkward is of course a motivation for Oil :)
in bash 4.3 and below (which is deployed on Ubuntu/Debian from 2016, not that old), empty arrays are confused with unset arrays (set -e). Not being able to use empty arrays makes them fundamentally broken
Did you mean set -u? And to be fair, I would consider this more of a case of set -u being broken; it seems that their fix for it was to not trigger (even when unset) on * and @ indexes.
Yes set -u. Either way, you can’t use empty arrays and “strict mode” together, which makes arrays virtually useless in those recent version of bash IMO.
I have a memory of running into that the first time, and I couldn’t understand what was going on. And I ran into many more times after that, until I just stopped even trying to use arrays.
It’s not as “unusual” as you might think though. There are many people running OpenBSD on their work machines, where bash is simply not included in the default install.
I could see why you’d want to make your script work without issue when bash isn’t an option. At the very core, a shell script should not do much: cleanup your local mailbox, aggregate multiple RSS feeds or perform a quick backup using rsync, …
I have run openbsd on a laptop for some time, and I was delighted to see that all my personnal scripts were still working like a charm (besides calls to seq(1), this one killed me…). I’ve also written some simple C software, and wrapper scripts for it using POSIX shells only, and I was glad that nobody from the BSD world bothered me with these scripts.
Another argument is that POSIX shells is much more simple to use and understand than the myriad of corner-cases and specificities that bash can have. It sure is a strengh to be able to use proper arrays, but bash manpage has 5 times more (48746 words) than sh (10224). Mastering bash as a language is definitely harder than sticking to POSIX, especially when you only write scripts that are < 20 lines.
I use systems without bash installed by default. If you’re going to write a complex script that is too awkward to write in POSIX sh, then why not use a better (less quirky) language in the first place? Python for example is probably available in almost as many places as bash now. If you’re writing a shell script, you’re probably aiming for portability anyway, so please write POSIX shell if you can.
I write many bash scripts where portability is not a concern and I think the language can be quite functional and even elegant when you are pulling together different cli tools into a new program.
Ubuntu shaved quite a bit off their (pre-systemd) boot times by just switching /bin/sh from bash to dash (although I can’t find the exact number right now). The increased performance may be a concern (or is at least nice) in some cases (bending over backwards for it would of course be silly).
Certainly if it’s not a concern and you enjoy the language then I don’t see a problem with using bash! My main issue is that I don’t think it’s at all an unusual goal to use POSIX sh for portability.
When I talk about writing shell applications, I personally am talking about using bash plus all the standard UNIX tools that come with every UNIX base install.
And when I say do the heavy lifting in awk, I mean more or less ignore the shell, and use awk. You get proper arrays, reasonable syntax for writing algorithms, relatively little shelling out, relatively sane string manipulation, and sidestep a bunch of issues around shell variable reexpansion.
If you’re writing anything complex at all, use a different language.
That’s what I was addressing.
To your point, there are customer environments where shipping UNIX systems with languages like Python is prohibited, either because of security constraints, maybe disk usage, etc.
bash versus POSIX shell? If bash ships as /bin/sh on a given UNIX system, I don’t see the difference as important enough to be worth any gymnastics to get one installed, but you may have a higher dimensional understanding of the differences than I do.
I don’t think it’s really a big issue. Certainly for myself it’s only a minor frustration if I need to install bash for a simple script which gains nothing from using bash. I’ve just seen the #!/bin/bash shebang far too many times when it’s not necessary, and I think that a lot of the time that’s just because people don’t know the difference. It’s certainly not the end of the world and if bash feels like the right tool for whatever your use case is then I’m not going to argue! It would just be nice if there was a little awareness of other systems and that people would default to #!/bin/sh and only consider bash if they find a specific need for it. I imagine that in most cases shell scripts are used there actually is no need.
I’m a little obsessed with building my workflow around shell scripts and I have never found a need for bash extensions (YMMV). The other big benefit other than portability as others have suggested is the digestibility of the manpages for, e.g. dash.
And the corollary: The first line of your shell script should be
#!/bin/sh
If you need bashism for something complex that needs to be maintainable, use #!/usr/bin/env bash as the article suggests, but 99% of ‘bash’ scripts that I’ve seen are POSIX shell scripts and work just fine with any POSIX shell.
And the corollary: The first line of your shell script should be
I disagree, enormously. There are some specific, isolated, and extremely uncommon situations where it’s important you target sh; the rest of the time, you are constraining yourself for no appreciable benefit.
In particular, if you did go and change the interpret to /bin/sh, you also need to change the second line, since (at least) pipefail is a bashism.
One does not necessarily follow the other. I’ve seen horrific apparently-posix-sh scripts with the most unmaintainable contortions to avoid using a widely known bash feature that would make the code far more legible. Eg bash arrays.
I don’t disagree; often there is no nice way in posix sh to do something that a bash array could. But I’ve found that once you begin to feel a desire for any kind of data structure in your program, it’s probably time to switch to a more robust programming language than (ba)sh.
I generally agree. When I look back at my professional career (16 years and counting), I’ve spent a remarkable amount of it essentially writing bash scripts. Which is surprise.
Most recently, for containers which can’t have a larger interpreter inside them for space reasons, ruling out Python etc
In particular, if you did go and change the interpret to /bin/sh, you also need to change the second line, since (at least) pipefail is a bashism.
Thanks. I have little experience with non-BASH shells and I didn’t want to assume that this would work in on a non-BASH script. Good to learn that it doesn’t.
Fair point. I am trying to target the 100%. It feels to me that’s it’s best to program against bash than risk in failing in a corner case on a shell which I never tested my script against.
For 99% of shell scripts its a non issue. A lot of people use things like arrays in bash when simple quoted strings work on every shell. Writing portable across multiple bourne shell interpreters is easier than you think it is. Most of my scripts run under any of ksh/zsh/bash/sh with zero changes. And testing wise… there isn’t much to test tbh, outside of -o pipefail only works for zsh and bash. Generally the -u catches those cases anyway.
What kind of corner case are you thinking of? Also as yule notes, run shellcheck on your scripts when writing them. That will reduce your edge cases significantly.
The annoying thing is that bash doesn’t disable bash extensions when invoked as sh. A couple weeks ago, I had to debug why a script written by a colleague didn’t work. It turned out that he used &> to redirect both stderr and stdout to a file. I didn’t even know this existed, and he didn’t even know it wasn’t portable. The nasty part is that this doesn’t cause an error. Instead, foo &> bar is parsed as foo & (not sure what happens to the > bar bit; file is created but not hooked up to stdout of foo), which causes race conditions with whatever comes next relying on foo having already completed.
Of course, with #! /usr/bin/env bash in the script, this wouldn’t have been a problem on my machine, but I maintain that it would be more productive if bash disabled all of its non-POSIX extensions when invoked as sh. More commonly, you see people use [[ ... ]] instead of [ ... ]. I’m still not even sure how [[ is different from [.
I’m still not even sure how [[ is different from [.
I vaguely know that and I switch out Bash for Python as soon as even simple string or array manipulation kicks in. My reasoning is that programs always become more complex over time. You start with simple string manipulation and soon end up with edge case handling and special cases. Best to switch to a language like Python which supports that.
I think (among other things) [[ disables word splitting, i.e. you can use variables without needing to quote them. At least that’s how it works in ksh.
Are they though? POSIX shell is so extremely limited that most scripts end up with some bashisms in them, if only to send a string to a program’s stdin. foo <<< "$var" is so much nicer than printf "%s" "$var" | foo (and no, you can’t use echo because POSIX is so vague that you can’t correctly tell echo to treat an argument as a string and not an option in a portable way)
I’ve made the mistake of thinking my script is POSIX, using #!/bin/sh, and then hearing from Ubuntu users that it doesn’t work in Dash. (I’ve also had problems where my script actually is POSX, but dash is so riddled with bugs that it still didn’t work on Debian-based systems.)
I respectfully disagree, using the /usr/bin/env method requires bash, so you can use all the bashisms you want, and require users of your scripts to have bash installed.
For 95% (wild-ass guess) of Unix users, this is the case. For the rest, if you’ve stated the requirements clearly, they’ll either accept the requirement or form a large enough crowd with pitchforks to force you to change.
*BSD, Solaris, and Android don’t have bash installed by default and it isn’t part of their base systems so it isn’t available until /usr/local is mounted and can’t be used anywhere in early boot. Ubuntu uses dash as /bin/sh, but does install bash by default. Embedded Linux systems typically use busybox or similar and so don’t have bash.
95% of Unix users probably covers all Android users, so that’s not a particularly useful metric. The interesting metric is the set of people you expect to use your script. If it’s GNU/Linux users, that’s fine. If it’s people running large desktop / server *NIX systems (macOS, *BSD, Linux), bash is probably available easily but it may not be installed on any given system so you’ve added a dependency.
By all means, if you depend on bashism then use bash. In general, I’ve found that my shell scripts fall into two categories:
Simple things, where the POSIX shell is completely adequate and so there’s no reason to use anything other than /bin/sh.
Complex things where I actually want a rich scripting language. Bash is a pretty terrible scripting language, but it’s a much lighter weight dependency than something like Python so I’ll often use bash for these things.
I use bash as my interactive shell on FreeBSD, so I don’t really mind having it as a dependency for things, but I consider it somewhat impolite to add dependencies in things I distribute when I don’t really need them.
Why #!/usr/bin/env bash as opposed to #!/bin/bash tho? I’ve seen that done for ruby scripts, and I figured it’s because maybe ruby is in /usr/local/bin, but isn’t bash always in /bin/bash?
in the BSDs it should be under /usr/local/bin or /usr/pkgs/bin. MacOS ships an old version of bash and one might want to install a newer version from homebrew, which goes under /usr/local/bin
Using env bash let me both use a newer homebrew bash on Macs, and keep the script working on Linux machines, and possibly even Windows via WSL. On Macs the newer homebrew bash was needed for a switch statement, if memory serves.
For ruby, chruby lets one install multiple versions on one system, and flip between them.
Look, bash is not a fancier version of make. Don’t assume that just because a command exits non-zero, that means something has gone wrong. Yes, it is often the case that many programs will exit non-zero to indicate that some kind of fatal error has occurred but this is far from universal. In fact, the oldest lowest-level Unix utilities use exit codes to communicate neutral status, not that the program has failed.
As a trivial example, the extremely useful grep -q will produce no output and will exit zero when it finds a match, even if an error occurred. Or maybe you write a shell function which should result in a boolean value. Yes, you can set a global variable in the function but that’s extremely gross. Easier to just return an int.
I write quite a lot of bash. -e is generally a good idea for stand-alone scripts but any time I find myself writing a non-trivial script, -uo pipefail ends up being more trouble than its worth.
Completely disagree. set -e is more trouble than it is worth. Most of the behavior is unintuitive and completely breaks on anything that doesn’t exit 0. Plenty of things exist non-zero without it being an error. The bash you end up writing to contort your logic to meet set -e‘s whims ends up being far less readable than straight forward scripts. Besides if you’re really concerned about errors, handle them, perhaps you can recover and not just bail the entire script.
Besides if you’re really concerned about errors, handle them, perhaps you can recover and not just bail the entire script.
I am not the only one writing the scripts. Others are too. And if they don’t write error handling, set -e is a cheap way of preventing that. Again till now, set -e has worked well in my case. But I added a more detailed link to the blogpost around caveats.
I am not the only one writing the scripts. Others are too. And if they don’t write error handling
One of the many fantastic reasons to have code reviews.
Once again, not all exit codes that are non-zero mean failure, and even if they do mean failure, they could be different failures that could indicate a need to retry, or a way to possibly recover or provide a more helpful error message than just dying. Sure you can mitigate the last one with a trap, but even then, it’s still more general than writing a real error message for the issue encountered.
Take ls‘s man page for instance (not that I advocate for using ls in scripts, typically you’d want find).
Exit status:
0 if OK,
1 if minor problems (e.g., cannot access subdirectory),
2 if serious trouble (e.g., cannot access command-line argument).
and it’s certainly not the only command to have different exit statuses for different situations. In my experience, set -e is more trouble than it’s worth, but as always, ymmv.
Ubuntu (and Debian?) makes a distinction between the system shell and a login shell. The system shell is dash because it’s extremely lightweight and is well-suited for automated tasks. Sometimes this bites people because it’s the default shell for cron. But as far as I know, bash has always been the default login shell. (Meaning, the shell you get when you log in on the system console.)
The options discussed in this article are helpful, but I’d be super cautious about “always”.
There are times when having pipefail set (as a for-instance) can be disastrous. I say educating people as to the available tools and suggesting that they probably make sense more often than not is the better path than absolutes in contexts like this.
As an aside, the first thing you end up with is default values, and the syntax:
PARAM_ONE=${1:-$DEFAULT_PARAM_ONE}
does the trick.
I think only Ruby comes close to having a nice modern language with easy shelling out without libraries installed. Perl is painful (positional arguments, aaagh), and Python requires you to pip up to get stuff but vanilla Ruby gives you backticks.
Yeah, I don’t know why this gets so many upvotes since the “article” is only a distilled version of the linked unofficial-bash-strict-mode with a clickbaity title (as you already said).
I prefer #!/bin/bash shbang myself so that scripts don’t run on macOS. If something needs to be portable I’d rather rewrite it in Python rather than bother with an ancient, buggy bash and the BSD versions of shell utilities. Plus, my coworkers can maintain Python.
Suggest -a11y
I also use
set -equite a lot. There is a very large and useful BashFAQ describing ways thatset -edoesn’t do what authors might expect: https://mywiki.wooledge.org/BashFAQ/105If you mean anything other than “accessibility”, could you please define what you’re talking about, or provide a link to a write-up of it? The abbreviation “a11y” is so universally used for “accessibility” (in the same way “i18n” is “internationalization”) that it’s effectively impossible to perform a search for other potential meanings.
The article was originally tagged
a11y, and:is as far as I know common syntax to instruct other lobsters to make a
suggestion to remove thea11ytag.You can see the moderation log here: https://lobste.rs/moderations?utf8=%E2%9C%93&moderator=%28Users%29&what%5Bstories%5D=stories
Edit: Seeing as the automatic moderation event took place 5 minutes after my comment, it appears that my understanding of it being common syntax is true.
Here I was thinking
-a11ywas a Bash option.Same here :)
OK. I didn’t realize this was trying to suggest something about the metadata of the post; the context of the rest of the comment made it seem as if you were suggesting some strange set of bash flags that I’d never heard of.
My bad. I don’t know how I ended up adding -a11y
This is exactly why I think these type of abbreviations are i5c [1]. It’s just a way to obfuscate words and confuse people. (Sorry for the overly salty reaction, but it’s a pet peeve.)
[1] idiotic
Same here, I would love to learn about this as well.
Thanks for that link, I never suspected there were so many undefined corner cases with
set -eYes, great link. I think especially example 5 demonstrates that
set -emight be more trouble than it’s worth.I wonder how many security issues are lurking underneath these inconsistent and weird semantics. This is the kind of muddled thinking that used to cause (and probably still causes) problems in PHP scripts.
Thanks. This is great learning for me. Fortunately, I switch to Python before such complications usually arise but you are right, one should not blindly add “set -e” to an existing BASH script they didn’t write.
I find all the admonishments to use POSIX shell rather than bash befuddling. Why would you not want to use Bash language features such as arrays and associative arrays? If your goal was portability across a wide swath of UNIX variants, then I understand that you might want to target POSIX shell, but that is a very unusual goal. In most instances why not leverage a shell language with very useful features?
Because arrays and associative arrays are two of the most broken features in bash. You can use them if you’re an expert, but your coworker who has to maintain the script will likely not become an expert. (I’ve been using shell for 15 years and implementing a shell for 4, and I still avoid them)
I implemented them more cleanly in Oil and some (not all) of the differences / problems are documented here.
https://www.oilshell.org/release/latest/doc/known-differences.html
declare -A arrayanddeclare -a assoc_arrayand seeing what happens.set -e). Not being able to use empty arrays makes them fundamentally brokenI could list more examples, but if you’ve used them even a little bit, you will run into such problems.
In Oil arrays and assoc arrays are usable. And you even have the
reprbuiltin to tell you what the state of the shell is, which is hard in bash.edit: I forgot that one of the first problems I wrote about is related to array subscripts:
Parsing Bash is Undecidable
Also the security problem I rediscovered in 2019 is also related to array subscripts:
https://github.com/oilshell/blog-code/tree/master/crazy-old-bug
A statement as simple as
echo $(( a ))is in theory vulnerable to shell code injection.a[$(echo 42; rm -rf /)].All ksh derived shells including bash have this problem. Note that you don’t even have to use arrays in your program to be vulnerable to this. Simply using arithmetic in a shell that has arrays is enough! Because arithmetic is dynamically parsed an evaluated, and accepts
$(command subs)in the array index expression.https://unix.stackexchange.com/questions/172103/security-implications-of-using-unsanitized-data-in-shell-arithmetic-evaluation/172109#172109
bash is poorly implemented along many dimensions, but I would say arrays are one of the worse areas. But I learned that it got a lot of that bad behavior from ksh, i.e. “bash-isms” are really “ksh-isms”. So the blame for such a bad language to some extent goes back to AT&T and not GNU.
Thanks. Without arrays, how do recommend handling cases where you need to build up a series of arguments to be passed to another command?
I oscillate between:
-O3or-fsanitize=address, this is a reasonable option, and the one I use.The fact that it’s awkward is of course a motivation for Oil :)
Did you mean
set -u? And to be fair, I would consider this more of a case ofset -ubeing broken; it seems that their fix for it was to not trigger (even when unset) on*and@indexes.Yes
set -u. Either way, you can’t use empty arrays and “strict mode” together, which makes arrays virtually useless in those recent version of bash IMO.I have a memory of running into that the first time, and I couldn’t understand what was going on. And I ran into many more times after that, until I just stopped even trying to use arrays.
If you need them that’s a good sign it’s time to rewrite the script in real programming language.
It’s not as “unusual” as you might think though. There are many people running OpenBSD on their work machines, where bash is simply not included in the default install. I could see why you’d want to make your script work without issue when bash isn’t an option. At the very core, a shell script should not do much: cleanup your local mailbox, aggregate multiple RSS feeds or perform a quick backup using rsync, … I have run openbsd on a laptop for some time, and I was delighted to see that all my personnal scripts were still working like a charm (besides calls to
seq(1), this one killed me…). I’ve also written some simple C software, and wrapper scripts for it using POSIX shells only, and I was glad that nobody from the BSD world bothered me with these scripts.Another argument is that POSIX shells is much more simple to use and understand than the myriad of corner-cases and specificities that bash can have. It sure is a strengh to be able to use proper arrays, but bash manpage has 5 times more (48746 words) than sh (10224). Mastering bash as a language is definitely harder than sticking to POSIX, especially when you only write scripts that are < 20 lines.
Even on Linux bash isn’t always included by default, especially on simple server setups/distros such as Alpine Linux.
The
seqthing is annoying, BSD hasjotwith different syntax. I wish they could agree on one or the other. Same withstat.In my experience, the moment my shell script becomes complex enough for a bash array is the moment it gets rewritten in Python.
The only features from bash that have convinced me to require it in recent memory are herestrings (
<<<) and the regex match operator (=~).Same here. Even for the simple scripts since a script would become more complicated over time.
I use systems without bash installed by default. If you’re going to write a complex script that is too awkward to write in POSIX sh, then why not use a better (less quirky) language in the first place? Python for example is probably available in almost as many places as bash now. If you’re writing a shell script, you’re probably aiming for portability anyway, so please write POSIX shell if you can.
I write many bash scripts where portability is not a concern and I think the language can be quite functional and even elegant when you are pulling together different cli tools into a new program.
Ubuntu shaved quite a bit off their (pre-systemd) boot times by just switching /bin/sh from bash to dash (although I can’t find the exact number right now). The increased performance may be a concern (or is at least nice) in some cases (bending over backwards for it would of course be silly).
Certainly if it’s not a concern and you enjoy the language then I don’t see a problem with using bash! My main issue is that I don’t think it’s at all an unusual goal to use POSIX sh for portability.
Because there are plenty of use cases, like installers, where relying on another language isn’t possible.
Why not do the heavy lifting in awk, if your’e in such a restricted environment?
When I talk about writing shell applications, I personally am talking about using bash plus all the standard UNIX tools that come with every UNIX base install.
Awk is one of those tools.
And when I say do the heavy lifting in awk, I mean more or less ignore the shell, and use awk. You get proper arrays, reasonable syntax for writing algorithms, relatively little shelling out, relatively sane string manipulation, and sidestep a bunch of issues around shell variable reexpansion.
And your code is more portable.
I believe that. I haven’t used awk that way and I’ll admit that’s because I don’t know it well enough.
I know I’m asking a lot but might you be able to link to an example of awk being used in this way?
I mostly end up using stupid simple awk invocations like awk ‘{print $3}’ :)
Here’s an example from the FreeBSD tree. Lines 107 through 788 are one big awk script. https://github.com/freebsd/freebsd/blob/master/sys/kern/makesyscalls.sh
It parses the syscall definitions here, and generates C: https://github.com/freebsd/freebsd/blob/master/sys/kern/syscalls.master
Why isn’t it possible and why can’t it be done in POSIX sh?
The OP’s post has two points.
That’s what I was addressing.
To your point, there are customer environments where shipping UNIX systems with languages like Python is prohibited, either because of security constraints, maybe disk usage, etc.
I don’t think it’s really a big issue. Certainly for myself it’s only a minor frustration if I need to install bash for a simple script which gains nothing from using bash. I’ve just seen the
#!/bin/bashshebang far too many times when it’s not necessary, and I think that a lot of the time that’s just because people don’t know the difference. It’s certainly not the end of the world and if bash feels like the right tool for whatever your use case is then I’m not going to argue! It would just be nice if there was a little awareness of other systems and that people would default to#!/bin/shand only consider bash if they find a specific need for it. I imagine that in most cases shell scripts are used there actually is no need.I’m a little obsessed with building my workflow around shell scripts and I have never found a need for bash extensions (YMMV). The other big benefit other than portability as others have suggested is the digestibility of the manpages for, e.g. dash.
And the corollary: The first line of your shell script should be
If you need bashism for something complex that needs to be maintainable, use
#!/usr/bin/env bashas the article suggests, but 99% of ‘bash’ scripts that I’ve seen are POSIX shell scripts and work just fine with any POSIX shell.I disagree, enormously. There are some specific, isolated, and extremely uncommon situations where it’s important you target
sh; the rest of the time, you are constraining yourself for no appreciable benefit.In particular, if you did go and change the interpret to
/bin/sh, you also need to change the second line, since (at least)pipefailis a bashism.I think being constrained to a smaller and simpler set of features is the appreciable benefit.
This applies doubly if somebody is going to have to maintain this code.
One does not necessarily follow the other. I’ve seen horrific apparently-posix-sh scripts with the most unmaintainable contortions to avoid using a widely known bash feature that would make the code far more legible. Eg bash arrays.
I don’t disagree; often there is no nice way in posix sh to do something that a bash array could. But I’ve found that once you begin to feel a desire for any kind of data structure in your program, it’s probably time to switch to a more robust programming language than (ba)sh.
I generally agree. When I look back at my professional career (16 years and counting), I’ve spent a remarkable amount of it essentially writing bash scripts. Which is surprise.
Most recently, for containers which can’t have a larger interpreter inside them for space reasons, ruling out Python etc
Thanks. I have little experience with non-BASH shells and I didn’t want to assume that this would work in on a non-BASH script. Good to learn that it doesn’t.
Good POSIX shell reference here:
http://shellhaters.org/
Fair point. I am trying to target the 100%. It feels to me that’s it’s best to program against bash than risk in failing in a corner case on a shell which I never tested my script against.
For 99% of shell scripts its a non issue. A lot of people use things like arrays in bash when simple quoted strings work on every shell. Writing portable across multiple bourne shell interpreters is easier than you think it is. Most of my scripts run under any of ksh/zsh/bash/sh with zero changes. And testing wise… there isn’t much to test tbh, outside of -o pipefail only works for zsh and bash. Generally the -u catches those cases anyway.
What kind of corner case are you thinking of? Also as yule notes, run shellcheck on your scripts when writing them. That will reduce your edge cases significantly.
The annoying thing is that bash doesn’t disable bash extensions when invoked as
sh. A couple weeks ago, I had to debug why a script written by a colleague didn’t work. It turned out that he used&>to redirect both stderr and stdout to a file. I didn’t even know this existed, and he didn’t even know it wasn’t portable. The nasty part is that this doesn’t cause an error. Instead,foo &> baris parsed asfoo &(not sure what happens to the> barbit; file is created but not hooked up to stdout offoo), which causes race conditions with whatever comes next relying on foo having already completed.Of course, with
#! /usr/bin/env bashin the script, this wouldn’t have been a problem on my machine, but I maintain that it would be more productive if bash disabled all of its non-POSIX extensions when invoked assh. More commonly, you see people use[[ ... ]]instead of[ ... ]. I’m still not even sure how[[is different from[.I vaguely know that and I switch out Bash for Python as soon as even simple string or array manipulation kicks in. My reasoning is that programs always become more complex over time. You start with simple string manipulation and soon end up with edge case handling and special cases. Best to switch to a language like Python which supports that.
I think (among other things) [[ disables word splitting, i.e. you can use variables without needing to quote them. At least that’s how it works in ksh.
Are they though? POSIX shell is so extremely limited that most scripts end up with some bashisms in them, if only to send a string to a program’s stdin.
foo <<< "$var"is so much nicer thanprintf "%s" "$var" | foo(and no, you can’t use echo because POSIX is so vague that you can’t correctly tell echo to treat an argument as a string and not an option in a portable way)I’ve made the mistake of thinking my script is POSIX, using
#!/bin/sh, and then hearing from Ubuntu users that it doesn’t work in Dash. (I’ve also had problems where my script actually is POSX, but dash is so riddled with bugs that it still didn’t work on Debian-based systems.)That’s why it’s a good idea to always lint your scripts with shellcheck. It will catch all bashisms when you set shellbang to /bin/sh
In which case
-o pipefailis undefined, at least according to shellcheck.In any case I find running
shellcheckon shell scripts to be really valuable. You don’t have to follow it blindly, but it has some very good hints.shellcheck is my favorite linter for bash scripts as well. It sometimes feels a bit aggressive but I prefer that as opposed to making a mistake.
I respectfully disagree, using the
/usr/bin/envmethod requires bash, so you can use all the bashisms you want, and require users of your scripts to have bash installed.For 95% (wild-ass guess) of Unix users, this is the case. For the rest, if you’ve stated the requirements clearly, they’ll either accept the requirement or form a large enough crowd with pitchforks to force you to change.
*BSD, Solaris, and Android don’t have bash installed by default and it isn’t part of their base systems so it isn’t available until
/usr/localis mounted and can’t be used anywhere in early boot. Ubuntu uses dash as/bin/sh, but does install bash by default. Embedded Linux systems typically use busybox or similar and so don’t have bash.95% of Unix users probably covers all Android users, so that’s not a particularly useful metric. The interesting metric is the set of people you expect to use your script. If it’s GNU/Linux users, that’s fine. If it’s people running large desktop / server *NIX systems (macOS, *BSD, Linux), bash is probably available easily but it may not be installed on any given system so you’ve added a dependency.
By all means, if you depend on bashism then use bash. In general, I’ve found that my shell scripts fall into two categories:
/bin/sh.I use bash as my interactive shell on FreeBSD, so I don’t really mind having it as a dependency for things, but I consider it somewhat impolite to add dependencies in things I distribute when I don’t really need them.
Why
#!/usr/bin/env bashas opposed to#!/bin/bashtho? I’ve seen that done for ruby scripts, and I figured it’s because mayberubyis in/usr/local/bin, but isn’t bash always in/bin/bash?Not on all systems. e.g. on FreeBSD and OpenBSD it’s /usr/local/bin/bash.
Or Nix and Guix, which have them in some entirely different place (e.g. on Guix is something like
/gnu/store/.../).in the BSDs it should be under /usr/local/bin or /usr/pkgs/bin. MacOS ships an old version of bash and one might want to install a newer version from homebrew, which goes under /usr/local/bin
Using env bash let me both use a newer homebrew bash on Macs, and keep the script working on Linux machines, and possibly even Windows via WSL. On Macs the newer homebrew bash was needed for a switch statement, if memory serves.
For ruby, chruby lets one install multiple versions on one system, and flip between them.
Personally I usually stick to POSIX scripts and I use following template:
set -f is not always desirable but I think it’s good default.
Oh, this again.
Look,
bashis not a fancier version ofmake. Don’t assume that just because a command exits non-zero, that means something has gone wrong. Yes, it is often the case that many programs will exit non-zero to indicate that some kind of fatal error has occurred but this is far from universal. In fact, the oldest lowest-level Unix utilities use exit codes to communicate neutral status, not that the program has failed.As a trivial example, the extremely useful
grep -qwill produce no output and will exit zero when it finds a match, even if an error occurred. Or maybe you write a shell function which should result in a boolean value. Yes, you can set a global variable in the function but that’s extremely gross. Easier to just return an int.I write quite a lot of bash.
-eis generally a good idea for stand-alone scripts but any time I find myself writing a non-trivial script,-uo pipefailends up being more trouble than its worth.Completely disagree.
set -eis more trouble than it is worth. Most of the behavior is unintuitive and completely breaks on anything that doesn’t exit 0. Plenty of things exist non-zero without it being an error. The bash you end up writing to contort your logic to meetset -e‘s whims ends up being far less readable than straight forward scripts. Besides if you’re really concerned about errors, handle them, perhaps you can recover and not just bail the entire script.I am not the only one writing the scripts. Others are too. And if they don’t write error handling,
set -eis a cheap way of preventing that. Again till now,set -ehas worked well in my case. But I added a more detailed link to the blogpost around caveats.One of the many fantastic reasons to have code reviews.
Once again, not all exit codes that are non-zero mean failure, and even if they do mean failure, they could be different failures that could indicate a need to retry, or a way to possibly recover or provide a more helpful error message than just dying. Sure you can mitigate the last one with a trap, but even then, it’s still more general than writing a real error message for the issue encountered.
Take
ls‘s man page for instance (not that I advocate for usinglsin scripts, typically you’d wantfind).and it’s certainly not the only command to have different exit statuses for different situations. In my experience,
set -eis more trouble than it’s worth, but as always, ymmv.Unfortunately,
set -uused to be broken until very recently (Bash 4.4).So I’m saying don’t use it unless you control the Bash version, or use a feature test:
How to do things safely in bash § How to begin
-o pipefailis not POSIX and fails on ubunut’s default shell which is dash, unfortunately.I also tend to add -x for debugging, super useful.
Ubuntu (and Debian?) makes a distinction between the system shell and a login shell. The system shell is dash because it’s extremely lightweight and is well-suited for automated tasks. Sometimes this bites people because it’s the default shell for cron. But as far as I know, bash has always been the default login shell. (Meaning, the shell you get when you log in on the system console.)
The third statement:
cd "$(dirname "$0")". So many scripts assume the user runs them from the script directory.(not sure if it works on Mac)
The options discussed in this article are helpful, but I’d be super cautious about “always”.
There are times when having pipefail set (as a for-instance) can be disastrous. I say educating people as to the available tools and suggesting that they probably make sense more often than not is the better path than absolutes in contexts like this.
I agree but I think if you are writing shell scripts for automation or testing or any other devops situation, it is best to add these two lines.
Disagree.
I’ve encountered situations where I’m stringing together a very large pipeline and there are components in said pipeline that are entirely optional.
In those cases, pipefail isn’t what you want.
Again, we’re quibbling about the nature of the word ‘always’ here.
I agree that for many use cases this is a good best practice to adopt.
I still usually turn on pipefail at the start of the script and disable it for the statements that need it.
That makes a lot of sense actually, although I still give “always” the hairy eyeball :)
There’s only one statement that’s ALWAYS true in computer science - “your code will have bugs” :)
As an aside, the first thing you end up with is default values, and the syntax:
does the trick.
I think only Ruby comes close to having a nice modern language with easy shelling out without libraries installed. Perl is painful (positional arguments, aaagh), and Python requires you to pip up to get stuff but vanilla Ruby gives you backticks.
Decent content, but declining to upvote due to clickbait title.
Yeah, I don’t know why this gets so many upvotes since the “article” is only a distilled version of the linked unofficial-bash-strict-mode with a clickbaity title (as you already said).
I prefer
#!/bin/bashshbang myself so that scripts don’t run on macOS. If something needs to be portable I’d rather rewrite it in Python rather than bother with an ancient, buggy bash and the BSD versions of shell utilities. Plus, my coworkers can maintain Python.For larger scripts, I find this useful in the preamble, so you get stacktraces:
Thanks. TIL, I will try it out :)