My experience with this from a decade or so ago is “write posix shell, or solaris users will yell at you”.
None of these arguments are really very compelling to me. Part of this is because almost all of the examples of “when should I use shell” in the article are programs you write for yourself, not things you distribute for other people to use. Throwaway code is great, and I love it, but the thing about it is that portability is completely irrelevant in that context.
But it’s also because people often conflate “portability” to mean two distinct things:
A) this is portable across operating systems or hardware architectures
vs
B) this is portable across various implementations of an existing standard, all of which are themselves already portable across OSes and machines
Obviously A is very important. B is … kind of nice to have sometimes? But more often than not, it doesn’t matter for scripts. (It is much more important for libraries.) 99% of the time there is no reason your prospective alpine user can’t just install bash if they really want to run your script; they just like to complain about having to do so.
(That said, obviously don’t write non-posix scripts and use #!/bin/sh as your shebang; if you’re going to rely on a specific implementation, don’t be misleading about it. When most people legitimately complain about use of bash features, they’re really complaining about this.)
this all feels pretty valid. my argument mostly centers around learning & using 1 dialect of shell instead of several, just in case portability turns out to be important later. and sometimes those scripts you write for yourself worm their way on to other people’s machines!! it can be a very randomly useful habit. but i can see how that’s not compelling to everyone!!
For what it’s worth, I enjoyed reading the article and appreciated the perspective even if I don’t agree with it for what I do. It might be a stronger argument if you specifically mention the “even if you don’t need portability today, you might need it later” angle in the article; everyone’s context is different, and people can judge for themselves how likely that eventuality would be.
I guess the most-sensible argument for using POSIX even for private scripts is kinda like the argument for not extensively modding your shell environment–you’ll be used to working within the constraints when you’re suddenly forced to.
But yeah, I agree that a very small slice of Shell needs that level of portability.
One thing I really like about Nix is being able to push the bootstrapping problem off to that level, but then having pretty firm control of the interpreter and dependencies. I sketched an outline of this in https://www.t-ravis.com/post/shell/no_look_no_leap_shell_with_nix/
great analogy, working within those small constraints leaves you feeling confident and comfortable anywhere you go. and if some random person copies a few of my personal shell scripts, it’s nice knowing that they’re likely to work.
But more often than not, it doesn’t matter for scripts. (It is much more important for libraries.) 99% of the time there is no reason your prospective alpine user can’t just install bash if they really want to run your script
The main reason, in my mind, for using shell scripting is that the user doesn’t have to install anything. If you’re happy to add a dependency that needs installing, Lua is far nicer to use and is smaller than bash, Python is bigger but has a far more batteries-included standard library.
All of this matters more once you start thinking about containerised deployments. Most uses of Alpine that I’ve encountered are not people running Alpine as their OS, they’re people using Alpine to make tiny container images. Adding bash to the Apline base image makes it 20% bigger (base image is 5 MiB, bash is a bit over 1 MiB). Is the value that you’re getting from bash really 20% of the total value for a minimal *NIX userland?
I think for a lot of the programs that you should be writing in Bash, rewriting them in Python or whatever would be a net loss of readability. To grab an example off of my current project, this would be a lot uglier as Python:
[[ -f .env ]] && echo "Using .env file" && source .env
go run ./funcs/almanack-api "$@"
It depends upon your familiarity with the language. I personally find sh to be an impenetrable mess of characters (with perl a close second), and what I think should be a simple task, like redirecting stderr to a pipe, can be done, but with a syntax that makes me pine for INTERCAL.
Building bash takes a non-trivial amount of time, and it’s unjustified for just a single task which can also be completed in POSIX sh. See the bash build dependency in the Linux kernel itself a few minor version ago.
Speaking of navigating the Open Group website, https://shellhaters.org is a friendly jumping-off point into the shell-specific parts of the POSIX specification. The talk associated with this website (“The Shell Hater’s Handbook” by Ryan Tomayko) can be watched here: https://www.youtube.com/watch?v=olH-9b3VJfs
I have personally found AWK to be a surprisingly good alternative to shells for larger than average scripts.
Why?
Portability (AWK is a part of POSIX)
Very clean syntax
Powerful associative arrays
Super-powerful string manipulations facilities
Easy shell interoperability
As an example here is a simplistic command runner [1] I’ve developed in AWK. As you can check [2] it runs unchanged under Linux/macOS/Win (via Git Bash).
Okay more seriously, I think whether you should stick with POSIX really depends on how important portability is to you. Because there are a lot of cases where “shell+arrays” is the right tool for the job, if you know you don’t have to run it on another OS!
Also a tip:
make a take-my-estrogen alarm clock (perhaps your laptop would beep)
I’ve done similar things and this doesn’t work because it assumes you’re at the computer when the alarm goes off. I find it easier to open a textfile with the note, so you see it as soon as you’re back.
I generally disagree with the preference for writing in POSIX shell, and think that shell portability should not be a common concern in the majority of places shell is used. If you’re writing scripts that you know will be distributed widely, of course use POSIX shell. If you know out the gate that something you’re writing has to be portable across many different UNIXen, of course use POSIX shell.
I do suspect, however, that the vast majority of shell is not distributed, only need to target one platform, and in these cases, I’d recommend using whatever shell your target platform has installed (or install your favorite shell; I’m not your boss).
Shell portability is such a thorny topic that people like to pretend is less of a problem than it actually is. Yes, POSIX shell is generally portable. However, the reality is that to do anything meaningful in shell, you have to call out to other utilities, and these are likely the real cause of portability woes.
Basic problems are GNU coreutils vs the basic utilities provided on any other UNIX (I’m not gonna say BSD here, because even amongst BSDs, basic utilities have grown enough differing features). But the problem spirals depending on what your shell script is going to do.
Services? Get ready to deal with rcctl, service, systemctl, svcadm and more. Networking? How many different incompatible incantations of ifconfig are there? It’s probably time to switch to ip on Linux. What about ipadm? Firewalls? lol pf, FreeBSD’s ancient pf, iptables, nftables, ipfw, etc. My point here is that portability has way more to do with the utilities that you’re calling than just the shell language itself. Also, re: utilities, even if you use the same utilities, since you can’t declare dependencies in shell, I hope that your targets have compatible versions and only use the common set of features amongst your installbase :)
This all comes from frustration trying to use “”“compatible””” shell scripts across Solaris and BSDs and how POSIX shell has been the least of my problems in making shell scripts work in these environments. I think in the majority of cases, it’s more work than it is worth, and so I think it’s best to build using the best tools for you and your team that are available on the platform you’re targeting.
I will say, I do like the idea of using the limitations of POSIX shell as a litmus test for when you need to write something in a real language. As a shell-hater, I do believe that bash and zsh (and even ksh93) affords too many conveniences that make people resist switching to a proper language.
I also agree that many things are just so much faster to implement in shell than other languages, but it’s hard for me to reconcile the trade-off of ease of development vs how error-prone shell can be. Shellcheck does help mitigate a ton though, as was mentioned in the article.
I do still write shell, because it’s so much faster to implement, but I do try and keep it to just automating running a small set of commands and I’d never distribute anything written in shell as a tool/product.
Yeah, writing portable shellscripts is not at all dissimilar from trying to write e.g. Python 2/3 polyglots. We don’t think of the former as polyglots, probably because the common subset is standardized somewhere, but in practice it feels similar to me. Similar to Python 2/3 polyglots, I only write these for the most minimal of bootstrap tasks to get to a better suited language (and that may be bash, sometimes).
POSIX also defines a set of core utilities and how their interface works, so you can use them in POSIX sh without a problem (think cut, sed, awk, find, printf, xargs, mv, cp). Sure, that doesn’t help if you want to do system-specific things like services or networking, since there’s not even a pretence of compatibility between those systems. But that doesn’t necessarily justify adding more dependencies such as Bash just because you’re doing system-specific stuff anyway. POSIX sh is likely to be available everywhere, and fewer dependencies are better than more.
POSIX also defines a set of core utilities and how their interface works, so you can use them in POSIX sh without a problem (think cut, sed, awk, find, printf, xargs, mv, cp)
These are sufficient for a very small scope of tools ime. File manipulations, etc. With any of them, it’s very trivial to accidentally opt-in to features specific to the environment that you’re working in. Moreover, the question gets raised — what files are you manipulating? How system-agnostic are they? If you’re doing a bunch of file manipulations on files specific to RHEL 8, how has avoiding dependencies on bash or GNU coreutils helped you?
fewer dependencies are better than more.
This is part of the problem with writing stuff in (any) shell though. What are your dependencies? In many languages, you at least have some type of manifest that lists the dependencies and the versions. In C, at least you can tease stuff out from cmake/autotools/whatever or in the worst case, by the header files/linker.
In a shell language, any single line of code can introduce a new dependency. There’s no real way to verify in a shell script what dependencies there are, what versions of those dependencies there are, or even what flavor of those dependencies there are.
I don’t understand why people view shell scripts as being free of dependencies, when often the dependencies are just more subtle. It becomes way more obvious when you try some of these shell script tools on a BSD or a commercial UNIX how pernicious this problem is.
Yes, and this is a great use-case for Nix. You can package up a shellscript together with its dependencies (including the precise version of coreutils etc. that it uses) and it’s just like packaging up any other dependencies.
If you target POSIX, then you can use POSIX shell and POSIX utilities without worrying about them as dependencies. That works across BSD and commercial UNIXes too. Your project might need more tools than those, but you can still rely on the POSIX utilities being available with at least the standardised flags.
I feel like this is a great recommendation and a general good practice for writing shell that could be run anywhere, on any shell.
I’ve written a ton of shell in the last couple of years and it’s running mostly in containers. It’s primarily bash, although I often started new stuff with posix shell and quickly fell into bash mode when I wanted something like arrays.
The biggest problem I’ve hit is writing some stuff that ended up using newer bash features not available in the version of Bash that Apple ships in macOS. I install and use the latest Bash from Homebrew so I’ve always kept up.
POSIX shell is the language I use most frequently. As well as shellcheck my main reference is man-pages-posix; these are the manuals of standard utilities converted from POSIX to manpages (there isn’t a page for sh syntax itself though). This ensures that, as well as only using portable syntax, I only use portable flags and tools. (I’ve mentioned this before but many people seem to have not come across it). The RATIONALE sections of many pages are also interesting, explaining the historical reasons for the standardised syntax and pitfalls to avoid.
Over at KISS linux’s community we are continuing Dylan Araps’ work with a package manager and set of tools written entirely in POSIX shell.
I wish this had existed a few weeks ago, as it would have helped me make my points better :)
At $DAYJOB, we had an instance where we needed some automation that could run on developers’ workstations. That could be Mac or various Linuxes. So I wrote a lovingly handcrafted 130-ish line shell script that used only POSIX shell. It was actually fun to have a knowledge transfer session later about how it’s different than Bash, and about why I used printf instead of echo, how readonly works, etc.
printf is severely underrated. i see people trying to use echo to print json constantly in shell, when i show them printf their world shifts dramatically.
echo “{\”foo\”:\”$bar\”}”
vs
printf '{“foo”:”%s”}' “$bar”
Thank you for this, this is actually super useful to me. ofc I know printf from “real” programming languages and I knew that shell had printf but I never really connected it as being useful there too
I believe shell can be so much better. I’m not convinced that needing arrays is a sign that shell doesn’t fit your problem anyway. Shell is about interacting with the operating system and gluing together processes with ease. I’m hopeful that www.oilshell.org will succeed and raise our expectations for shell programming.
Sorta? A lot of programs will interpret the value in --foo 1 2 3 as three values separated by spaces, but other programs interpret it as one value which contains spaces. There’s no easy way to indicate a string must be interpreted as an array, which also means things like no multidimensional arrays.
Yeah. My experience with shell programming mostly falls in the, “only if nothing else is available,” mental category. It is tedious doing shell-like things in Python but much more readable.
I’m interested in feedback on OSH (https://www.oilshell.org/) from POSIX shell users … It should run EVERYTHING in your scripts – if not, that’s a bug! Here a couple reasons I thought of that you might use it:
(1) More precise error messages
$ osh -c '[ x y ]'
[ x y ]
^
[ -c flag ]:1: (test) Expected unary operator, got 'x' (2 args)
versus
$ dash -c '[ x y ]'
dash: 1: [: x: unexpected operator
In a long script, I think the more precise errors are good. Although one problem is that they could also be interleaved in concurrent output, so we might want an option to force 1 line errors. Feedback is appreciated!
The infrastructure is there to make all errors precise, with column info, and even seeing past multiple levels of parsing. But we need to scrub them to point at the right locations, and could use help. It should be pretty easy/fun for someone who knows Python/MyPy, and perhaps has written a toy tree interpreter (e.g. Lisp or similar).
(2) More detailed tracing:
$ dash -x -c 'x=$(date +%x; echo hi)'
+ date +%x
+ echo hi
+ x=03/12/2023
hi
It deals with newlines in a less misleading way, shows PIDs for command subs, etc.
$ osh -x -c 'x=$(date +%x; echo hi)'
+ 18053 date '+%x'
+ 18053 echo hi
+ x=$'03/12/2023\nhi'
More detailed tracing that tells you how many processes were started, and the exit codes:
$ osh -o xtrace_rich -x -c 'x=$(date +%x; echo hi)'
| command sub 18217
| 18217 command 18218: date '+%x'
; 18217 process 18218: status 0
. 18217 builtin echo hi
; process 18217: status 0
+ x=$'03/12/2023\nhi'
You can also use shopt --set xtrace_rich in the program instead of -o.
I stole a trick from old-school init.d and break my shell scripts into very tiny chunks that are numbered. 00-check-requirements.sh, 01-install-build-tools.sh, 02-setup-database.sh. I can test each one individually and prevent weird interactions between chunks of shell that are doing different things. Shared functions and environment variables can go in a common file. Then I can run them all easily in order from a parent script.
Another simple thing I do is “set -xeo pipefail” at the top of every single file. Prevents the majority of subtle errors.
Finally I use shellcheck obsessively, but that’s mentioned in the article.
I love the style this is written (and laid out) in. It reminds me of classic textfiles, but also of the “ed is the STANDARD. Text editor.” rant. All snail posix shell!
If you’re targeting multiple unixlikes, maybe this is an option. But if you’re not gonna actually test those scripts in CI, there are so many ways even a POSIX shell script can break, due to how they are effectively ‘stringy’ typed. And if you’re running on CI, you might as well just install bash and enjoy the improvements it has.
I find it odd to advice people to strive for compatibility without mentioning CI. At the end of the day, standards are great, but not sufficient for ensuring that it actually works.
if someone is going to learn a shell dialect for CI, it may as well be posix shell. it’s nearly as powerful as bash, and offers a bunch of benefits. once you learn the dialect, it’s really not that bad. i don’t miss any bash features, and find myself happy with the smaller world of posix.
if that doesn’t work for you, by all means, go ahead and use bash.
I mean that the interface between shell and the system is all based on strings: filenames and arguments for executables. If I gave you a script for FreeBSD and asked you if it would work on Debian 9, it would be hard to tell, since there is no type system or anything that enables the machine to check whether the arguments potentially passed are actually available.
I use Linux and Windows and MacOS regularly. Every one of them has bash. Even with every system having the same shell, making a script portable is difficult, because the shell is reaching out to many different things to do its work, and they’re distributed and versioned differently in different environments. The difference between [ and [[ ranks far, far below “every system has a different tar”. tar is not a posix utility, but admin is? https://pubs.opengroup.org/onlinepubs/9699919799/utilities/admin.html
I often find it easier to write Python using only the standard library, because with a Python install, you have just one version number to worry about. I know my oldest Python is 3.8, so that’s what I target and it’s much easier to make portable than trying to figure out how to do it in a shell script.
Yes, SCCS is the version control system defined by POSIX, which is the reason why admin is included in POSIX. I agree that unstandardised tar is a problem, but the POSIX answer is pax.
My whole argument is that the standard requires things that are so antiquated that they are irrelevant, so I’m not sure why telling me that it’s in the standard is supposed to convince me of anything, anything at all. I know that it’s in the standard. That’s my whole point. The fact that it’s in the standard is what makes the standard bad! Meanwhile, tar, an altogether ubiquitous utility that everyone knows is not in the standard. It is long past time to either evolve this standard or let it go.
macOS has bash, but the installed bash is the last GPLv2 version, which is around 15 years old. If you’re using bashisms, there’s a good chance that they won’t work there.
my entire point is that portability extends beyond the shell and the posix utilities standard is old and that every shell script portability concern I’ve ever had has not been within the shell itself
I’ve recently had to fix a shell script that assumed GNU extensions for sed and awk, so I have some sympathy there, but my experience is that most *NIX platforms support a fairly large overlapping set of extensions to the core UNIX utilities, whereas bash extensions are rarely supported anywhere other than bash.
the very first thing I said is that I use Windows regularly. This is kinda my entire point: that portable is not actually meaningful when the people saying it are saying “well, it’s portable for me”.
one of the many reasons i don’t use bash is because there are so many subtle differences between bash implementations, as you pointed out. bash on macos is not bash on linux, and the coreutils are all different.
If you really need to be portable, perl might be best. If you need to be portable and available everywhere out of the box…uh I don’t think there’s anything remotely cross platform that’s on windows and Linux by default (unless you count wsl2, which I don’t).
for many common tasks, shell is absolutely faster. i wrote some short examples.
download a file:
# ruby
require 'net/http'
Net::HTTP.start("example.com") do |http|
resp = http.get("/whatever.txt")
open("whatever.txt", "wb") do |file|
file.write(resp.body)
end
end
# shell
curl -O https://example.com/whatever.txt
replace the string “cats” with “dogs” in all files ending with .txt:
# ruby
file_names = Dir.glob("*.txt")
file_names.each do |file_name|
text = File.read(file_name)
new_contents = text.gsub(/cats/, "dogs")
File.open(file_name, "w") {|file| file.puts new_contents }
end
# shell
for i in *.txt; do
sed 's/cats/dogs/g' "$i" > "$i.tmp"
mv "$i.tmp" "$i"
done
Well, not sure if using curl counts as POSIX shell, if so there’s always
`curl -O https://example.com/whatever.txt`
In ruby.
Although I’m a bit partial to using open3 instead of backticks. While I wouldn’t extensively code golf your second example, I gotta say, if I was doing what you’d asked in a quick-but-perhaps-not-as-readable-version, it might look more like:
Dir.glob("*.txt").each do |file|
File.write(file, File.read(file).gsub(/cats/, "dogs"))
end
curl doesn’t count as posix shell, i was just using it as an example of the conciseness of shell in general. i don’t think that using shelling out of your programming language counts ;) otherwise any language would be an adequate shell alternative!
half the power is being able to just whip shell up in your terminal on-the-fly. shell will also be available in many places ruby is not!
i do like the conciseness of ruby. ever since i read why’s ruby guide, the language has held a special place in my heart. have you seen crystal?
Oh absolutely, and I reach for shell often, just disagreed with the original thought about shell vs. any other language
I’ve been watching crystal and tried it for a few things, but isn’t worth paying the innovation token at $job when we have Go (a poor replacement for shell, imo). But I think for a shell replacement to be feasible it for sure needs to be interpreted
It’s so much worse than @j3s implies. Not only is -i not POSIX, -i on some platforms requires the suffix to use for the backup file, on others it’s optional. The fact that it’s optional makes it spectacularly dangerous because it’s easy to accidentally have it consume an argument. If you must use -i, at least use it with a suffix.
you can take <() from my cold, dead hands
Linky to spare folks a punctutation search: that’s Bash’s syntax for process substitution. (I hope I got that right, Forty-Bot?)
Yes
Thanks.
My experience with this from a decade or so ago is “write posix shell, or solaris users will yell at you”.
None of these arguments are really very compelling to me. Part of this is because almost all of the examples of “when should I use shell” in the article are programs you write for yourself, not things you distribute for other people to use. Throwaway code is great, and I love it, but the thing about it is that portability is completely irrelevant in that context.
But it’s also because people often conflate “portability” to mean two distinct things:
A) this is portable across operating systems or hardware architectures vs B) this is portable across various implementations of an existing standard, all of which are themselves already portable across OSes and machines
Obviously A is very important. B is … kind of nice to have sometimes? But more often than not, it doesn’t matter for scripts. (It is much more important for libraries.) 99% of the time there is no reason your prospective alpine user can’t just install bash if they really want to run your script; they just like to complain about having to do so.
(That said, obviously don’t write non-posix scripts and use
#!/bin/shas your shebang; if you’re going to rely on a specific implementation, don’t be misleading about it. When most people legitimately complain about use of bash features, they’re really complaining about this.)this all feels pretty valid. my argument mostly centers around learning & using 1 dialect of shell instead of several, just in case portability turns out to be important later. and sometimes those scripts you write for yourself worm their way on to other people’s machines!! it can be a very randomly useful habit. but i can see how that’s not compelling to everyone!!
For what it’s worth, I enjoyed reading the article and appreciated the perspective even if I don’t agree with it for what I do. It might be a stronger argument if you specifically mention the “even if you don’t need portability today, you might need it later” angle in the article; everyone’s context is different, and people can judge for themselves how likely that eventuality would be.
thx for the advice! i’ll think about it.
I guess the most-sensible argument for using POSIX even for private scripts is kinda like the argument for not extensively modding your shell environment–you’ll be used to working within the constraints when you’re suddenly forced to.
But yeah, I agree that a very small slice of Shell needs that level of portability.
One thing I really like about Nix is being able to push the bootstrapping problem off to that level, but then having pretty firm control of the interpreter and dependencies. I sketched an outline of this in https://www.t-ravis.com/post/shell/no_look_no_leap_shell_with_nix/
great analogy, working within those small constraints leaves you feeling confident and comfortable anywhere you go. and if some random person copies a few of my personal shell scripts, it’s nice knowing that they’re likely to work.
The main reason, in my mind, for using shell scripting is that the user doesn’t have to install anything. If you’re happy to add a dependency that needs installing, Lua is far nicer to use and is smaller than bash, Python is bigger but has a far more batteries-included standard library.
All of this matters more once you start thinking about containerised deployments. Most uses of Alpine that I’ve encountered are not people running Alpine as their OS, they’re people using Alpine to make tiny container images. Adding bash to the Apline base image makes it 20% bigger (base image is 5 MiB, bash is a bit over 1 MiB). Is the value that you’re getting from bash really 20% of the total value for a minimal *NIX userland?
I think for a lot of the programs that you should be writing in Bash, rewriting them in Python or whatever would be a net loss of readability. To grab an example off of my current project, this would be a lot uglier as Python:
It depends upon your familiarity with the language. I personally find
shto be an impenetrable mess of characters (withperla close second), and what I think should be a simple task, like redirectingstderrto a pipe, can be done, but with a syntax that makes me pine for INTERCAL.Haha, I do hate the redirect syntax, true.
Building bash takes a non-trivial amount of time, and it’s unjustified for just a single task which can also be completed in POSIX sh. See the bash build dependency in the Linux kernel itself a few minor version ago.
In addition to the resources at the bottom, I learned a lot about writing POSIX shell from this post by Rich Felker:
http://www.etalabs.net/sh_tricks.html
and from reading the musl libc configure script:
https://github.com/bminor/musl/blob/master/configure
Previously on… Lobsters: https://lobste.rs/s/uk5esr/rich_s_sh_posix_shell_tricks
Since I wrote this1, I’ve switched to writing plain POSIX sh. Having scripts work on Ubuntu derivatives, alpine, and MacOS is really important.
The only issue that we keep hitting is use of
echowhich has a bunch of non portable options. Remembering to useprintfis important.Speaking of navigating the Open Group website, https://shellhaters.org is a friendly jumping-off point into the shell-specific parts of the POSIX specification. The talk associated with this website (“The Shell Hater’s Handbook” by Ryan Tomayko) can be watched here: https://www.youtube.com/watch?v=olH-9b3VJfs
this is really nice!! ty for sharing
I have personally found AWK to be a surprisingly good alternative to shells for larger than average scripts.
Why?
Portability (AWK is a part of POSIX)
Very clean syntax
Powerful associative arrays
Super-powerful string manipulations facilities
Easy shell interoperability
As an example here is a simplistic command runner [1] I’ve developed in AWK. As you can check [2] it runs unchanged under Linux/macOS/Win (via Git Bash).
[1] https://github.com/xonixx/makesure/blob/main/makesure.awk
[2] https://github.com/xonixx/makesure/actions/runs/702594092
laughs maniacally in powershell
Okay more seriously, I think whether you should stick with POSIX really depends on how important portability is to you. Because there are a lot of cases where “shell+arrays” is the right tool for the job, if you know you don’t have to run it on another OS!
Also a tip:
I’ve done similar things and this doesn’t work because it assumes you’re at the computer when the alarm goes off. I find it easier to open a textfile with the note, so you see it as soon as you’re back.
I generally disagree with the preference for writing in POSIX shell, and think that shell portability should not be a common concern in the majority of places shell is used. If you’re writing scripts that you know will be distributed widely, of course use POSIX shell. If you know out the gate that something you’re writing has to be portable across many different UNIXen, of course use POSIX shell.
I do suspect, however, that the vast majority of shell is not distributed, only need to target one platform, and in these cases, I’d recommend using whatever shell your target platform has installed (or install your favorite shell; I’m not your boss).
Shell portability is such a thorny topic that people like to pretend is less of a problem than it actually is. Yes, POSIX shell is generally portable. However, the reality is that to do anything meaningful in shell, you have to call out to other utilities, and these are likely the real cause of portability woes.
Basic problems are GNU coreutils vs the basic utilities provided on any other UNIX (I’m not gonna say BSD here, because even amongst BSDs, basic utilities have grown enough differing features). But the problem spirals depending on what your shell script is going to do.
Services? Get ready to deal with rcctl, service, systemctl, svcadm and more. Networking? How many different incompatible incantations of ifconfig are there? It’s probably time to switch to ip on Linux. What about ipadm? Firewalls? lol pf, FreeBSD’s ancient pf, iptables, nftables, ipfw, etc. My point here is that portability has way more to do with the utilities that you’re calling than just the shell language itself. Also, re: utilities, even if you use the same utilities, since you can’t declare dependencies in shell, I hope that your targets have compatible versions and only use the common set of features amongst your installbase :)
This all comes from frustration trying to use “”“compatible””” shell scripts across Solaris and BSDs and how POSIX shell has been the least of my problems in making shell scripts work in these environments. I think in the majority of cases, it’s more work than it is worth, and so I think it’s best to build using the best tools for you and your team that are available on the platform you’re targeting.
I will say, I do like the idea of using the limitations of POSIX shell as a litmus test for when you need to write something in a real language. As a shell-hater, I do believe that bash and zsh (and even ksh93) affords too many conveniences that make people resist switching to a proper language.
I also agree that many things are just so much faster to implement in shell than other languages, but it’s hard for me to reconcile the trade-off of ease of development vs how error-prone shell can be. Shellcheck does help mitigate a ton though, as was mentioned in the article.
I do still write shell, because it’s so much faster to implement, but I do try and keep it to just automating running a small set of commands and I’d never distribute anything written in shell as a tool/product.
Yeah, writing portable shellscripts is not at all dissimilar from trying to write e.g. Python 2/3 polyglots. We don’t think of the former as polyglots, probably because the common subset is standardized somewhere, but in practice it feels similar to me. Similar to Python 2/3 polyglots, I only write these for the most minimal of bootstrap tasks to get to a better suited language (and that may be bash, sometimes).
POSIX also defines a set of core utilities and how their interface works, so you can use them in POSIX sh without a problem (think cut, sed, awk, find, printf, xargs, mv, cp). Sure, that doesn’t help if you want to do system-specific things like services or networking, since there’s not even a pretence of compatibility between those systems. But that doesn’t necessarily justify adding more dependencies such as Bash just because you’re doing system-specific stuff anyway. POSIX sh is likely to be available everywhere, and fewer dependencies are better than more.
These are sufficient for a very small scope of tools ime. File manipulations, etc. With any of them, it’s very trivial to accidentally opt-in to features specific to the environment that you’re working in. Moreover, the question gets raised — what files are you manipulating? How system-agnostic are they? If you’re doing a bunch of file manipulations on files specific to RHEL 8, how has avoiding dependencies on bash or GNU coreutils helped you?
This is part of the problem with writing stuff in (any) shell though. What are your dependencies? In many languages, you at least have some type of manifest that lists the dependencies and the versions. In C, at least you can tease stuff out from cmake/autotools/whatever or in the worst case, by the header files/linker.
In a shell language, any single line of code can introduce a new dependency. There’s no real way to verify in a shell script what dependencies there are, what versions of those dependencies there are, or even what flavor of those dependencies there are.
I don’t understand why people view shell scripts as being free of dependencies, when often the dependencies are just more subtle. It becomes way more obvious when you try some of these shell script tools on a BSD or a commercial UNIX how pernicious this problem is.
Yes, and this is a great use-case for Nix. You can package up a shellscript together with its dependencies (including the precise version of coreutils etc. that it uses) and it’s just like packaging up any other dependencies.
If you target POSIX, then you can use POSIX shell and POSIX utilities without worrying about them as dependencies. That works across BSD and commercial UNIXes too. Your project might need more tools than those, but you can still rely on the POSIX utilities being available with at least the standardised flags.
I feel like this is a great recommendation and a general good practice for writing shell that could be run anywhere, on any shell.
I’ve written a ton of shell in the last couple of years and it’s running mostly in containers. It’s primarily bash, although I often started new stuff with posix shell and quickly fell into bash mode when I wanted something like arrays.
The biggest problem I’ve hit is writing some stuff that ended up using newer bash features not available in the version of Bash that Apple ships in macOS. I install and use the latest Bash from Homebrew so I’ve always kept up.
POSIX shell is the language I use most frequently. As well as
shellcheckmy main reference isman-pages-posix; these are the manuals of standard utilities converted from POSIX to manpages (there isn’t a page forshsyntax itself though). This ensures that, as well as only using portable syntax, I only use portable flags and tools. (I’ve mentioned this before but many people seem to have not come across it). The RATIONALE sections of many pages are also interesting, explaining the historical reasons for the standardised syntax and pitfalls to avoid.Over at KISS linux’s community we are continuing Dylan Araps’ work with a package manager and set of tools written entirely in POSIX shell.
I wish this had existed a few weeks ago, as it would have helped me make my points better :)
At $DAYJOB, we had an instance where we needed some automation that could run on developers’ workstations. That could be Mac or various Linuxes. So I wrote a lovingly handcrafted 130-ish line shell script that used only POSIX shell. It was actually fun to have a knowledge transfer session later about how it’s different than Bash, and about why I used
printfinstead ofecho, howreadonlyworks, etc.printf is severely underrated. i see people trying to use echo to print json constantly in shell, when i show them printf their world shifts dramatically.
Don’t both of these contain format string vulnerabilities? Or does shell printf escape quotes?
Yes. It does not.
Speaking of echoing JSON https://gist.github.com/abathur/e4f7ad5ac92820882dfe35cff52357b5 :p
Thank you for this, this is actually super useful to me. ofc I know printf from “real” programming languages and I knew that shell had printf but I never really connected it as being useful there too
re-training my echo muscle to use printf is one of the many valuable skills that posix shell taught me :3
The preformatted linebreaks make the site very hard to read on a phone
true, i noticed that too. i’ll have to do something to fix it, i dread css and stuff tho.
The text is really hard to read for some reason.
Try using lynx or links from a Linux shell.
I believe shell can be so much better. I’m not convinced that needing arrays is a sign that shell doesn’t fit your problem anyway. Shell is about interacting with the operating system and gluing together processes with ease. I’m hopeful that www.oilshell.org will succeed and raise our expectations for shell programming.
Arrays are nice for passing flag parameters.
The argument list is an array in POSIX sh.
Sorta? A lot of programs will interpret the value in
--foo 1 2 3as three values separated by spaces, but other programs interpret it as one value which contains spaces. There’s no easy way to indicate a string must be interpreted as an array, which also means things like no multidimensional arrays.There is one and only one array in POSIX sh:
$@, which is accessed by$*,$@, or$1,$2, …, and modified byshiftandset.Also handy for reading a whole bunch of parameters out of something like sqlite in one go
Yeah. My experience with shell programming mostly falls in the, “only if nothing else is available,” mental category. It is tedious doing shell-like things in Python but much more readable.
I’m interested in feedback on OSH (https://www.oilshell.org/) from POSIX shell users … It should run EVERYTHING in your scripts – if not, that’s a bug! Here a couple reasons I thought of that you might use it:
(1) More precise error messages
versus
In a long script, I think the more precise errors are good. Although one problem is that they could also be interleaved in concurrent output, so we might want an option to force 1 line errors. Feedback is appreciated!
The infrastructure is there to make all errors precise, with column info, and even seeing past multiple levels of parsing. But we need to scrub them to point at the right locations, and could use help. It should be pretty easy/fun for someone who knows Python/MyPy, and perhaps has written a toy tree interpreter (e.g. Lisp or similar).
(2) More detailed tracing:
It deals with newlines in a less misleading way, shows PIDs for command subs, etc.
More detailed tracing that tells you how many processes were started, and the exit codes:
You can also use
shopt --set xtrace_richin the program instead of-o.https://www.oilshell.org/release/latest/doc/xtrace.html
I’m showing dash here, but bash is basically the same in both of these areas.
Try it on your scripts and let us know what happens!
I stole a trick from old-school init.d and break my shell scripts into very tiny chunks that are numbered. 00-check-requirements.sh, 01-install-build-tools.sh, 02-setup-database.sh. I can test each one individually and prevent weird interactions between chunks of shell that are doing different things. Shared functions and environment variables can go in a common file. Then I can run them all easily in order from a parent script.
Another simple thing I do is “set -xeo pipefail” at the top of every single file. Prevents the majority of subtle errors.
Finally I use shellcheck obsessively, but that’s mentioned in the article.
Beware that
set -o pipefailis bash, not POSIX.As some of you know, I also use it to develop web sites with it: mkws. It’s great for massaging plain text content in HTML tags.
This looks like C?
Templating is done via shell code: https://mkws.sh/docs#templates, https://adi.onl/pp.html!
I love the style this is written (and laid out) in. It reminds me of classic textfiles, but also of the “ed is the STANDARD. Text editor.” rant. All snail posix shell!
If you’re targeting multiple unixlikes, maybe this is an option. But if you’re not gonna actually test those scripts in CI, there are so many ways even a POSIX shell script can break, due to how they are effectively ‘stringy’ typed. And if you’re running on CI, you might as well just install bash and enjoy the improvements it has.
I find it odd to advice people to strive for compatibility without mentioning CI. At the end of the day, standards are great, but not sufficient for ensuring that it actually works.
if someone is going to learn a shell dialect for CI, it may as well be posix shell. it’s nearly as powerful as bash, and offers a bunch of benefits. once you learn the dialect, it’s really not that bad. i don’t miss any bash features, and find myself happy with the smaller world of posix.
if that doesn’t work for you, by all means, go ahead and use bash.
what do you mean?
I mean that the interface between shell and the system is all based on strings: filenames and arguments for executables. If I gave you a script for FreeBSD and asked you if it would work on Debian 9, it would be hard to tell, since there is no type system or anything that enables the machine to check whether the arguments potentially passed are actually available.
https://wiki.c2.com/?StringlyTyped
I use Linux and Windows and MacOS regularly. Every one of them has bash. Even with every system having the same shell, making a script portable is difficult, because the shell is reaching out to many different things to do its work, and they’re distributed and versioned differently in different environments. The difference between
[and[[ranks far, far below “every system has a differenttar”. tar is not a posix utility, butadminis? https://pubs.opengroup.org/onlinepubs/9699919799/utilities/admin.htmlI often find it easier to write Python using only the standard library, because with a Python install, you have just one version number to worry about. I know my oldest Python is 3.8, so that’s what I target and it’s much easier to make portable than trying to figure out how to do it in a shell script.
Yes, SCCS is the version control system defined by POSIX, which is the reason why
adminis included in POSIX. I agree that unstandardisedtaris a problem, but the POSIX answer ispax.My whole argument is that the standard requires things that are so antiquated that they are irrelevant, so I’m not sure why telling me that it’s in the standard is supposed to convince me of anything, anything at all. I know that it’s in the standard. That’s my whole point. The fact that it’s in the standard is what makes the standard bad! Meanwhile, tar, an altogether ubiquitous utility that everyone knows is not in the standard. It is long past time to either evolve this standard or let it go.
The standard does evolve, but, being a standard, obviously not much. As far as I know, SCCS is in POSIX just so that there is a standardised VCS.
macOS has bash, but the installed bash is the last GPLv2 version, which is around 15 years old. If you’re using bashisms, there’s a good chance that they won’t work there.
my entire point is that portability extends beyond the shell and the posix utilities standard is old and that every shell script portability concern I’ve ever had has not been within the shell itself
I’ve recently had to fix a shell script that assumed GNU extensions for sed and awk, so I have some sympathy there, but my experience is that most *NIX platforms support a fairly large overlapping set of extensions to the core UNIX utilities, whereas bash extensions are rarely supported anywhere other than bash.
the very first thing I said is that I use Windows regularly. This is kinda my entire point: that portable is not actually meaningful when the people saying it are saying “well, it’s portable for me”.
one of the many reasons i don’t use bash is because there are so many subtle differences between bash implementations, as you pointed out. bash on macos is not bash on linux, and the coreutils are all different.
If you really need to be portable, perl might be best. If you need to be portable and available everywhere out of the box…uh I don’t think there’s anything remotely cross platform that’s on windows and Linux by default (unless you count wsl2, which I don’t).
POSIX sh essentially is portable and available by default across everything (all *NIXes) except windows.
uhhh, lemme put my ruby hat on and say, naw
for many common tasks, shell is absolutely faster. i wrote some short examples.
download a file:
replace the string “cats” with “dogs” in all files ending with .txt:
Well, not sure if using curl counts as POSIX shell, if so there’s always
In ruby.
Although I’m a bit partial to using
open3instead of backticks. While I wouldn’t extensively code golf your second example, I gotta say, if I was doing what you’d asked in a quick-but-perhaps-not-as-readable-version, it might look more like:curl doesn’t count as posix shell, i was just using it as an example of the conciseness of shell in general. i don’t think that using shelling out of your programming language counts ;) otherwise any language would be an adequate shell alternative!
half the power is being able to just whip shell up in your terminal on-the-fly. shell will also be available in many places ruby is not!
i do like the conciseness of ruby. ever since i read why’s ruby guide, the language has held a special place in my heart. have you seen crystal?
Oh absolutely, and I reach for shell often, just disagreed with the original thought about shell vs. any other language
I’ve been watching crystal and tried it for a few things, but isn’t worth paying the innovation token at $job when we have Go (a poor replacement for shell, imo). But I think for a shell replacement to be feasible it for sure needs to be interpreted
shell can be even shorter with
sed -i 's/cats/dogs/g' "$i"to avoid the tmp file. Or is there an advantage to the tmp file method?sed -iisn’t posix ;) so i’ve gotten used to using the tmpfile method. it’s what-idoes under the hood anyways.fair point!
It’s so much worse than @j3s implies. Not only is
-inot POSIX,-ion some platforms requires the suffix to use for the backup file, on others it’s optional. The fact that it’s optional makes it spectacularly dangerous because it’s easy to accidentally have it consume an argument. If you must use-i, at least use it with a suffix.