Thanks, this is a great deep dive. I’m considering adopting your simple style rule (“Do not use anything but the two or three argument forms of [.”)
In fact, dash, mksh, and zsh all agree that the result of [ -a -a -a -a ] is 1 when the file -a doesn’t exist, not a syntax error! Bash is the odd man out.
I’ve discovered too many bizarre Bashisms to count, and hope you steer oilshell away from Bash behavior emulation and towards the POSIX shell spec. Even Bash’s supposed POSIX compatibility mode has Bashisms poking out. There’s one non-POSIX thing I dearly miss in /bin/sh: set -o pipefail. It’s difficult to write safe shell scripts without it, so much so that it should probably be in the spec.
Thanks. The next post will be about Oil’s equivalents, so I’ll be interested to hear your feedback.
You can also just use [[, although it’s less portable. The one thing I don’t like about [[ (besides aesthetics, I prefer test), is that == does globbing, as I mention in the appendix. That should have been a different operator, as =~ is for regexes.
Bash and all the shells I’ve tested are more POSIX compatible than I would have thought. (Bash does have a tendency to hide unrelated bug fixes behind set -o posix though.)
The bigger issue is that POSIX is not good enough anymore. POSIX doesn’t have set -o pipefail like you say, and it also doesn’t have even have local. For example, there are some Debian guidelines floating around that say use POSIX but add local and a few other things. Human-written scripts can’t get by with strict POSIX. Even echo -- is a problem.
This is the motivation behind the “spec tests” in OSH – to discover a more complete spec. Looking at what existing shells do is exactly how POSIX was made, although the process probably wasn’t automated and it was done many years ago.
I’m basically implementing what shells agree on. But I do have a bias toward for bash behavior when it’s not ridiculous, because bash is widely deployed. When all shells disagree, you have to pick something, and picking dash or mksh makes no sense. POSIX typically doesn’t say anything at all in these cases, so it’s not much help.
This is the motivation behind the “spec tests” in OSH – to discover a more complete spec…I’m basically implementing what shells agree on.
While we’re on the subject, here’s a point of disagreement between the shells that you might find interesting: assignment from a heredoc. If the POSIX shell spec has anything to say on this one, I couldn’t find it. Defining these kind of behaviors in a more complete shell spec does seem to me like a very valuable endeavor on its own.
But I do have a bias toward for bash behavior when it’s not ridiculous, because bash is widely deployed. When all shells disagree, you have to pick something, and picking dash or mksh makes no sense. POSIX typically doesn’t say anything at all in these cases, so it’s not much help.
I’d probably fall back on Bourne shell behavior as found in present day BSDs, or the Korn shell, or heck even dash, before going in for an obvious Bashism. Both Bourne and Korn exhibit careful, minimal design. Bash on the other hand was “anything goes” for a while there, with predictable implications for quality and security (“Wouldn’t it be cool if you could export functions to children via the env?!” => shellshock).
But, if it’s a question of how facilities common to all shells should behave, then choosing the Bash behavior isn’t necessarily bad.
I put his example as case 0, and his fix as case 1. Interestingly the “before” one works in mksh and zsh, but the “after fix” fails in those two shells.
dash accepts all of them and bash fails at all of them.
Case 2 is my rewrite of this, which works in OSH.
But going even further, I think this construct can always be expressed more cleanly as a separate assignment and then here doc. I did think about this issue, because OSH prints a warning that it’s probably not what you want:
osh warning: WARNING: Got redirects in assignment
Though I think this example is conflating two issues: the command sub + here doc, and the fact that sed has two standard inputs – the pipe and tr. I didn’t tease those things apart and I suspect that would help reason about this.
It’s definitely interesting but I’m going to leave it for now because it’s not from a “real” script… I still have a lot of work to do on those! But it’s in the repo in case anyone ever hits it.
It’s definitely interesting but I’m going to leave it for now because it’s not from a “real” script…
This was from a real script (that post is from my blog) but after a bit of git grepping I still can’t find it, so how real can it be eh? I agree your rewrite to $(some complex multiline stuff) is cleaner than the same in backticks, but the questions around when a here-document should be interpreted remain.
Though I think this example is conflating two issues: the command sub + here doc, and the fact that sed has two standard inputs – the pipe and tr.
It doesn’t depend on sed actually. Simpler test case:
foo=`cat`<<EOM
hello world
EOM
echo "$foo"
/bin/dash prints “hello world”, while Bash hangs waiting for input.
Oh sorry it wasn’t clear from the blog post where the example came from!
I just tested it out, and the simpler example works on dash, mksh, and zsh, but fails on bash. That is interesting and something I hadn’t considered. Honestly it breaks my model of how here docs are parsed. I wrote about that here [1].
And while you can express this in OSH, it looks like OSH is a little stricter than bash even. So I’ll have to think about this.
Right now I think there are some lower hanging fruit like echo -e and trap and so forth, but these cases are in the repo and won’t be lost.
I’m happy that Oil shell (the author) and Elvish (https://elvish.io/) are both innovating with shell development again.
This is definitely an area that had flatlined for the last decade outside of plugin development.
I’ve written tons of ZSH over the last two years as I basically developed my own ZSH framework as a hobby and most of the time I spend fighting the language. Even beyond the learning curve I still run into some many strange issues.
I’d love a new modern language to work with for scripting. Plugins are just not enough to fill the gap.
Thanks, this is a great deep dive. I’m considering adopting your simple style rule (“Do not use anything but the two or three argument forms of
[.”)I’ve discovered too many bizarre Bashisms to count, and hope you steer oilshell away from Bash behavior emulation and towards the POSIX shell spec. Even Bash’s supposed POSIX compatibility mode has Bashisms poking out. There’s one non-POSIX thing I dearly miss in
/bin/sh:set -o pipefail. It’s difficult to write safe shell scripts without it, so much so that it should probably be in the spec.Thanks. The next post will be about Oil’s equivalents, so I’ll be interested to hear your feedback.
You can also just use
[[, although it’s less portable. The one thing I don’t like about[[(besides aesthetics, I prefertest), is that==does globbing, as I mention in the appendix. That should have been a different operator, as=~is for regexes.Bash and all the shells I’ve tested are more POSIX compatible than I would have thought. (Bash does have a tendency to hide unrelated bug fixes behind
set -o posixthough.)The bigger issue is that POSIX is not good enough anymore. POSIX doesn’t have
set -o pipefaillike you say, and it also doesn’t have even havelocal. For example, there are some Debian guidelines floating around that say use POSIX but addlocaland a few other things. Human-written scripts can’t get by with strict POSIX. Evenecho --is a problem.This is the motivation behind the “spec tests” in OSH – to discover a more complete spec. Looking at what existing shells do is exactly how POSIX was made, although the process probably wasn’t automated and it was done many years ago.
I’m basically implementing what shells agree on. But I do have a bias toward for bash behavior when it’s not ridiculous, because bash is widely deployed. When all shells disagree, you have to pick something, and picking dash or mksh makes no sense. POSIX typically doesn’t say anything at all in these cases, so it’s not much help.
While we’re on the subject, here’s a point of disagreement between the shells that you might find interesting: assignment from a heredoc. If the POSIX shell spec has anything to say on this one, I couldn’t find it. Defining these kind of behaviors in a more complete shell spec does seem to me like a very valuable endeavor on its own.
I’d probably fall back on Bourne shell behavior as found in present day BSDs, or the Korn shell, or heck even dash, before going in for an obvious Bashism. Both Bourne and Korn exhibit careful, minimal design. Bash on the other hand was “anything goes” for a while there, with predictable implications for quality and security (“Wouldn’t it be cool if you could export functions to children via the env?!” => shellshock).
But, if it’s a question of how facilities common to all shells should behave, then choosing the Bash behavior isn’t necessarily bad.
Yes that’s the kind of thing that I’ve been testing. I copied it into my test framework:
https://github.com/oilshell/oil/commit/a79ebc8437781b8edb8fd8ad03276fc6255af1f3
Here are the results:
http://www.oilshell.org/git-branch/dev/oil4/5ca7bacb/andy-home/spec/blog-other1.html
I put his example as case 0, and his fix as case 1. Interestingly the “before” one works in mksh and zsh, but the “after fix” fails in those two shells.
dash accepts all of them and bash fails at all of them.
Case 2 is my rewrite of this, which works in OSH.
But going even further, I think this construct can always be expressed more cleanly as a separate assignment and then here doc. I did think about this issue, because OSH prints a warning that it’s probably not what you want:
Though I think this example is conflating two issues: the command sub + here doc, and the fact that
sedhas two standard inputs – the pipe and tr. I didn’t tease those things apart and I suspect that would help reason about this.It’s definitely interesting but I’m going to leave it for now because it’s not from a “real” script… I still have a lot of work to do on those! But it’s in the repo in case anyone ever hits it.
That’s a neat cross-shell test framework.
This was from a real script (that post is from my blog) but after a bit of git grepping I still can’t find it, so how real can it be eh? I agree your rewrite to
$(some complex multiline stuff)is cleaner than the same in backticks, but the questions around when a here-document should be interpreted remain.It doesn’t depend on
sedactually. Simpler test case:/bin/dashprints “hello world”, while Bash hangs waiting for input.Oh sorry it wasn’t clear from the blog post where the example came from!
I just tested it out, and the simpler example works on dash, mksh, and zsh, but fails on bash. That is interesting and something I hadn’t considered. Honestly it breaks my model of how here docs are parsed. I wrote about that here [1].
And while you can express this in OSH, it looks like OSH is a little stricter than bash even. So I’ll have to think about this.
Right now I think there are some lower hanging fruit like
echo -eandtrapand so forth, but these cases are in the repo and won’t be lost.[1] http://www.oilshell.org/blog/2016/10/17.html
Yeah, echo is terrible. When I went through my “learning about shell for real” experiences I ended up decided that printf is the way
I’m happy that Oil shell (the author) and Elvish (https://elvish.io/) are both innovating with shell development again.
This is definitely an area that had flatlined for the last decade outside of plugin development.
I’ve written tons of ZSH over the last two years as I basically developed my own ZSH framework as a hobby and most of the time I spend fighting the language. Even beyond the learning curve I still run into some many strange issues.
I’d love a new modern language to work with for scripting. Plugins are just not enough to fill the gap.