Hi – I’m the author of the post that’s being subtweeted by Rachel.
I definitely intended to grab some attention by framing “pipefail” as the Root Cause of this incident, but I thought I did an okay job of backing up the sensationalism with some deep technical content. I tweeted this as “the most convoluted outage that I have ever professionally witnessed.”
Looking at the comments in https://lobste.rs/s/5ldmtp/pipefail_how_missing_shell_option_slowed it seems like there was a pretty widespread misinterpretation that pipefail is supposed to protect against running p2 in p1 | p2. Rachel’s article is good at explaining why that can’t work due to the nature of pipes. Of course any actual outage is going to have multiple levels of causal analysis, so it’s not totally fair to say pipefail was or wasn’t the root cause.
p1 | p2
To be fair, the corporate announcement probably being alluded to used code that looked like
(reader | tee tempfile) && deploy tempfile
where set -o pipefail would have helped. But I totally agree about the overall sentiment that shell scripting is full of traps and it’s hard to write and review scripts. Not even shellcheck warns about that line.
set -o pipefail
The first version of the blog post did not however, it was updated after a discussion on lobste.rs with the author of the post, https://lobste.rs/s/5ldmtp/pipefail_how_missing_shell_option_slowed#c_i9kaaf
So what do we take away from this discussion? Should we abandon shell scripts for critical use-cases? Should we rewrite them in Go/Python/Ruby instead?
The takeaway in my eye is that you should check for errors!
Given a command line p1 | p2 with pipefail set, how do we determine which program failed?
If you’re piping p1 into p2, which one failed is sort of immaterial. They both were running and p2 consumed as much of p1’s output as it could before whichever one died. I think the moral is don’t expect pipes to be atomic when by design they are streaming.
Bash populates a special variable called PIPESTATUS for every command. It contains an array with each value being the return code of each component of a pipeline.
$ declare -p PIPESTATUS
declare -a PIPESTATUS=(="1")
$ false | cat
$ declare -p PIPESTATUS
declare -a PIPESTATUS=(="1" ="0")
I’m counting my bash experience in decades by now and I honestly never knew this. Amazing. Thanks!