To sysadmins, a program’s output or its lack thereof is a vital part of its behavior; adding things to the output is anything but harmless, and often forces us to fix the program right away.
“Forces us to fix the program right away” seems like a Words As Intended, from the perspective of a maintainer.
This author has a long history of similarly-hot takes on the topic of software compatibility and change over time, and their position consistently is hard to distinguish from “other people are not allowed to change their software, but I am still allowed to change mine”.
In the specific case of things invoked from shell scripts, I don’t think it’s controversial. UNIX provides streams of text, any change in the output format is a breaking change. If the tool provided structured output (e.g. JSON) then it would be less of an issue because tools could easily discard unknown fields, but anything else will be connected to ad-hoc parsers.
Sometimes making a breaking change is the right call. Supporting legacy behaviour can be a significant maintenance burden. That needs to be weighed against the cost to the downstream consumers of the break. In the case of two self-contained, trivial, shell script wrappers that are unlikely to ever need modification, the cost is immeasurably close to zero and so any non-zero breakage downstream should be enough of a cause to stop.
Lumping stdout and stderr together as “the program’s output” without any semantic difference (“any chance in the output format is a breaking change”) flies in the face of all my experience using UNIX.
stdout, sure, I’ll accept that as a versioned API. It’s the result of the requested process; it shouldn’t be more than that.
But stderr isn’t and hasn’t ever been. Errors are an unbounded and fractal set. Most programs pass stderr to their subprocesses. Even glibc prints messages on stderr!
[…] two self-contained, trivial, shell script wrappers […]
Not important at all, but I thought GNU grep changed its behaviour based on argv[0]?
Lumping stdout and stderr together as “the program’s output” without any semantic difference (“any chance in the output format is a breaking change”) flies in the face of all my experience using UNIX.
It’s common to treat ‘any output on stderr’ as an error. For example, ctest has a mode that treats output on stderr as a test failure, so any test using egrep in a pipeline processing it’s output will now fail.
Not important at all, but I thought GNU grep changed its behaviour based on argv[0]?
Nope, see the article. It used to but that was changed years ago and the wrapper scripts were added for backwards compatibility. The new change added the error message in output from these scripts. It was more effort to add the deprecation notice than to do nothing and keep the wrappers in perpetuity, all because of a belief that POSIX should be proscriptive rather than prescriptive and descriptive.
It’s common to treat ‘any output on stderr’ as an error.
I’ve seen to treat ‘any non-zero exit’ code as an error; but, this is the first time I’ve ran into this culture.
For example, ctest has a mode that treats output on stderr as a test failure, so any test using egrep in a pipeline processing it’s output will now fail.
A test framework seems like a very special case which I hope any developer or maintainer would be happy to hear got broke. This example feels like it would be tagged as Works As Intended in the grep bugtracker.
[…] a belief that POSIX should be proscriptive rather than prescriptive and descriptive.
I’m but a simple journeyman who long ago accepted that APIs change in arbitrary and usually undocumented ways.
It comes from sysadmin culture. It’s important to tune cron jobs so that they are silent in normal operation, and any output is an alert that needs attention and debugging.
When writing scripts, some programs can be uncooperative and fail to set their exit status reliably, so it’s good practise to also check stderr. This is usually only necessary for programs that are not designed to be scripted. It’s a nasty turn for tools that have been mainstays of scripting in unix for 45 years to suddenly become uncooperative.
What part of the release notes? Can you please quote the section which says this change is supposed to fix something that was broken?
Because the only text I see that’s related to egrep and fgrep is:
The egrep and fgrep commands, which have been deprecated since
release 2.5.3 (2007), now warn that they are obsolescent and should
be replaced by grep -E and grep -F.
Adding a warning when using deprecated functionality is a perfectly valid change in a lot of contexts, but it’s not a fix. It’s not marked as a fix, the maintainers don’t say it’s a fix, it’s not a kind of change which is usually considered a fix, it’s not even something which can be interpreted to fall under any colloquial definition of the word “fix”.
This hit close to home for me because I recently updated a 3-year old dependency that had dozens of pointlessly backwards-incompatible changes, including 3 major version changes.
Things like making a function name present-tense, removing some 3-line helper functions that were used in every old example all over the internet, changing perfectly-normal behavior into a fatal error, etc.
I think this cavalier attitude towards backwards-incompatible changes needs to go away. If everyone’s going to have to update their code, it better be for a good reason like security, significant reduction of maintenance burden, a big new feature, etc. It can’t be “purity” or “consistency” or “someone submitted a PR so I just merged it”. Because every silly backwards-incompatible change makes it that much harder for important backwards-incompatible changes.
I think the GNU Grep maintainership should try to remember that peoples’ time is valuable and this kind of change-for-change’s-sake doesn’t do anything except make people work more. It’s cute to follow standards and specs and whatever, but work needs to be done and the day is only so long. Sometimes, mistakes were made and become expected behaviour, let it be!
Why should they? Are the amorphous “people” paying them?
In absence of a financial relationship, what’s left is mindshare and usage. Perhaps this will be the push for downstream consumers to use a different grep? I doubt it, though.
I think it’s totally reasonable to say, “if GNU grep were being released today for the first time, it would be a weird mistake to include these other commands that just call grep(1) with certain options.” But that’s not the situation we’re in. The egrep and fgrep commands have been around for decades.
This is kind of a microcosm of what’s wrong with the C/Unix world, though. “Somebody did it this way decades ago, now everyone who ever builds anything for the rest of eternity has to support doing it that way” is not a good philosophy, but it is the C/Unix philosophy.
It really is good, from time to time, to evaluate whether things are worth keeping around and clean house a bit.
It has been the GNU philosophy, more or less, but certainly not the C/Unix philosophy.
In evaluating whether things are worth keeping, one would do a cost/benefit analysis, and you would expect there to be something in the “benefit” column for a change to go through.
Quoted above is the philosophy of the author of the post on “New warning messages might as well be fatal errors”.
This is not a “GNU philosophy” – it’s a C philosophy, one that has actively held the language back. If you don’t believe that this is in fact a C thing, here is a rant from someone who literally works on the standard. See the section titled “We Consider Warning Changes, Breaking”. Or from further down:
And yet, as a Committee, we struggle with adding const to 4 function return values because it would add warnings to potentially dangerous code. We raised objections to deprecating the old K&R syntax despite tangible evidence of both innocent screw ups and also tangible vulnerabilities from developers passing in the wrong types. We almost added undefined behavior to the preprocessor, just to bend over backwards and make a bespoke C implementation “do the right thing”. We are always teetering on the edge of doing the objectively wrong thing for backwards compatibility reasons. And this, dear reader, is what scares me the most about the future of C.
It’s also a Unix philosophy. So much pain is caused by “somebody did this once in the 1970s and as a result it has to stay legal for anyone anywhere to do, any time, forever”.
No real disagreement now that you are using the indefinite article. The Unix philosophy is a thing though, and the people who uphold it have had no qualms about scrapping the old and starting anew.
Here’s the connection with GNU:
In particular, don’t reject a new feature, or remove an old one, merely because a standard says it is “forbidden” or “deprecated”.
Much like which, fgrepegrep arn’t portable, they are part of GNU Grep (and others), but they are not standardized. If you want to be reasonably sure your code will run consistently on all posix platforms, probably best to avoid those shortcuts.
POSIX (which is now the same specification as the Single UNIX Specification) is meant to be a lowest common denominator. The way that POSIX is extended is the following sequence:
A system adds an extension.
Other systems add similar extensions.
If the extensions are incompatible, the vendors agree on a common useful set of interfaces.
This de-facto standard is proposed for incorporation into the de-June standard.
The next version of POSIX incorporates it.
Unfortunately there is a subset of the community that believes ‘not in POSIX’ means ‘should not exist’ rather than ‘should not be relied on to work everywhere’, an attitude that would prevent POSIX from ever evolving.
Checking the FreeBSD man page, egrep, fgrep, and rgrep are all supported. They exist on GNU and BSD platforms (including Darwin), so they’re already at step 3. The correct thing to do is advance them to step 4 and add them to the next version of POSIX, not remove them from some platforms that support them.
No, they have been in Unix since the 1970s, they are portable, or at least they were before this boneheaded GNU change. (Boneheaded because it is causing pain and they are proceeding ahead stubbornly despite reasonable complaints.)
The history of these programs is that they were originally completely separate, implemented using different algorithms. The -E and -F flags came much later after computers became big enough that all the different modes could fit into a single program. If you look at the early days of metaconfig or autoconf, I bet you will find that egrep and fgrep were more portable than grep -E and grep -F.
A day or so after writing the above I checked some old man pages, which indicated that grep -E and grep -F came from System V, and BSDish systems did not get a multi-algorithm grep until the mid-1990s.
You know that’s usually written at the bottom of the man page and it’s trivial to check?
I didn’t know that. I appreciate the information. The snark, not so much.
Anyway as BSDs support [e|f]grep, that would mean that prior to GNU removing them they would work on 99.99%[1] of production unix-likes. So the decision to remove the aliases from GNU makes even less sense.
I can personally attest that OpenBSD ships commands that only partially conform to POSIX. Quoting OpenBSD locale(1):
With respect to locale support, most libraries and programs in the OpenBSD base system, including the locale utility, implement a subset of the IEEE Std 1003.1-2008 (“POSIX.1”) specification.
I agree. I never really understood why they existed for so long. And while such discussions go on for decades, tools like rg, ag, ack, etc. Become more and more popular without any of these neckbeard dramas.
I refrain from using -P on shellscripts because it is gnu specific and OSX users will come to me claiming my script “has an error”. Is -F equally GNU specific?
This tempest in a teapot seems stirred up by the belief that new warning messages might as well be fatal errors … which … is quite a hot take.
“Forces us to fix the program right away” seems like a Words As Intended, from the perspective of a maintainer.
This author has a long history of similarly-hot takes on the topic of software compatibility and change over time, and their position consistently is hard to distinguish from “other people are not allowed to change their software, but I am still allowed to change mine”.
I find it pretty easy to distinguish, personally.
In the specific case of things invoked from shell scripts, I don’t think it’s controversial. UNIX provides streams of text, any change in the output format is a breaking change. If the tool provided structured output (e.g. JSON) then it would be less of an issue because tools could easily discard unknown fields, but anything else will be connected to ad-hoc parsers.
Sometimes making a breaking change is the right call. Supporting legacy behaviour can be a significant maintenance burden. That needs to be weighed against the cost to the downstream consumers of the break. In the case of two self-contained, trivial, shell script wrappers that are unlikely to ever need modification, the cost is immeasurably close to zero and so any non-zero breakage downstream should be enough of a cause to stop.
Lumping
stdoutandstderrtogether as “the program’s output” without any semantic difference (“any chance in the output format is a breaking change”) flies in the face of all my experience using UNIX.stdout, sure, I’ll accept that as a versioned API. It’s the result of the requested process; it shouldn’t be more than that.But
stderrisn’t and hasn’t ever been. Errors are an unbounded and fractal set. Most programs passstderrto their subprocesses. Even glibc prints messages on stderr!Not important at all, but I thought GNU grep changed its behaviour based on
argv[0]?It’s common to treat ‘any output on stderr’ as an error. For example, ctest has a mode that treats output on stderr as a test failure, so any test using egrep in a pipeline processing it’s output will now fail.
Nope, see the article. It used to but that was changed years ago and the wrapper scripts were added for backwards compatibility. The new change added the error message in output from these scripts. It was more effort to add the deprecation notice than to do nothing and keep the wrappers in perpetuity, all because of a belief that POSIX should be proscriptive rather than prescriptive and descriptive.
I’ve seen to treat ‘any non-zero exit’ code as an error; but, this is the first time I’ve ran into this culture.
A test framework seems like a very special case which I hope any developer or maintainer would be happy to hear got broke. This example feels like it would be tagged as Works As Intended in the
grepbugtracker.I’m but a simple journeyman who long ago accepted that APIs change in arbitrary and usually undocumented ways.
It comes from sysadmin culture. It’s important to tune
cronjobs so that they are silent in normal operation, and any output is an alert that needs attention and debugging.When writing scripts, some programs can be uncooperative and fail to set their exit status reliably, so it’s good practise to also check stderr. This is usually only necessary for programs that are not designed to be scripted. It’s a nasty turn for tools that have been mainstays of scripting in unix for 45 years to suddenly become uncooperative.
I agree, but this is yet another example of “there’s no such thing as plaintext”.
“Fix” implies something was broken before.
It seems to me that the maintainers believe they made a fix. You’re welcome to both your own opinion and choice of software.
Where do the maintainers indicate that they fixed something?
The release notes?
What part of the release notes? Can you please quote the section which says this change is supposed to fix something that was broken?
Because the only text I see that’s related to egrep and fgrep is:
Adding a warning when using deprecated functionality is a perfectly valid change in a lot of contexts, but it’s not a fix. It’s not marked as a fix, the maintainers don’t say it’s a fix, it’s not a kind of change which is usually considered a fix, it’s not even something which can be interpreted to fall under any colloquial definition of the word “fix”.
This hit close to home for me because I recently updated a 3-year old dependency that had dozens of pointlessly backwards-incompatible changes, including 3 major version changes.
Things like making a function name present-tense, removing some 3-line helper functions that were used in every old example all over the internet, changing perfectly-normal behavior into a fatal error, etc.
I think this cavalier attitude towards backwards-incompatible changes needs to go away. If everyone’s going to have to update their code, it better be for a good reason like security, significant reduction of maintenance burden, a big new feature, etc. It can’t be “purity” or “consistency” or “someone submitted a PR so I just merged it”. Because every silly backwards-incompatible change makes it that much harder for important backwards-incompatible changes.
I think the GNU Grep maintainership should try to remember that peoples’ time is valuable and this kind of change-for-change’s-sake doesn’t do anything except make people work more. It’s cute to follow standards and specs and whatever, but work needs to be done and the day is only so long. Sometimes, mistakes were made and become expected behaviour, let it be!
Why should they? Are the amorphous “people” paying them?
In absence of a financial relationship, what’s left is mindshare and usage. Perhaps this will be the push for downstream consumers to use a different grep? I doubt it, though.
They “should” because it’s the right thing to do. This was not a good reason to force downstream action.
It seems the GNU grep maintainers don’t share your morals. And as you wrote “work needs to be done,” I suspect you don’t understand theirs.
This is one of the worst takes I’ve ever read.
The sooner these deprecated commands are removed from GNU grep the better.
Then the complaints can be “wHy DiDn’T yOu WaRn uS???”
I think it’s totally reasonable to say, “if GNU grep were being released today for the first time, it would be a weird mistake to include these other commands that just call grep(1) with certain options.” But that’s not the situation we’re in. The egrep and fgrep commands have been around for decades.
This is kind of a microcosm of what’s wrong with the C/Unix world, though. “Somebody did it this way decades ago, now everyone who ever builds anything for the rest of eternity has to support doing it that way” is not a good philosophy, but it is the C/Unix philosophy.
It really is good, from time to time, to evaluate whether things are worth keeping around and clean house a bit.
It has been the GNU philosophy, more or less, but certainly not the C/Unix philosophy.
In evaluating whether things are worth keeping, one would do a cost/benefit analysis, and you would expect there to be something in the “benefit” column for a change to go through.
Quoted above is the philosophy of the author of the post on “New warning messages might as well be fatal errors”.
This is not a “GNU philosophy” – it’s a C philosophy, one that has actively held the language back. If you don’t believe that this is in fact a C thing, here is a rant from someone who literally works on the standard. See the section titled “We Consider Warning Changes, Breaking”. Or from further down:
It’s also a Unix philosophy. So much pain is caused by “somebody did this once in the 1970s and as a result it has to stay legal for anyone anywhere to do, any time, forever”.
No real disagreement now that you are using the indefinite article. The Unix philosophy is a thing though, and the people who uphold it have had no qualms about scrapping the old and starting anew.
Here’s the connection with GNU:
https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html
And someone who pointed this out in objection to the grep change:
https://lists.gnu.org/archive/html/bug-grep/2022-09/msg00013.html
Why is the world better by removing these wrappers?
Why? They are in all unix systems, cause no harm, and people use them.
[Comment removed by author]
and why is that better
Much like
which,fgrepegreparn’t portable, they are part of GNU Grep (and others), but they are not standardized. If you want to be reasonably sure your code will run consistently on all posix platforms, probably best to avoid those shortcuts.POSIX (which is now the same specification as the Single UNIX Specification) is meant to be a lowest common denominator. The way that POSIX is extended is the following sequence:
Unfortunately there is a subset of the community that believes ‘not in POSIX’ means ‘should not exist’ rather than ‘should not be relied on to work everywhere’, an attitude that would prevent POSIX from ever evolving.
Checking the FreeBSD man page, egrep, fgrep, and rgrep are all supported. They exist on GNU and BSD platforms (including Darwin), so they’re already at step 3. The correct thing to do is advance them to step 4 and add them to the next version of POSIX, not remove them from some platforms that support them.
There are plenty of flags in GNU Grep (and many, many other tools) that aren’t in POSIX either. When can we expect removal of all those?
Maybe the error should be on
POSIXLY_CORRECT=1thenThat would certainly be saner
incredible that they didn’t just do that
actually there was some discussion of this on the bug-grep mailing list:
https://lists.gnu.org/archive/html/bug-grep/2022-09/msg00000.html
@sjamaan
No, they have been in Unix since the 1970s, they are portable, or at least they were before this boneheaded GNU change. (Boneheaded because it is causing pain and they are proceeding ahead stubbornly despite reasonable complaints.)
The history of these programs is that they were originally completely separate, implemented using different algorithms. The
-Eand-Fflags came much later after computers became big enough that all the different modes could fit into a single program. If you look at the early days of metaconfig or autoconf, I bet you will find thategrepandfgrepwere more portable thangrep -Eandgrep -F.A day or so after writing the above I checked some old man pages, which indicated that
grep -Eandgrep -Fcame from System V, and BSDish systems did not get a multi-algorithmgrepuntil the mid-1990s.So the decision by the GNU project to deprecate
egrepandfgrepin favor ofgrep -Eandgrep -Fwas to bring GNUgrepinto alignment with POSIX?Looking at OpenBSD
grepI can see it has-Eand-Foptions but I don’t know if that grep is POSIX compliant.You know that’s usually written at the bottom of the man page and it’s trivial to check?
Also why would it not be?
On the other hand, BSDs do provide egrep and fgrep so…
I didn’t know that. I appreciate the information. The snark, not so much.
Anyway as BSDs support
[e|f]grep, that would mean that prior to GNU removing them they would work on 99.99%[1] of production unix-likes. So the decision to remove the aliases from GNU makes even less sense.[1] wild-ass guess
I can personally attest that OpenBSD ships commands that only partially conform to POSIX. Quoting OpenBSD
locale(1):I agree. I never really understood why they existed for so long. And while such discussions go on for decades, tools like rg, ag, ack, etc. Become more and more popular without any of these neckbeard dramas.
I refrain from using -P on shellscripts because it is gnu specific and OSX users will come to me claiming my script “has an error”. Is -F equally GNU specific?
-E and -F, along with the syntax of ERE (-E) is all specified in POSIX