1. 76
  1.  

  2. 13

    I have read it elsewhere (probably in some Plan9 related docs) that it was the takeover of the UNIX system (From the eastern school of design that prioritized simplicity over everything else) by GNU (From the western school of design that did not value simplicity as much) that lead to the mess that is today.

    Honorable mention must also be made to the UUOC people who wanted to add more options to the programs for performance rather than fork a new program to do it.

    This isn’t to say there’s no cost to adding options – more options means more maintenance burden, but that’s a cost that maintainers pay to benefit users, which isn’t obviously unreasonable considering the ratio of maintainers to users.

    I do not agree that the cost is only for the maintainers of the program. The cost is also borne by the readers of the shellscript who has to understand the often very terse options.

    1. 20

      I think it’s unfair to blame the plethora of command-line options in GNU utilities solely on some sort of free-wheeling California exuberance, as compared to Yankee parsimony. Most GNU utilities are syntheses of existing commercial Unix utillities. I have no idea of where color options to ls originated, but once they were available, users not unreasonably wanted them, and it made sense for GNU ls to add a command line option to handle them.

      Similar with stuff like environment variables, which are just command line options set in another way.

      Edit the above was not related to my point so I have struck it out.

      As to the burden of maintainance, once an option is added, it can almost never be taken away, because that would break existing scripts.

      1. 3

        Environment variables are global, while command line options are lexical.

        It could be better if all the options are designed and introduced in a uniform fashion, instead of sporadically added for some peculiar niche.

        1. 4

          Agree re: envvars, I’ve removed that sentence.

          It could be better if all the options are designed and introduced in a uniform fashion, instead of sporadically added for some peculiar niche.

          Unix is old. Stuff accretes organically.

          1. 1

            Indeed, ever wonder why dd for example has weird arguments? It is a port from another os.

            In my first Unix(tm) job the old unix heads at the time had a joke:

            Want to know how old a command is? If its short with a lot of options its been there forever.

            Or put another way, quick name 3 single letter switches that ls doesn’t use. No man pages/-h/docs to help! Its surprising how many letters that thing takes. bsd or gnu for that matter.

        2. 3

          I think it’s unfair to blame the plethora of command-line options in GNU utilities solely on some sort of free-wheeling California exuberance, as compared to Yankee parsimony.

          I think that the western school valued completeness over simplicity, and it is debatable when a program becomes complete (perhaps it expands until it can read your email).

          Regarding the plethora of options, I grew up on *BSD and later AIX, HP-UX and finally, Solaris, and I remember distinctly that the GNU utilities were not a synthesis of previously existing flags. They actively originated a very large number of flags, and for a long time, only GNU utilities supported most of the arcane flags.

          1. 3

            Regarding the plethora of options, I grew up on *BSD and later AIX, HP-UX and finally, Solaris, and I remember distinctly that the GNU utilities were not a synthesis of previously existing flags.

            Ah Solaris, want to opine over /usr/bin, /usr/ucb/bin, /usr/xpg4/bin etc? >.< God was that a pita to manage/explain when people would run things with a different bin in the start of PATH and wonder why their working shell script no longer worked. Good memories of bad times.

            1. 1

              Thanks for your insight in how other systems were used! I basically got immersed in Unix culture after Linux became a thing and have never used anything but Linux or *BSDs in anger.

          2. 20

            I have read it elsewhere (probably in some Plan9 related docs) that it was the takeover of the UNIX system (From the eastern school of design that prioritized simplicity over everything else) by GNU (From the western school of design that did not value simplicity as much) that lead to the mess that is today.

            I believe it was Rob Pike who said “cat came back from Berkeley waving flags.”

            And I myself have said several times: “GNU’s Not Unix means exactly that.”

            A lot of GNU inherits from ITS and the MIT AI Lab, not necessarily Unix. The info documentation system, Emacs (TECO), GNU’s neutered su, the prevalence of Lisp, etc all came from ITS and/or the AI Lab.

            1. 2

              I suppose this is a difference in philosophy about the user of the program. It feels like there’s a spectrum of users that want to use one tool for all things (insert gratuitous emacs joke here) vs using one tool for each thing.

              I wonder what that heat map would look like, only with “frequency of use” expanded out to show usage in that OS’s actual scripts, various free or open source projects, and - if I had a magic wand across all systems in the universe - in various shell histories.

            2. 8

              I have plenty of complaints about PowerShell, but passing structured data around … isn’t among my complaints.

              If UNIX standardized on whatever was the JSON of the early 70s for IPC, would it have survived and thrived?

              1. 2

                The concrete format could be replaced easily, especially if it’s only used within pipelines. Flip the switch and all utilities switch from (say) JSON to YAML. If people stored some data that they’d get out of pipelines, just provide some conversion utility.

                Of course, there are going some leaky abstractions somewhere, but having every utility use its own custom text format is definitely more friction.

                The only strong counterargument that I can think of is that processing a structured format sometimes has a considerable overhead.

                1. 5

                  I like S-expressions for this task. It seems like most things can be represented with trees. They’re not trivially easy to work with using “primitive” editors that you might have found on early Unices, but they’re far more readable than XML and they also can be used as a programming language syntax, which is a testament to their usefulness. I couldn’t see myself using a programming language that uses YAML as a syntax.

                  That said, nearly anything standardized is better than plain text. So long as you can edit it as plain text and it encodes some high-level structure, I think it could be useful for this application. PowerShell might take the structure idea a little too far, but you can still (mostly) pretend that it’s just a regular Unix shell.

                  1. 3

                    Now my code needs to handle 2 input and output formats, depending on what it’s connected to. No thanks.

                    1. 6

                      not necessarily - it could be handled by something like libxo. not that that doesn’t have it’s own problems, but libraries do exist :)

                      1. 2

                        You might not have to if there are suitable tools available to convert between the common formats.

                  2. 7

                    I’ll be honest: when I read the usage printout of a command and find the flag that does exactly what my strange and obscur use case require, I feel really, really happy. Maybe this is bloat, but this is bloat that make me productive, and I have trouble getting worked up over that notion.

                    1. 4

                      All generalizations suck, but it needs to be stated that this isn’t a problem of untyped text streams, as much as the author (and others) would like to make it so.

                      The problem can generically be stated as organic accretion, but there’s probably a bit of human laziness underneath it all. I’d summarize it as “Compound piping among several commands is tough. Adding new options is easy!” While not true in each and every instance, and for many of these cases probably adding the option was tougher than the composition, it’s “true enough” where that over a period of decades the cruft begins to add up.

                      I wonder if further standards around the Unix Philosophy might help here, specifically in regards to code budgeting (tm) and other purposeful artificial (and perhaps functionally orthogonal) constraints on design.

                      1. 5

                        Organic accretion is a property of software. If something exists that works, it’s easier to add to it than remove from it. This ends up being desired by users too - you’re absolutely free to run Slackware 3.1 or at least the command line tools from it, but people don’t do so voluntarily.

                        But, imagine if command line options didn’t exist - we’d end up with a lot more binaries, and distributions where the number of binaries grows over time. Is that better or worse? It seems like it would be slower (piping means context switches), reading scripts seems of similar complexity since the number of things to know is essentially the same, and it may or may not improve flexibility, depending on the supported combinations of the command line options vs. the composability of separate commands.

                        It’s hard to read the article without sensing that the real complaint is that software is becoming more capable over time, as opposed to the form of that capability.

                        (That said, I would like a mini-man that tells me the most common options that’s faster to read, and be able to fall back to the “manual” when something obscure comes up.)

                        1. 4

                          (That said, I would like a mini-man that tells me the most common options that’s faster to read, and be able to fall back to the “manual” when something obscure comes up.)

                          https://tldr.sh/ ?

                          1. 1
                          2. 2

                            Organic accretion is a property of software, and it happens, oddly enough, organically.

                            I don’t think we’re so far apart.

                            Let’s take ls. Assume that ls has one and only one output: a series of columns with everything you’d like to know about the filesystem you’re pointing it at. Now there very well may be some command-line options, but they all would be around what the target is: whether to recurse, whether to follow symbolic links, and so on. No command-line options around display.

                            Would that work? Sure it would. You’d just pipe whatever came out to grep, awk, or whatnot to do the rest of the work that command-line options currently do. But wait! Your point is valid: that’s a lot of piping and context-switching. Heck, you’re tanking performance and you could easily end up with tons of binaries for any simple job. Think of the versioning problem alone. Much easier to add to something that already works (and you know) than use three or four tools you don’t know for the job, increasing coupling and dependencies and making whatever you’re doing much harder to understand later.

                            But what we’re both complaining about is an implementation detail: how various command-line functions work together and under-the-hood. Could we just smash up long command-line pipes into one executable with one binary, optimizing caching and piping under the hood? Sure we could. We don’t, but such optimizations take place all of the time in programming languages. There’s no reason that they couldn’t take place on the command line, except one: it’s easier to accrete off something you already understand and works than it is to join up several things you might not know so well into something that might be harder for others to grok. If we could smash up and break apart complex command-lines at will, we’d be doing the same thing as adding options without having to modify code or making any one command so bloody complex.

                        2. 4
                          • If we have fewer tools that does more, we get convenient interfaces, see the worn out example of python’s requests module of the importance of an all knowing “outer layer”/facade to an API, and the explosion of npm packages an example of the granular opposite.
                          • The minimal number of most common cli commands to know, for those not invested/not on platforms where all specialized tools exist, would go up drastically if the most common ones had less flags.
                          • It’s quicker to type ls -r than ls | tac so the extra complexity saves keystrokes when used interactively, and lessens the reliance on another tools’ availability/interface being stable.

                          The tools’ sizes being bigger and more complex – can that be compared to (both “good” and “bad” properties of) statically linked programs?

                          Regarding the man page/–help size, maybe it’s a presentation issue which could be approached with e.g. promoting what’s important to many.

                          Treating the same programs/builtins as libraries sometimes (small & specific) and applications sometimes (“please do all of it for me”) is more of a testimony of how adaptable the interface of pipes, streams etc is, and how many Real Life (TM) problems that were solved with “adding more methods” as Luu writes. Where to put that complexity (new “libraries” vs new “applications”) probably sorts itself out given time, just as libraries are written/extracted because they solve an unsolved issue in a real application.

                          (This comment is mostly a reaction to the Lobsters comments, I thought the article was pretty objective and made good points for both sides.)

                          1. 2

                            the language was always going to turn into a write-only incoherent mess when taken as a whole. At least with tools that bundle up more functionality and options than is UNIX-y users can replace a gigantic set of wildly inconsistent tools with a merely large set of tools that, while inconsistent with each other, may have some internal consistency…. the old school UNIX mavens would have sat down in the same room and thought longer and harder until they came up with a set of consistent tools that has “unusual simplicity”. But that was never going to scale…

                            Maybe from a practical “people gonna people” perspective, the mess was inevitable.

                            But from an absolute perspective, the mavens were right. It’s not inevitable.

                            A thoughtful set of primitives that can be composed together gives you much greater efficiency as a programmer: your “power : manual-reading” ratio goes way up. The APLs demonstrate this principle beautifully.

                            The options-accretion approach results in a massive amount of both duplicated code and duplicated cognitive load (ie, you have to learn the same fundamental thing over and over under different names).

                            1. 2

                              My hobby: opening up McIlroy’s UNIX philosophy on one monitor while reading manpages on the other.

                              The meaning of this sentence is quite ambiguous. Looking at the numbers, I guess they are talking about gnu/linux systems or macOS, which are known to have an acute hormone imbalance syndrome regarding command line options. Yet, other unix systems are much closer to the original spirit, with considerably leaner standard tools. For instance, OpenBSD takes great care in providing a beautifully minimal set of tools by default.

                              1. 10

                                While it’s true that OpenBSD utilities tend to have fewer options, are you really arguing that, for example, ls going from 11 options to 28 (2.55x increase) is fundamentally different from ls going 11 to 58 (5.27x)?

                                For people who say that this this growth in options is a sign of insufferable bloat, if Linux is “known to have an acute hormone imbalance syndrome regarding command line options”, how can it be that bloating up half as much is “beautifully minimal”?

                                1. 4

                                  I agree. The one true way is that of plan9. Unfortunately, it is not really practical to use it always and we have to settle for bloated compromises like OpenBSD.

                                  1. 5

                                    Can’t tell if sarcastic or not.

                                    1. 4

                                      That’s good sarcasm, then!

                                      Actually, it wasn’t. Sometimes I feel like I’m one of the last members of some sort of unix die-hard tribe, when my normal and balanced opinions get interpreted as sarcasm or trolling.

                                      1. 7

                                        Nothing illustrates the protean nature of the word “Unix” better than a self-described “[member of a] unix die-hard tribe” advocating the use of a fork of a fork of a fork of a Berkeley system to approach the never-realized apotheosis of the AT&T ideal…

                              2. 2

                                When tar has nearly as many flags as the UN headquarters you know you have done something wrong at some point…

                                1. 4

                                  Reading through tar(1), a lot of the flags deal with subtleties and complexities in various filesystems and the like. Nothing really stands out as “excessive bloat” to me.

                                  The same applies to something like curl, which has many more flags and it’s actually quite confusing at times, but they’re all there for a reason: to deal with the complexities of network transfers.

                                2. 1

                                  I agree with Doug McIlroy - specialized and on target, “. For the ls example, it’s about file attributes. Now … you can have filesystems with different attributes, so perhaps having a plug-in architecture that matches underlying filesystem to only those attributes might make sense to scope the utilities use of those few flags, likewise what the man page shows. Yes there are issues with modals, but having concise utilities has some powerful benefits, especially if I broadcast them to a million nodes of a cluster/cloud to achieve a time critical parallel effect and response.