The book The UNIX Programming Environment by Kernighan and Pike is also a great read on much the same topic. It’s expensive to buy new but I found a cheap second-hand copy.
The thing is, I find the argument really compelling… but it doesn’t seem to have caught on. Some people really like this style of computing - see the Plan 9 holdouts, for example - but UNIX moved away from it pretty quickly once it left Bell Labs (hence this paper, which uses BSD and SysV changes as examples). GNU is definitely not UNIX, and Windows never was.
It seems like the UNIX style appeals strongly to a few people, but has no mass appeal. Maybe it appeals strongly to developers whose main focus is building software systems, but less to end users - and hence less to developers whose main focus is building software for those end users? Contrast Kernighan and Pike’s paper with Jamie Zawinski’s Unity of Interface, which he described as, “An argument for why kitchen-sink user interfaces (such as web browsers, and emacs) are a good idea.”
Modularity is great if it comes with composability. UNIX originally justified its existence with a typesetting system and the UNIX kind of modularity where you consume a text file and produce a text file was great for that. The only stage that wasn’t plain text was the set of printer control commands that were sent to the printer device (over a stream interface, which looked like a text stream if you squinted a bit). It doesn’t actually work for anything non-trivial.
Here’s a toy example: Consider ls -Slh. In a UNIX purist world, -l is the only one of these that you actually need in ls and actually you don’t need that because if -l is the default you can get the non-verbose output with ls -l | cut -f 9 -w and you can make that a shell alias if you use it often enough. You don’t need -S because sort can sort things, so you can do ls -l | sort -k5. Except that now you’ve gone away from the UNIX spirit a bit because now both sort and cut have the functionality to split a line of text into fields. Okay, now that you’ve done that how do you add the -h bit as a separate command? You can do it with some awk that splits the input and rewrites that one column, but now you’ve added an entire new Turing-complete language interpreter to the mix. You could, in fact, just do ls | awk and implement the whole sort part in awk as well. If you do this, you’ll discover that awk scripts compose a lot better than UNIX pipelines because they have functions that can take structured values.
Many years ago, I sent a small patch to OpenBSD’s du to add the -d option, which I use with around 90% of du invocations (du -h -d1 is my most common du invocation). The patch was rejected because you can implement the -d equivalent with -c and a moderately complex find command. And that’s fine in isolation, but it doesn’t then compose with the other du flags.
The UNIX-Haters Handbook explained the problem very well: We’ve implemented some great abstractions in computing for building reusable components: they’re functions that consume and produce rich data types. Composing systems implemented in terms of these primitives works great. Witness the huge number of C/C++ libraries, Python packages, and so on in the world. In contrast, plumbing together streams of text works well for trivial examples, can be forced to work for slightly more complex examples, and quickly becomes completely impossible for more complex ones.
Even some of the most successful kitchen-sink UIs have been successful because they support composition. Word and Photoshop, for example, each have rich ecosystems of extensions and plugins that add extra features and owe their success in a large part to these features. You can build image-processing pipelines in UNIX with ImageMagick and friends but adding an interactive step in the middle (e.g. point at the bit you want to extract) is painful, whereas writing a Photoshop workflow that is triggered from a selection is much easier. Successful editors, such as vim, emacs, and VS Code, are popular because of their extensions far more than their core functionality: even today, NeoVim and Vim are popular, yet nvi has only a few die-hard users.
Yeah I agree the sort -k5 thing is annoying. Several alternative shells solve that to some extent, like PowerShell, nushell, and maybe Elvish (?). They have structured data, and you can sort by a column by naming it, not by coming up with a sort invocation that re-parses it every time (and sort is weirder than I thought.)
However I want a Bourne shell to have it, and that’s Oil, although it’s not done yet. I think what PowerShell etc. do has a deficiency in that it creates problems of composition. It doesn’t follow what what I’m calling the Perlis-Thompson principle (blog posts forthcoming).
Unfortunately, it’s not really something that you can solve in a shell because the problem is the set of communication primitives that it builds on. PowerShell is incredibly badly named, because it’s not really a shell (a tool whose primary purpose is invoking external programs), it’s a REPL. It has the converse problem: there’s no isolation between things in PowerShell.
Something like D-BUS would be a reasonably solid building block. If the shell managed a private D-BUS namespace for the current session then you could write composed commands that run a bunch of programs with RPC endpoints in both directions and some policy for controlling how they go together. You could then have commands that exposed rich objects to each other.
’d be quite curious to know what a shell based around the idea of exporting D-BUS objects would look like, where if you wrote foo | bar then it would invoke both with D-BUS handles in well-known locations for each, rather than file descriptors for stdin / stdout. For example, a shell might provide a /shell/pipeline/{uuid} namespace and set an environment variable with that UUID and another with the position in the pipeline for each process so that foo would expose something in /shell/pipeline/3c660423-ee16-11eb-b8cf-00155d687d03/1 and look for something exposed as /shell/pipeline/3c660423-ee16-11eb-b8cf-00155d687d03/2, expose the /0 and /3 things for input from and output to the terminal (and these could be a richer set of interfaces than a PTY, but might also provide the file descriptor to the PTY or file directly if desired). Each shell session might run its own D-BUS bus or hook into an existing one and the shell might also define builtins for defining objects from scripts or hooking in things from other places in a D-BUS system.
(very late reply, something reminded me of this comment recently)
I don’t see why you need DBus or anything similar to solve the sort -k5 problem? I think you just need something like TSV over pipes, and then a sort-by tool that takes a column name to sort by. An enhancement would be to attach types to column names, so you don’t have to choose whether to do sort (lexicographic) or sort -n (numeric) – that would be in the data rather than the code.
There are already people who use CSV/TSV, XML/HTML, and JSON over pipes. And entire shell-based toolkits and languages around them, like jq, csvkit, etc.
So I think there is a lot of room to improve the shell without changing anything about the kernel. Sure you don’t have metadata for byte streams, but these tools should be strict about parse errors, and it’s very easy to tell a TSV vs. HTML vs. JSON document apart. I think better parse errors would go a long way, and that can be done in user space.
I think I submitted my patch to add -d in 2004 or 2005, so it took OpenBSD over a decade to actually add the feature. I was running a locally patched du for the entire time I was using OpenBSD. That interaction put me off contributing to OpenBSD - the reaction of ‘this feature is a waste of time, you can achieve the same thing with this complicated find expression’ and the hostility with which the message was delivered made me decide not to bother with writing any other code that would require that I interacted with the same people.
Sorry to hear that. In general I find it is extremely difficult to reject patches gracefully, even when you are trying to. It’s one of those things where text loses nuance that would do the job in real life, and so you have to be extra enthusiastic about it. I usually try to start on a positive statement ‘cause that’s what people will see first, something like “I love this, thanks for submitting it. But, there’s no way I can accept this because…”. If it’s a (smallish) bug fix rather than a feature addition I often try to accept it anyway, even if it kinda sucks, and then clean it up myself.
I’m all for technical rigor, but if a project wants to attract contributions, it’s nice to have a reputation for being easy to work with. And it’s better for everyone if people are encouraged to submit patches without them fearing a nasty rejection. You never know when someone sending a patch is a 13 year old self-taught hacker with more enthusiasm than sense, just starting to play with getting involved in the world outside their own head.
How frustrating to receive that response, “it is not being added as it is not part of POSIX”, since -d got added some years later, with the commit message explicitly acknowledging its omission from POSIX standards at the time. :\
On the upside, looks like you had the right intentions all along so I send some kudos your way for trying :)
How frustrating to receive that response, “it is not being added as it is not part of POSIX”, since -d got added some years later, with the commit message explicitly acknowledging its omission from POSIX standards at the time. :\
To be fair, eight years is a long time and the project might have changed its mind about POSIX.
I started this comment by writing that I was not sure the reasoning was inconsistent. David’s patch was rejected in part due to portability concerns. And schwarze’s commit message does mention compatibility with other BSDs and GNU as justification.
But on the other hand, support for -d appeared in NetBSD around the same time as David’s patch and in FreeBSD even before that. Soooo… you’re probably right :-)
haha, yeah I mean, it’s fair to stick to POSIX, but I guess in David’s case it was a matter of the person reviewing the submission being more stringent about that than the person/people who adopted the change later.
Out of curiosity I checked the FreeBSD du.c commit history and found that the -d option was added in Oct 1996! Glad they kept the full commit history upon each transition to a new versioning technology (since I’ve definitely encountered projects where that history was lost). Ah well, anyway, that’s more than I expected to learn about the history of du this week! haha :)
To me, the closest thing to the Unix philosophy today is the Go standard library. It has lots of functions with simple APIs that take io.Readers or io.Writers and lets you compose them into whatever particular tool you need. I guess this makes sense, since it’s from Rob Pike.
The thing about the Unix philosophy is it’s not just about “small” tools that do one thing. It’s about finding the right abstraction that covers a wider range of cases by being simpler. Power comes from having fewer features.
For example, file systems before Unix were more complicated and featureful. We see today that for example, iOS makes files complicated in an old school mainframe like way where some data is in the Photos library, some is locked in a particular app, some needs iTunes to be extracted, and some is actually in the Files app. This might be the right choice in terms of creating a lot of small seemingly simple interfaces instead of one actually simple interface that can do everything, but it makes it much harder to extend things in ways not envisioned by the software developers.
To me, the closest thing to the Unix philosophy today is the Go standard library. It has lots of functions with simple APIs that take io.Readers or io.Writers and lets you compose them into whatever particular tool you need. I guess this makes sense, since it’s from Rob Pike.
Doug McIlroy, the eventual inventor of Unix pipes, wrote in 1964 (!):
We should have some ways of coupling programs like garden hose–screw in another segment when it becomes necessary to massage data in another way. This is the way of IO also.
That is the way of Go also. Go takes that idea and pushes it very far. It is a language of composition and coupling.
It’s fun to think of Go programs like that. Like a website could be a bunch of http handler functions slotted into a mux, and an HTML template parser is wired up to the handers’ response witers. It’s nice to think like that in terms of packages too. My favorite packages are those that I can think of as a “part” that I can hook up to other parts.
The thing about the Unix philosophy is it’s not just about “small” tools that do one thing. It’s about finding the right abstraction that covers a wider range of cases by being simpler. Power comes from having fewer features.
Definitely. I think the “do one thing…” etc sayings kind of miss the point. And io.Writers and io.Readers are definitely that kind of abstaction. They have one method and can therefore easily cover a wide range of types. I think Go’s interfaces work that way in general too, since they grow more specific and “narrow” with more methods and the number of types that implement them shrinks.
I discovered, read and enjoyed this paper recently, and found that it crystallised one of the fundamental design axioms for me into a way that stuck in my head, so wrote it up briefly here yesterday: Unix tooling - join, don’t extend.
The book The UNIX Programming Environment by Kernighan and Pike is also a great read on much the same topic. It’s expensive to buy new but I found a cheap second-hand copy.
The thing is, I find the argument really compelling… but it doesn’t seem to have caught on. Some people really like this style of computing - see the Plan 9 holdouts, for example - but UNIX moved away from it pretty quickly once it left Bell Labs (hence this paper, which uses BSD and SysV changes as examples). GNU is definitely not UNIX, and Windows never was.
It seems like the UNIX style appeals strongly to a few people, but has no mass appeal. Maybe it appeals strongly to developers whose main focus is building software systems, but less to end users - and hence less to developers whose main focus is building software for those end users? Contrast Kernighan and Pike’s paper with Jamie Zawinski’s Unity of Interface, which he described as, “An argument for why kitchen-sink user interfaces (such as web browsers, and emacs) are a good idea.”
Modularity is great if it comes with composability. UNIX originally justified its existence with a typesetting system and the UNIX kind of modularity where you consume a text file and produce a text file was great for that. The only stage that wasn’t plain text was the set of printer control commands that were sent to the printer device (over a stream interface, which looked like a text stream if you squinted a bit). It doesn’t actually work for anything non-trivial.
Here’s a toy example: Consider
ls -Slh. In a UNIX purist world,-lis the only one of these that you actually need inlsand actually you don’t need that because if-lis the default you can get the non-verbose output withls -l | cut -f 9 -wand you can make that a shell alias if you use it often enough. You don’t need-Sbecausesortcan sort things, so you can dols -l | sort -k5. Except that now you’ve gone away from the UNIX spirit a bit because now bothsortandcuthave the functionality to split a line of text into fields. Okay, now that you’ve done that how do you add the-hbit as a separate command? You can do it with someawkthat splits the input and rewrites that one column, but now you’ve added an entire new Turing-complete language interpreter to the mix. You could, in fact, just dols | awkand implement the whole sort part inawkas well. If you do this, you’ll discover thatawkscripts compose a lot better than UNIX pipelines because they have functions that can take structured values.Many years ago, I sent a small patch to OpenBSD’s
duto add the-doption, which I use with around 90% ofduinvocations (du -h -d1is my most commonduinvocation). The patch was rejected because you can implement the-dequivalent with-cand a moderately complexfindcommand. And that’s fine in isolation, but it doesn’t then compose with the otherduflags.The UNIX-Haters Handbook explained the problem very well: We’ve implemented some great abstractions in computing for building reusable components: they’re functions that consume and produce rich data types. Composing systems implemented in terms of these primitives works great. Witness the huge number of C/C++ libraries, Python packages, and so on in the world. In contrast, plumbing together streams of text works well for trivial examples, can be forced to work for slightly more complex examples, and quickly becomes completely impossible for more complex ones.
Even some of the most successful kitchen-sink UIs have been successful because they support composition. Word and Photoshop, for example, each have rich ecosystems of extensions and plugins that add extra features and owe their success in a large part to these features. You can build image-processing pipelines in UNIX with ImageMagick and friends but adding an interactive step in the middle (e.g. point at the bit you want to extract) is painful, whereas writing a Photoshop workflow that is triggered from a selection is much easier. Successful editors, such as vim, emacs, and VS Code, are popular because of their extensions far more than their core functionality: even today, NeoVim and Vim are popular, yet nvi has only a few die-hard users.
Yeah I agree the
sort -k5thing is annoying. Several alternative shells solve that to some extent, like PowerShell, nushell, and maybe Elvish (?). They have structured data, and you can sort by a column by naming it, not by coming up with asortinvocation that re-parses it every time (andsortis weirder than I thought.)However I want a Bourne shell to have it, and that’s Oil, although it’s not done yet. I think what PowerShell etc. do has a deficiency in that it creates problems of composition. It doesn’t follow what what I’m calling the Perlis-Thompson principle (blog posts forthcoming).
So Oil will support an interchange format for tables, not have an in-memory representation of tables: https://github.com/oilshell/oil/wiki/TSV2-Proposal (not implemented)
I also think the shell / awk split is annoying, and Shell, Awk, and Make Should Be Combined.
Unfortunately, it’s not really something that you can solve in a shell because the problem is the set of communication primitives that it builds on. PowerShell is incredibly badly named, because it’s not really a shell (a tool whose primary purpose is invoking external programs), it’s a REPL. It has the converse problem: there’s no isolation between things in PowerShell.
Something like D-BUS would be a reasonably solid building block. If the shell managed a private D-BUS namespace for the current session then you could write composed commands that run a bunch of programs with RPC endpoints in both directions and some policy for controlling how they go together. You could then have commands that exposed rich objects to each other.
’d be quite curious to know what a shell based around the idea of exporting D-BUS objects would look like, where if you wrote
foo | barthen it would invoke both with D-BUS handles in well-known locations for each, rather than file descriptors for stdin / stdout. For example, a shell might provide a/shell/pipeline/{uuid}namespace and set an environment variable with that UUID and another with the position in the pipeline for each process so thatfoowould expose something in/shell/pipeline/3c660423-ee16-11eb-b8cf-00155d687d03/1and look for something exposed as/shell/pipeline/3c660423-ee16-11eb-b8cf-00155d687d03/2, expose the/0and/3things for input from and output to the terminal (and these could be a richer set of interfaces than a PTY, but might also provide the file descriptor to the PTY or file directly if desired). Each shell session might run its own D-BUS bus or hook into an existing one and the shell might also define builtins for defining objects from scripts or hooking in things from other places in a D-BUS system.(very late reply, something reminded me of this comment recently)
I don’t see why you need DBus or anything similar to solve the
sort -k5problem? I think you just need something like TSV over pipes, and then asort-bytool that takes a column name to sort by. An enhancement would be to attach types to column names, so you don’t have to choose whether to dosort(lexicographic) orsort -n(numeric) – that would be in the data rather than the code.There are already people who use CSV/TSV, XML/HTML, and JSON over pipes. And entire shell-based toolkits and languages around them, like jq, csvkit, etc.
Projects listed at the end of this page: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
So I think there is a lot of room to improve the shell without changing anything about the kernel. Sure you don’t have metadata for byte streams, but these tools should be strict about parse errors, and it’s very easy to tell a TSV vs. HTML vs. JSON document apart. I think better parse errors would go a long way, and that can be done in user space.
Interestingly enough, OpenBSD added the -d option to
dusome years ago: https://cvsweb.openbsd.org/src/usr.bin/du/du.c?rev=1.26&content-type=text/x-cvsweb-markupAnd the reason is to be compatible with other BSDs and Linux and so we’re back to the original topic ;)
I think I submitted my patch to add
-din 2004 or 2005, so it took OpenBSD over a decade to actually add the feature. I was running a locally patcheddufor the entire time I was using OpenBSD. That interaction put me off contributing to OpenBSD - the reaction of ‘this feature is a waste of time, you can achieve the same thing with this complicatedfindexpression’ and the hostility with which the message was delivered made me decide not to bother with writing any other code that would require that I interacted with the same people.Sorry to hear that. In general I find it is extremely difficult to reject patches gracefully, even when you are trying to. It’s one of those things where text loses nuance that would do the job in real life, and so you have to be extra enthusiastic about it. I usually try to start on a positive statement ‘cause that’s what people will see first, something like “I love this, thanks for submitting it. But, there’s no way I can accept this because…”. If it’s a (smallish) bug fix rather than a feature addition I often try to accept it anyway, even if it kinda sucks, and then clean it up myself.
I’m all for technical rigor, but if a project wants to attract contributions, it’s nice to have a reputation for being easy to work with. And it’s better for everyone if people are encouraged to submit patches without them fearing a nasty rejection. You never know when someone sending a patch is a 13 year old self-taught hacker with more enthusiasm than sense, just starting to play with getting involved in the world outside their own head.
Interesting, found the original mailing list response where the “rejection” was given: https://marc.info/?l=openbsd-tech&m=115649277726459&w=2
How frustrating to receive that response, “it is not being added as it is not part of POSIX”, since
-dgot added some years later, with the commit message explicitly acknowledging its omission from POSIX standards at the time. :\On the upside, looks like you had the right intentions all along so I send some kudos your way for trying :)
To be fair, eight years is a long time and the project might have changed its mind about POSIX.
I started this comment by writing that I was not sure the reasoning was inconsistent. David’s patch was rejected in part due to portability concerns. And schwarze’s commit message does mention compatibility with other BSDs and GNU as justification.
But on the other hand, support for
-dappeared in NetBSD around the same time as David’s patch and in FreeBSD even before that. Soooo… you’re probably right :-)haha, yeah I mean, it’s fair to stick to POSIX, but I guess in David’s case it was a matter of the person reviewing the submission being more stringent about that than the person/people who adopted the change later.
Out of curiosity I checked the FreeBSD
du.ccommit history and found that the -d option was added in Oct 1996! Glad they kept the full commit history upon each transition to a new versioning technology (since I’ve definitely encountered projects where that history was lost). Ah well, anyway, that’s more than I expected to learn about the history ofduthis week! haha :)To me, the closest thing to the Unix philosophy today is the Go standard library. It has lots of functions with simple APIs that take io.Readers or io.Writers and lets you compose them into whatever particular tool you need. I guess this makes sense, since it’s from Rob Pike.
The thing about the Unix philosophy is it’s not just about “small” tools that do one thing. It’s about finding the right abstraction that covers a wider range of cases by being simpler. Power comes from having fewer features.
For example, file systems before Unix were more complicated and featureful. We see today that for example, iOS makes files complicated in an old school mainframe like way where some data is in the Photos library, some is locked in a particular app, some needs iTunes to be extracted, and some is actually in the Files app. This might be the right choice in terms of creating a lot of small seemingly simple interfaces instead of one actually simple interface that can do everything, but it makes it much harder to extend things in ways not envisioned by the software developers.
Yeah, he mentioned this in “Less is Exponentially More”:
It’s fun to think of Go programs like that. Like a website could be a bunch of http handler functions slotted into a mux, and an HTML template parser is wired up to the handers’ response witers. It’s nice to think like that in terms of packages too. My favorite packages are those that I can think of as a “part” that I can hook up to other parts.
Definitely. I think the “do one thing…” etc sayings kind of miss the point. And io.Writers and io.Readers are definitely that kind of abstaction. They have one method and can therefore easily cover a wide range of types. I think Go’s interfaces work that way in general too, since they grow more specific and “narrow” with more methods and the number of types that implement them shrinks.
I discovered, read and enjoyed this paper recently, and found that it crystallised one of the fundamental design axioms for me into a way that stuck in my head, so wrote it up briefly here yesterday: Unix tooling - join, don’t extend.
It was from your blog post that I found the original paper :)
That’s made my lunchtime :-)