I find with these ideas the more I play around with them, the more cool stuff I discover.
With so much tech these days, you put in a lot of work to master one thing, only to have it changed and upgraded next year. Or you realize there’s yet something else you have to add to your stack.
With the Unix Philosophy, the more I know, the more I’m able to know. It’s like a snowball. This is much more fun!
Nice article! One of the most prominent examples that comes to mind of Unix style composition is SSH being an authenticated and encrypted pipe.
First we had scp and rsync using the SSH transport. Nowadays it’s more often git or another SCM.
I think it also works this way on Windows – i.e. doesn’t git on windows use ssh? Windows probably has some kind of shared library for encrypted networking, and shared library for auth. But I think composition using SSH is more straightforward and “won”.
So ssh also solves an O(M * N) problem. On the one hand you have applications; on the other you have auth and encryption methods (or maybe it’s O(M * N * K) ).
I don’t remember all the details but I know SSH supports a bunch of key formats. It also supports password auth and key auth. And I guess the point is that I don’t really have to rememnber all the details :) They are “upgraded” within SSH, and made more secure, but scp, rsync, git, and hg don’t need to change.
Some other notes:
Agree that coprocesses are fiddly! A lot of Unix is old and crufty but it still composes better than alternatives. We need new thinking (or at least new implementation) on making data-centric and composable systems.
Why is json-rpc-shell and the linked projects separate from the system shell? It looks like it parses | and then pipes to popen(), which just opens a shell? I think the reason may be that they need to hold network connections open? Are there other reasons?
For Oil, I’ve designed a different version of coprocess than bash and ksh that I think may help solve this problem… i.e. you should have the coprocesses hold network connections and not the shell itself, and they can be reused.
For Windows, read up on PSSessions, They allow AAA encrypted connections with structured data transport for one-shot commands or long-running sessions that are shared between commands in a shell. I think newer versions of PowerShell will use SSH as a transport layer for PSSessions though.
The JSON-RPC shell started separate with the motivation of not having to repeat myself (not even thinking about a shell read loop), but yes, I often think about this as well. The connection can also keep state, which I’ve used with WebSocket on a company project. And the WebSocket specification doesn’t want you to interrupt connections anyway. A language server protocol server is also something you might want to keep to get anywhere.
It might be possible to split it out, yet consider a full example:
json-rpc> notify {"events": ["war", "famine", "death"]} | tee response
(I thought this would be hard to parse without clobbering the f() { echo func; } syntax, but I have an idea for how to do it now.)
Combined with the coprocesses, I hope you can build something like a JSON-RPC shell in Oil.
I actually worked on something like this 5+ years ago – it was a shell to talk to Google’s distributed file system and RPC services. Basically because they added a bunch of auth that was crazy slow. It would take 5-10 seconds to do auth, so every shell command would have a 5-10 second overhead.
So the obvious solution is to maintain the connection inside the shell, and someone had done that, and I extended it. But now I think it’s better to use coprocesses because you don’t necessarily want 50 different network connections for 10 services inside your shell. It’s more robust to delegate that to another process.
One of the biggest problems with this stream based composibility is similar to another problem that also is on the front page. It’s possible to conceive of edge cases in the exchange of these streams that can cause catastrophic failure.
Not sure what can be done from a GUI standpoint. I think just having a live visualization of changes would be a good first step, but I’m not sure things that are naturally code-related can ever be made into a gui. Maybe something like Node-RED sort of fits?
Also, since no one else is saying it; you probably want to avoid linking to code that uses blatantly anti-semetic slur. Even the body of your article mentions an irc bot named “ZyklonB.”
I probably want to, on the other hand it’s just the result of a joke on letter iteration. git filter-branch because of something so silly? Shrug. Though I might have just come up with even stupider names for the bunch.
GUI composition at some level is easier to fake on an opinionated window manager like a tiling window manager. Pop up a second window and communicate back and forth. What makes this feel so janky on overlapping windows is the two windows can more easily get lost/separated from each other and then you aren’t looking at both and it gets confusing.
This article just sort of happened, because I couldn’t stop myself from thinking about it.
Those are the best kind :)
I find with these ideas the more I play around with them, the more cool stuff I discover.
With so much tech these days, you put in a lot of work to master one thing, only to have it changed and upgraded next year. Or you realize there’s yet something else you have to add to your stack.
With the Unix Philosophy, the more I know, the more I’m able to know. It’s like a snowball. This is much more fun!
Nice article! One of the most prominent examples that comes to mind of Unix style composition is SSH being an authenticated and encrypted pipe.
First we had scp and rsync using the SSH transport. Nowadays it’s more often git or another SCM.
I think it also works this way on Windows – i.e. doesn’t git on windows use ssh? Windows probably has some kind of shared library for encrypted networking, and shared library for auth. But I think composition using SSH is more straightforward and “won”.
So ssh also solves an O(M * N) problem. On the one hand you have applications; on the other you have auth and encryption methods (or maybe it’s O(M * N * K) ).
I don’t remember all the details but I know SSH supports a bunch of key formats. It also supports password auth and key auth. And I guess the point is that I don’t really have to rememnber all the details :) They are “upgraded” within SSH, and made more secure, but scp, rsync, git, and hg don’t need to change.
Some other notes:
|and then pipes to popen(), which just opens a shell? I think the reason may be that they need to hold network connections open? Are there other reasons?For Windows, read up on PSSessions, They allow AAA encrypted connections with structured data transport for one-shot commands or long-running sessions that are shared between commands in a shell. I think newer versions of PowerShell will use SSH as a transport layer for PSSessions though.
The JSON-RPC shell started separate with the motivation of not having to repeat myself (not even thinking about a shell read loop), but yes, I often think about this as well. The connection can also keep state, which I’ve used with WebSocket on a company project. And the WebSocket specification doesn’t want you to interrupt connections anyway. A language server protocol server is also something you might want to keep to get anywhere.
It might be possible to split it out, yet consider a full example:
The syntax can be wild.
Ah OK, I think Oil can accomodate the syntax for JSON. It actually has a similar problem because there is a JSON bulltin in Oil.
I originally thought it would be square brackets (in part because the arg list is lazily and not eagerly evaluated):
But now I think it should be a more traditional parens:
Noted here: https://github.com/oilshell/oil/issues/955
(I thought this would be hard to parse without clobbering the
f() { echo func; }syntax, but I have an idea for how to do it now.)Combined with the coprocesses, I hope you can build something like a JSON-RPC shell in Oil.
I actually worked on something like this 5+ years ago – it was a shell to talk to Google’s distributed file system and RPC services. Basically because they added a bunch of auth that was crazy slow. It would take 5-10 seconds to do auth, so every shell command would have a 5-10 second overhead.
So the obvious solution is to maintain the connection inside the shell, and someone had done that, and I extended it. But now I think it’s better to use coprocesses because you don’t necessarily want 50 different network connections for 10 services inside your shell. It’s more robust to delegate that to another process.
Comments on related PB&J distributed processing paper which mentions the need for a JSON shell: http://www.oilshell.org/blog/2021/07/cloud-review.html#january-2017-project-goals
Thanks for the info!
On a tangent this just got me to learn about C++11 trailing return types. Whoa!
On the top level like that, it’s actually a C++14 feature, see Return type deduction.
One of the biggest problems with this stream based composibility is similar to another problem that also is on the front page. It’s possible to conceive of edge cases in the exchange of these streams that can cause catastrophic failure. Not sure what can be done from a GUI standpoint. I think just having a live visualization of changes would be a good first step, but I’m not sure things that are naturally code-related can ever be made into a gui. Maybe something like Node-RED sort of fits?
Also, since no one else is saying it; you probably want to avoid linking to code that uses blatantly anti-semetic slur. Even the body of your article mentions an irc bot named “ZyklonB.”
I probably want to, on the other hand it’s just the result of a joke on letter iteration.
git filter-branchbecause of something so silly? Shrug. Though I might have just come up with even stupider names for the bunch.I’m not sure what you mean, since those attacks appear HTTP/2 specific, and don’t relate to HTTP 1, which is also based on text streams. Example?
GUI composition at some level is easier to fake on an opinionated window manager like a tiling window manager. Pop up a second window and communicate back and forth. What makes this feel so janky on overlapping windows is the two windows can more easily get lost/separated from each other and then you aren’t looking at both and it gets confusing.