Meh I don’t think that is a fair summary. It’s pointing out some problems with shell, some good parts, and explaining 2 current research projects.

does this protocol require support from the kernel

Yes, it’s a kernel feature. That said, it’s only a couple of dozen lines of code. Implementing it in Linux would be fairly easy. I’ll probably do it at some point.

do I assume correctly that there’s a typo, and the second part should read: the receiver issues … to receive?

Uh, yes. Ho hum. The sender specifies the things it can produce, the receiver selects the one that it wants.

I proposed this at a FreeBSD DevSummit many years ago and the consensus was it was pretty interesting but no one had time to work on it. I put together the prototype during a Microsoft Hackathon a couple of years back. It’s not currently in a state to be upstreamed because it does not currently bound the amount of kernel memory that it can use for the buffer that holds the types. When I have a few spare days to work on it, I’ll try to get it finished and ready to upstream. The entire diff is under 1KLoC, including the libxo changes, some libc wrappers that provide a more friendly interface than the raw ioctls, and some really long comments explaining the state machine.

I’d like to do a Linux version and the ssh changes to demonstrate that it’s a portable abstraction before I try to upstream it anywhere.

I’ve determined that it is impossible to do this cleanly without involving the kernel.

Thirdly, terminal developers seem an Opinionated lot, and maybe it would be best to not give cause

Frankly, to make any progress at all with regard to shells, maybe we need a clean break from the people who have been clinging to “a stream of bytes” is all you need for half a century.

There might be a flaw in this ioctl design if negotiating the communication protocol doesn’t peel off a separate pipe: something may print garbage to stdout, eg due to a library-specific debug environment variable, an unexpected error, a stray debug printf, or a noisy library.

Can you explain this a bit more?

Also, consider this scenario: { command1; command2 } | { command3; command4 }. What happens if the commands mix protocol support? What should if one of them gets SIGKILL in the middle of writing a long packet - undefined behavior?

That’s an interesting case. I’ve never seen something like that in the wild, so didn’t consider it. I resume that command1 and command2 are run in sequence with the same pipe as stdout and command3 and command4 have the same pipe as standard input? In my implementation, only command1 and command2 would have the opportunity to do content negotiation, but command4 would need to handle whatever they negotiated.

That’s not great, but it’s no worse than today where none of the commands have any opportunity to negotiate. You can always provide command-line options to specify input / output formats if you don’t want to use auto-negotiation (and if you’re building a pipeline like this, you’re probably providing them already). If you want to guarantee no content negotiation, you can do {command1 ; command2} | cat | {command3 ; command4}. That does a redundant copy, but you could also insert something before command3 that breaks out of the content-negotiation protocol.

One of my main use cases for this was integration with GUI clipboard / drag-and-drop infrastructure. On macOS, for example, there’s a pbcopy and pbpaste command, but you need to explicitly pass a type to pbpaste if you want to paste anything other than plain text. I want to be able to write pipelines like pbpaste | svg2png -whatever | pbcopy and have pbpaste automatically select the SVG representation from the pasteboard and pbcopy set the contents as PNG.

I must strongly discourage thought of embedding of web views.

This was just an example, I think the JSON / XML output is more interesting for things like screen readers. Try typing ls in a terminal with a screen reader sometime to see the problem here! Apparently most blind people turn their rate up really high and listen for patterns in the output, which is a pretty awful experience.

Fourthly, as you know there are a lot of terminal emulators in a lot of weird places; the kernel implements one for virtual terminals, vim has a fake one, phones, browsers, tmux. If the typical command is spittin’ HTML+JS the question of polyfilling has nasty answers.

The kernel mechanism just provides a communication channel to the terminal emulator. Once you’ve peeled off a pipe, it runs the same content negotiation protocol as any other pipe and it’s fine to just not generate the additional formatted data if the terminal emulator just wants to display terminal output. You can imagine something like vim or an IDE using this to get FixIt hints in the LSP format on a secondary stream when compiling, for example.