I don’t think she should add that to her post, because it’s beside the point: her post is about program interaction with the terminal, and -- neither turns on nor turns off interaction with the terminal. Mentioning -- would take her off into the weeds of command-line argument conventions, which is an entirely different topic. A worthy topic, but a different one.
But - is entirely on-topic for her post: it explicitly connects a program to its stdin or stdout when you might normally think, if you didn’t know the - convention, that a program will only talk to plain files. She might, though, have usefully mentioned that /dev/stdin and /dev/stdout can be used as filenames, even with programs that don’t know and don’t honor the - convention!
Guideline 13:
For utilities that use operands to represent files to be opened for either reading or writing, the ‘-’ operand should be used only to mean standard input (or standard output when it is clear from context that an output file is being specified).
At my last job we had a bit of a tug of war between users like myself who prefer readline vi keybindings and those that prefer the default emacs keybindings. When trying to solve the problem I discovered that the majority of the key bindings do not overlap, so I create a merged config, https://github.com/lollipopman/bash-rsi
Not the focus of this particular blog post, but Julia Evans wrote another terminal blog post a while back which makes note of this under the topic of “Entering Text in the Terminal” https://jvns.ca/blog/2024/07/08/readline/ (the vim setting comes up later in the post than the readline sunb-heading).
Personally I think changing this readline behavior would throw me off more than actually switching my default shell.
TUI programs (like less or htop) will usually quit when you press q.
I guess vi would like a word ;)
By the way another convention that we take for granted is preserving the main screen by using the alternate one while in TUI mode. Most programs do this, but for example more and top do not. Some BSD implementations of less and vi also don’t.
I hate that convention of less. There are plenty of times when I’m just paging through a file and want to use some information with another command. I hit “q” and then—poof the info is now gone.
It’s weird how much of this is voluntarily implemented by the applications, rather than being an OS interface. Every application has to detect terminal and control color output itself. Every application performs arg parsing itself. The closest these thing get to being a standard OS interface is having a couple of GNU libraries with POSIX’s blessing.
Every application has to detect terminal and control color output itself. Every application performs arg parsing itself. The closest these thing get to being a standard OS interface is having a couple of GNU libraries with POSIX’s blessing.
Except that this isn’t really true. UNIX systems generally provide curses (or ncurses, etc) and terminfo libraries as part of their base OS facilities; these abstract over terminal type detection and emitting formatted output. They also provide, e.g., getopt(), which is standardised in POSIX, for argument parsing.
It’s true that you need not use any of these things if you don’t want to, for whatever reason. But that’s a good thing, because it allows people to experiment with improvements to those facilities, or even just different approaches, without requiring a change to some mandatory OS mechanism.
The POSIX getopt is very limited (single-character options only). The longer options are supported by getopt_long, but that’s a GNU extension. It’s fairly widely supported now but it wasn’t 10-20 years ago.
Similarly, terminfo vs termcap is still an issue, though most termcap systems let you install terminfo separately. Curses came with many flavours, but it is also an absolutely appalling API and so most TUI things reimplement it rather than using it.
The Windows console APIs are much nicer than the *NIX model of throw a load of in-band signalling through a stream and hope that the thing on the other end does something sensible with it.
Like I said, I think it’s good that people can experiment and try new things or alternative approaches and implementations. As far as I know, that’s not really an option with the Windows model!
It’s a matter of taste to say that the Windows stuff is nicer, but I believe it’s a matter of fact to say (despite my own distaste in general for in-band signalling) that the UNIX terminal model is one of the longest standing and most successful case studies in backwards-compatibility and cross-platform interoperability, across a vast array of hardware and software, over many decades, that we have in the field. It’s often not fantastic in many dimensions but we’ve come a long way with it and continue to incrementally improve the ecosystem to meet new challenges, and the openness and practicality of that ecosystem is a lot nicer (for me personally) than anything in the relative backwater that is the Windows console API.
That’s what I’ve meant by a couple of gnu libraries with a POSIX blessing.
And behaviors like turning off colors when piping still need to be implemented by the application itself.
It kinda works anyway, and most programs are remarkably consistent, but that’s achieved not with APIs and abstraction layers, but by filing bug reports (like these) with every app/library until they fix it.
A port that reads from or writes to a terminal has a terminal mode; this is either cooked or raw. …
A terminal port in cooked mode provides some standard processing to make the terminal easy to communicate with. For example, under unix, cooked mode on input reads from the terminal a line at a time and provides editing within the line, while cooked mode on output might translate linefeeds to carriage-return/linefeed pairs. In general, the precise meaning of cooked mode is operating-system dependent, and furthermore might be customizable by means of operating-system utilities. The basic idea is that cooked mode does whatever is necessary to make the terminal handle all of the usual user-interface conventions for the operating system, while keeping the program’s interaction with the port as normal as possible.
A terminal port in raw mode disables all of that processing. In raw mode, characters are directly read from and written to the device without any translation or interpretation by the operating system. On input, characters are available as soon as they are typed, and are not echoed on the terminal by the operating system. In general, programs that put ports in raw mode have to know the details of interacting with the terminal. In particular, raw mode is used for writing programs such as text editors.
[programs] don’t always mimic [emacs/readline keybindings] exactly: for example atuin seems to use Ctrl-A as a prefix, so Ctrl-A doesn’t go to the beginning of the line.
I have been wondering about this for a long time, perhaps someone knows… Why does GNU Screen use C-a and tmux use C-b as their default prefixes, given the long history, wide spread, and useful function of these bindings as beginning-of-line and backward-char?
The exceptions to this are:
some programs (like git, cat, and nc) don’t have any line editing support at all (except for backspace, Ctrl-W, and Ctrl-U)
Because they are not interactive, work in the mentioned earlier cooked mode, and rely on a TTY/PTY driver when not used in pipelines? Also curious: why git is mentioned? I can’t think of its command that would read stdin and expect it from a user (vs. another programme).
The most frustrating inconsistency for me is handling of cmd-line flags. Is -akx one flag? or multiple? This is one of my annoyances with terraform–it feels pretty lopsided how most follow the short vs long flag convention. I don’t know why some programs go out of their way to misalign with it.
I deeply appreciate clap for making the standard convention available on a very fast runtime (i’m sure there are similar things in other langs–it’s just the one i’m most familiar with; it certainly helps considering the tendency for folks to want to rewrite everything in rust :^) )
What a blast from the past. I wish I’d came across something like this back in day when I was learning this stuff. This falls into the category of things that ‘make it click’s but that aren’t documented anywhere. They’re rather a part of culture, common practices… Well… unwritten rules.
This should be included in guides of how to use the command line, shellscript tutorials, etc
In regards to rule 6 and coloring, is there any shell smart enough to determine when the sole output of a bunch of pipes is the terminal, so it can enable color throughout? (I know you wouldn’t always want this, but I’d prefer it as the default.)
I’m not sure that would work because the intermediate stages might not handle colours. For example, if you pipe through short or awk, it may get confused by the escape sequences and shorty by the control code of the colour escape sequences rather than the text.
Somewhere, I have some FreeBSD patches I should dig out that added the ability to send multiple streams and do per-stream content negotiation to TTYs. This borrowed the same protocol that drag and drop commonly uses, where the sender advertises support for a set of types and the receiver picks the preferred one. I extended libxo (the library that produces pretty coloured output, JSON, XML, or HTML for most FreeBSD command-line tools) to offer these types and let the receiver pick one, so you could pipe to something that would collect HTML and open a web browser with it (or integrate that in the shell or terminal emulator, so it could render HTML inline).
Arcan had a more principled version of this, but my implementation was easy to retrofit. The libxo changes added up to about five lines of code.
I’m well aware that the intermediate stages might not know what to do with escape sequences. What I was envisioning is a bit more like the example in my other comment, where I run something ls | grep | head, and the shell goes “Aha! I know the color options for all of those”, and inserts them on my behalf.
Obviously not universally applicable or foolproof, but anything is more useful than what we have now, which is the loss of color as soon as you hit ‘|’ (unless you add all the flags manually, which I’ll bet a kidney most of us aren’t, especially as the pipe gets longer.)
This piqued my curiosity again, so I tried ls --color=always | grep --color=always … — it’s unpredictable in a bad way.
For instance, in my example test, src is a directory and ls has coloured it blue. If I grep for s, then grep colours the s in red, and the rest of the word is white. If I grep for r, then we have a blue s, red r, and white c. This took me a moment before I thought about the control sequences involved.
I’m sure there’s still instances where it’s neat, but this kind of behaviour makes me nope out.
Besides the example you give, there is another problem with the isatty heuristic – sometimes (often? usually?) you want to pipe output to a PAGER like less -R that can handle color codes. When that happens, you need to look up how to do the equivalent of --color=always. So, I just default to color always behavior but provide a --plain instead, honor $NO_COLOR, and I also wrote noc so you can just stuff those 4 characters |noc inside some pipeline to get no-color like streams.
Of course, as others in this subthread have pointed out, then you cannot “re-colorize” things because the thing understanding the semantics has lost control. The whole situation is a mess. Roughly, the “in-band nature” of the coloring of text cells rather than some kind of out-of-band second stream breaks that old chestnut “Write programs to handle text streams, because that is a universal interface.” To preserve color it seems like you would still need a way to correlate text cells in the two streams (e.g. coordinates of some kind) which probably then corrupts the internal simplicity of some text filters. Not sure if @david_chisnall’s libxo mentioned here or Arcan are the answers or really address the “whole system of programs problem”, TBH, or just one or two aspects. In other words, is it a bug or a feature that “grep” need not negotiate its IO formats with its two neighbors? Many a grep-author might say feature / performance necessity, but many an end-user might differ.
arcan-tui doesn’t inband anything on top of stdio and do address the ‘whole system’. All window management, interaction, clipboard, accessibility, internationalisation, alternate stream setup etc. goes over a sideband per window. The libarcan-tui consumer has access to a semantic palette and a legacy (red is now green problem) palette for when we do need to spawn a pty to compose a client that do check isatty).
The outer display server sends the preferred palette (and available glyph set) that maps to the semantic one on first connection. It can remap this at any time, and if that happens, there is an event fired (recolor), though this is mainly used by the terminal emulator written for it.
When you signal that you are finished drawing contents, the screen resolves colours according to the current palette into RGB values and writes that into a packing format, TPACK, that is also used for screenshots and screenrecording. The format itself contains a header on cursor information; if it is an I frame or a P frame and a set of rows with shaping information/offset and the cells to for that row.
You can see that in this old (2021) clip: https://videos.files.wordpress.com/JtFbZP4u/clip_11_migrate_mp4_hd.mp4 the screen on the left with the white window is a tui client. When I migrate the window to another machine (the laptop on the right), which has a different colour scheme preference, the background turns dark.
Very neat seeming! Thanks for the detailed reply. Can Arcan handle “layers” such as if you have a mygrep that wants to boldface or inverse or otherwise tweak/color/attribute transform “matching text” of already syntax-highlighted text (within a pipeline) feeding it to then something else (maybe only potentially the text window, maybe another pipeline stage)? Just some simple API mygrep.c can call to do that? Is the communication bi-directional? Can this “layering pattern matcher” program query the display server for the existing rendition and then just do its own “color inversion” (like inverse text in ANSI, whatever “inverse” really means here..)? (EDIT: BTW, I feel the answer will be “but of course!”, but I thought I should ask anyway.)
Everytime you discuss Arcan, it always sounds like a really complete solution and I feel I should give it an earnest try, but never do. Maybe this time will be different. :-)
I’m not sure if we have the same idea of ‘layers’ here - the screen as it stands is single-layer. There are basic provisions in place for creating another window and embedding that as an overlay (or explicit copy between two windows but that is less exciting), but it has not been used for text purposes.
You can see around the 0:48 mark (and before then, side-band transfer of a large:ish file pasted into the shell without blocking it, take that ‘block paste’ mode) where the video clip, as well as, a pdf viewer gets embedded. The shell (as in a tui client here) has control over its embedding and scaling properties, gets feedback (so bidirectional) about rows/cells consumed but doesn’t reason or have access to the per-pixel view of the contents yet the outer windowing system is free to decompose it. The option to say, do that to another tui window follow the same mechanisms - albeit no blending options are exposed.
As far as per cell formatting attributes go, you have the standard underline/alt-underline/inverse/bold as a bitmask (together with border_t,l,d,r to avoid wasting cell space on line-drawing characters, hence why the spreadsheet in that other demo doesn’t waste half its height on drawing borders). Then it’s up to the rendering side how that should resolve.
It’d be pretty hard for the shell to do that — it’d either have to know what options to add to each invoked command (erring on the side of safety, lest your shell make your working command into an incorrect one), or maybe set an environment variable (which those programs would have to opt into). Programs detect whether or not to enable colour automatically themselves for the most part (by checking what kind of thing stdout is with isatty(3)) — perhaps they could follow the pipes (!).
This also doesn’t work very well anyway; imagine you do grep blah | sed thingy, and grep gets colours enabled for it automatically. Then your sed expression happens to trash half of those ANSI codes. Putting colours in works if you’re only piping it to head/tail/cat/things of that nature, but you don’t have to do much interesting before it’d grate.
Sure, the universal case might be difficult/impossible, but the 80%-good-enough case might not be. And I’d have no problems falling back to a non-color version if I get gibberish.
Not even 80%, really. Maybe 50% of the time, I just want to take some colored output and do some light filtering/paging on it without losing color or looking up how to do it myself.
A shell figuring out I want color in ls | grep | head is what I want.
An allowlist of known-good programs and the way to chain them could still get us pretty far.
An allowlist of known-good programs and the way to chain them could still get us pretty far.
Yeah, absolutely. I wonder which existing shell would be easiest to try hooking something this onto.
I have an inner feeling of ‘ick’, and I guess what that’s telling me is that this (perfectly reasonable) thing we desire is really quite hard to do with the current design of the terminal stack (y’know, at least to do it well; it’s very clearly not designed to support it), and leads to wondering about aiming higher.
(Put another way, I suppose PowerShell could do this kind of thing pretty well ..)
Thanks for mentioning FORCE_COLOR. I hadn’t heard of it. Notice the asymmetry - while it is possible to write a little pipeline utility to strip ANSI color codes {mentioned in my other comment }, putting the color in the stream in the first place requires context-specific semantics the generating but not receiving (or possibly transforming) program knows.
If the shell arbitrarily lies about stdout being a tty or not, then knowing how a particular set of commands will work (in a script when pasted from a set of instructions, etc.) becomes indeterminate. If you want this, then you want to change your communication channel between the elements of the pipe to something other than pure text. This is what PowerShell and Nushell both do to varying degrees.
The 80% good-enough case you mention below is not good enough for something that has to work correctly 100% of the time, and the only right way to handle this in general terms for shells which pass text between them is to make it explicitly opt-in. There’s no good way to do this as opt-out.
Minor addition to rule 7: -- means end of arguments
This way you can handle filenames starting with a dash:
I don’t think she should add that to her post, because it’s beside the point: her post is about program interaction with the terminal, and
--neither turns on nor turns off interaction with the terminal. Mentioning--would take her off into the weeds of command-line argument conventions, which is an entirely different topic. A worthy topic, but a different one.But
-is entirely on-topic for her post: it explicitly connects a program to its stdin or stdout when you might normally think, if you didn’t know the-convention, that a program will only talk to plain files. She might, though, have usefully mentioned that/dev/stdinand/dev/stdoutcan be used as filenames, even with programs that don’t know and don’t honor the-convention!I think a lot of the keybinds that readline uses that come from emacs are because readline by default uses emacs mode; you can change it to use vi mode: https://www.gnu.org/software/bash/manual/html_node/Readline-vi-Mode.html
(annoyingly, programs that are only emulating readline’s default behavior will not be affected by this)
Also, the - convention seems to be a POSIX-ism too: https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap12.html#tag_12_02
The - convention goes back to the original getopt written by Aaron S Cohen in 1979 https://lobste.rs/s/rzmesn/getopt_smaller
At my last job we had a bit of a tug of war between users like myself who prefer readline vi keybindings and those that prefer the default emacs keybindings. When trying to solve the problem I discovered that the majority of the key bindings do not overlap, so I create a merged config, https://github.com/lollipopman/bash-rsi
Not the focus of this particular blog post, but Julia Evans wrote another terminal blog post a while back which makes note of this under the topic of “Entering Text in the Terminal” https://jvns.ca/blog/2024/07/08/readline/ (the vim setting comes up later in the post than the readline sunb-heading).
Personally I think changing this readline behavior would throw me off more than actually switching my default shell.
From me personally I’d add to the rule:
Pretty please.
To extend this with a super in depth look at these conventions see The Command Line Interface Guide: https://clig.dev/
I guess
viwould like a word ;)By the way another convention that we take for granted is preserving the main screen by using the alternate one while in TUI mode. Most programs do this, but for example
moreandtopdo not. Some BSD implementations oflessandvialso don’t.Did you read the next line?
Uh, apparently not :-/
I hate that convention of
less. There are plenty of times when I’m just paging through a file and want to use some information with another command. I hit “q” and then—poof the info is now gone.Try setting LESS=X (or passing it -X).
It’s weird how much of this is voluntarily implemented by the applications, rather than being an OS interface. Every application has to detect terminal and control color output itself. Every application performs arg parsing itself. The closest these thing get to being a standard OS interface is having a couple of GNU libraries with POSIX’s blessing.
Except that this isn’t really true. UNIX systems generally provide curses (or ncurses, etc) and terminfo libraries as part of their base OS facilities; these abstract over terminal type detection and emitting formatted output. They also provide, e.g., getopt(), which is standardised in POSIX, for argument parsing.
It’s true that you need not use any of these things if you don’t want to, for whatever reason. But that’s a good thing, because it allows people to experiment with improvements to those facilities, or even just different approaches, without requiring a change to some mandatory OS mechanism.
The POSIX
getoptis very limited (single-character options only). The longer options are supported bygetopt_long, but that’s a GNU extension. It’s fairly widely supported now but it wasn’t 10-20 years ago.Similarly, terminfo vs termcap is still an issue, though most termcap systems let you install terminfo separately. Curses came with many flavours, but it is also an absolutely appalling API and so most TUI things reimplement it rather than using it.
The Windows console APIs are much nicer than the *NIX model of throw a load of in-band signalling through a stream and hope that the thing on the other end does something sensible with it.
Like I said, I think it’s good that people can experiment and try new things or alternative approaches and implementations. As far as I know, that’s not really an option with the Windows model!
It’s a matter of taste to say that the Windows stuff is nicer, but I believe it’s a matter of fact to say (despite my own distaste in general for in-band signalling) that the UNIX terminal model is one of the longest standing and most successful case studies in backwards-compatibility and cross-platform interoperability, across a vast array of hardware and software, over many decades, that we have in the field. It’s often not fantastic in many dimensions but we’ve come a long way with it and continue to incrementally improve the ecosystem to meet new challenges, and the openness and practicality of that ecosystem is a lot nicer (for me personally) than anything in the relative backwater that is the Windows console API.
But then Windows doesn’t have a history of attaching terminals to the serial ports of the computer, and thus, can avoid in-band signalling.
The Windows console API is pretty heavily based on the VGA hardware’s text mode, unsurprising given its heritage!
That’s what I’ve meant by a couple of gnu libraries with a POSIX blessing.
And behaviors like turning off colors when piping still need to be implemented by the application itself.
It kinda works anyway, and most programs are remarkably consistent, but that’s achieved not with APIs and abstraction layers, but by filing bug reports (like these) with every app/library until they fix it.
What is meant by ‘cooked mode’ in this article?
It’s the informal name for canonical mode input processing, the opposite of raw mode.
From the MIT/GNU Scheme 12.1 documentation:
This answer on Stackexchange may also help: https://unix.stackexchange.com/a/21760.
I have been wondering about this for a long time, perhaps someone knows… Why does GNU Screen use C-a and tmux use C-b as their default prefixes, given the long history, wide spread, and useful function of these bindings as beginning-of-line and backward-char?
Because they are not interactive, work in the mentioned earlier cooked mode, and rely on a TTY/PTY driver when not used in pipelines? Also curious: why git is mentioned? I can’t think of its command that would read stdin and expect it from a user (vs. another programme).
git add -pand its derivatives do.Great article as usual from Julia Evans.
The most frustrating inconsistency for me is handling of cmd-line flags. Is
-akxone flag? or multiple? This is one of my annoyances withterraform–it feels pretty lopsided how most follow the short vs long flag convention. I don’t know why some programs go out of their way to misalign with it.I deeply appreciate
clapfor making the standard convention available on a very fast runtime (i’m sure there are similar things in other langs–it’s just the one i’m most familiar with; it certainly helps considering the tendency for folks to want to rewrite everything in rust :^) )What a blast from the past. I wish I’d came across something like this back in day when I was learning this stuff. This falls into the category of things that ‘make it click’s but that aren’t documented anywhere. They’re rather a part of culture, common practices… Well… unwritten rules. This should be included in guides of how to use the command line, shellscript tutorials, etc
In regards to rule 6 and coloring, is there any shell smart enough to determine when the sole output of a bunch of pipes is the terminal, so it can enable color throughout? (I know you wouldn’t always want this, but I’d prefer it as the default.)
I’m not sure that would work because the intermediate stages might not handle colours. For example, if you pipe through short or awk, it may get confused by the escape sequences and shorty by the control code of the colour escape sequences rather than the text.
Somewhere, I have some FreeBSD patches I should dig out that added the ability to send multiple streams and do per-stream content negotiation to TTYs. This borrowed the same protocol that drag and drop commonly uses, where the sender advertises support for a set of types and the receiver picks the preferred one. I extended libxo (the library that produces pretty coloured output, JSON, XML, or HTML for most FreeBSD command-line tools) to offer these types and let the receiver pick one, so you could pipe to something that would collect HTML and open a web browser with it (or integrate that in the shell or terminal emulator, so it could render HTML inline).
Arcan had a more principled version of this, but my implementation was easy to retrofit. The libxo changes added up to about five lines of code.
Thanks. Arcan looks interesting.
I’m well aware that the intermediate stages might not know what to do with escape sequences. What I was envisioning is a bit more like the example in my other comment, where I run something
ls | grep | head, and the shell goes “Aha! I know the color options for all of those”, and inserts them on my behalf.Obviously not universally applicable or foolproof, but anything is more useful than what we have now, which is the loss of color as soon as you hit ‘|’ (unless you add all the flags manually, which I’ll bet a kidney most of us aren’t, especially as the pipe gets longer.)
This piqued my curiosity again, so I tried
ls --color=always | grep --color=always …— it’s unpredictable in a bad way.For instance, in my example test,
srcis a directory and ls has coloured it blue. If I grep fors, then grep colours thesin red, and the rest of the word is white. If I grep forr, then we have a blues, redr, and whitec. This took me a moment before I thought about the control sequences involved.I’m sure there’s still instances where it’s neat, but this kind of behaviour makes me nope out.
Bummer. I hadn’t really thought too much about dueling colors in this scenario.
Besides the example you give, there is another problem with the
isattyheuristic – sometimes (often? usually?) you want to pipe output to a PAGER likeless -Rthat can handle color codes. When that happens, you need to look up how to do the equivalent of--color=always. So, I just default to color always behavior but provide a--plaininstead, honor$NO_COLOR, and I also wrotenocso you can just stuff those 4 characters|nocinside some pipeline to get no-color like streams.Of course, as others in this subthread have pointed out, then you cannot “re-colorize” things because the thing understanding the semantics has lost control. The whole situation is a mess. Roughly, the “in-band nature” of the coloring of text cells rather than some kind of out-of-band second stream breaks that old chestnut “Write programs to handle text streams, because that is a universal interface.” To preserve color it seems like you would still need a way to correlate text cells in the two streams (e.g. coordinates of some kind) which probably then corrupts the internal simplicity of some text filters. Not sure if @david_chisnall’s libxo mentioned here or Arcan are the answers or really address the “whole system of programs problem”, TBH, or just one or two aspects. In other words, is it a bug or a feature that “grep” need not negotiate its IO formats with its two neighbors? Many a grep-author might say feature / performance necessity, but many an end-user might differ.
arcan-tui doesn’t inband anything on top of stdio and do address the ‘whole system’. All window management, interaction, clipboard, accessibility, internationalisation, alternate stream setup etc. goes over a sideband per window. The libarcan-tui consumer has access to a semantic palette and a legacy (red is now green problem) palette for when we do need to spawn a pty to compose a client that do check isatty).
The outer display server sends the preferred palette (and available glyph set) that maps to the semantic one on first connection. It can remap this at any time, and if that happens, there is an event fired (recolor), though this is mainly used by the terminal emulator written for it.
When you signal that you are finished drawing contents, the screen resolves colours according to the current palette into RGB values and writes that into a packing format, TPACK, that is also used for screenshots and screenrecording. The format itself contains a header on cursor information; if it is an I frame or a P frame and a set of rows with shaping information/offset and the cells to for that row.
You can see that in this old (2021) clip: https://videos.files.wordpress.com/JtFbZP4u/clip_11_migrate_mp4_hd.mp4 the screen on the left with the white window is a tui client. When I migrate the window to another machine (the laptop on the right), which has a different colour scheme preference, the background turns dark.
Very neat seeming! Thanks for the detailed reply. Can Arcan handle “layers” such as if you have a
mygrepthat wants to boldface or inverse or otherwise tweak/color/attribute transform “matching text” of already syntax-highlighted text (within a pipeline) feeding it to then something else (maybe only potentially the text window, maybe another pipeline stage)? Just some simple APImygrep.ccan call to do that? Is the communication bi-directional? Can this “layering pattern matcher” program query the display server for the existing rendition and then just do its own “color inversion” (like inverse text in ANSI, whatever “inverse” really means here..)? (EDIT: BTW, I feel the answer will be “but of course!”, but I thought I should ask anyway.)Everytime you discuss Arcan, it always sounds like a really complete solution and I feel I should give it an earnest try, but never do. Maybe this time will be different. :-)
I’m not sure if we have the same idea of ‘layers’ here - the screen as it stands is single-layer. There are basic provisions in place for creating another window and embedding that as an overlay (or explicit copy between two windows but that is less exciting), but it has not been used for text purposes.
If you look at: https://arcan-fe.com/wp-content/uploads/2022/10/cooperate-1.mp4
You can see around the 0:48 mark (and before then, side-band transfer of a large:ish file pasted into the shell without blocking it, take that ‘block paste’ mode) where the video clip, as well as, a pdf viewer gets embedded. The shell (as in a tui client here) has control over its embedding and scaling properties, gets feedback (so bidirectional) about rows/cells consumed but doesn’t reason or have access to the per-pixel view of the contents yet the outer windowing system is free to decompose it. The option to say, do that to another tui window follow the same mechanisms - albeit no blending options are exposed.
As far as per cell formatting attributes go, you have the standard underline/alt-underline/inverse/bold as a bitmask (together with border_t,l,d,r to avoid wasting cell space on line-drawing characters, hence why the spreadsheet in that other demo doesn’t waste half its height on drawing borders). Then it’s up to the rendering side how that should resolve.
It’d be pretty hard for the shell to do that — it’d either have to know what options to add to each invoked command (erring on the side of safety, lest your shell make your working command into an incorrect one), or maybe set an environment variable (which those programs would have to opt into). Programs detect whether or not to enable colour automatically themselves for the most part (by checking what kind of thing stdout is with
isatty(3)) — perhaps they could follow the pipes (!).This also doesn’t work very well anyway; imagine you do
grep blah | sed thingy, andgrepgets colours enabled for it automatically. Then yoursedexpression happens to trash half of those ANSI codes. Putting colours in works if you’re only piping it to head/tail/cat/things of that nature, but you don’t have to do much interesting before it’d grate.Sure, the universal case might be difficult/impossible, but the 80%-good-enough case might not be. And I’d have no problems falling back to a non-color version if I get gibberish.
Not even 80%, really. Maybe 50% of the time, I just want to take some colored output and do some light filtering/paging on it without losing color or looking up how to do it myself.
A shell figuring out I want color in
ls | grep | headis what I want.An allowlist of known-good programs and the way to chain them could still get us pretty far.
Yeah, absolutely. I wonder which existing shell would be easiest to try hooking something this onto.
I have an inner feeling of ‘ick’, and I guess what that’s telling me is that this (perfectly reasonable) thing we desire is really quite hard to do with the current design of the terminal stack (y’know, at least to do it well; it’s very clearly not designed to support it), and leads to wondering about aiming higher.
(Put another way, I suppose PowerShell could do this kind of thing pretty well ..)
There is an attempt with the FORCE_COLOR env. var., a counter-part of NO_COLOR.
Thanks for mentioning
FORCE_COLOR. I hadn’t heard of it. Notice the asymmetry - while it is possible to write a little pipeline utility to strip ANSI color codes {mentioned in my other comment }, putting the color in the stream in the first place requires context-specific semantics the generating but not receiving (or possibly transforming) program knows.If the shell arbitrarily lies about stdout being a tty or not, then knowing how a particular set of commands will work (in a script when pasted from a set of instructions, etc.) becomes indeterminate. If you want this, then you want to change your communication channel between the elements of the pipe to something other than pure text. This is what PowerShell and Nushell both do to varying degrees.
The 80% good-enough case you mention below is not good enough for something that has to work correctly 100% of the time, and the only right way to handle this in general terms for shells which pass text between them is to make it explicitly opt-in. There’s no good way to do this as opt-out.