1. 13
  1.  

  2. 6

    The historical reason for colour-pairs is that some hardware terminals worked like that. It’s easier to emulate colour-pairs on a terminal that supports independent colour, than to emulate independent colour on a terminal that only supports pairs, so that’s the way the API was designed.

    For most applications, colour pairs aren’t a big deal. Think of them like CSS rules in HTML: we use this pair of colours for a heading, that pair of colours for body text, and so forth. You don’t need a colour pair for every possible combination of foreground and background, you only need a colour pair for every kind of information your app will display.

    The big limitation of colour-pairs is if you want to display arbitrary colours - things like image-to-ascii-art tools, tmux, that kind of thing. But that’s quite a minority of tools, so using a lower-level library like terminfo or termbox or hard-coding escape sequences is pretty reasonable.

    1. 1

      For most applications, colour pairs aren’t a big deal. Think of them like CSS rules in HTML: we use this pair of colours for a heading, that pair of colours for body text, and so forth. You don’t need a colour pair for every possible combination of foreground and background, you only need a colour pair for every kind of information your app will display.

      I mention this in the mailing list post. The problem with this is that it couples your program architecture to ncurses, which the library shouldn’t require.

    2. 5

      Well. I propose making pipes multi-channel & segmented, and using that to write metadata/attributes without inline escaping. stdout.channel("color").write("red"); stdout.write("Hello"). Drag Unix kicking and screaming into the Year of the Justifiably Defensive Lobster!

      1. 8

        Having inline escape codes might not seem pretty, but it’s good for the same reason UTF-8 is good:

        • anything will accept it, even if it doesn’t understand colour.
        • programmers can put a lot less effort in, especially initially, if they don’t care about all of the features.

        Or: don’t add more dimensions to a problem that can already be serialised into 1D, if you start solving problems by adding dimensions then you will end up with a hypercube.

        Drag Unix kicking and screaming into the Year of the Justifiably Defensive Lobster!

        Slippery sidenote: I’m not a fan of using the word ‘modern’ to describe software in a good light, or saying something is bad because it’s “not modern”.

        If there’s is one lesson that we should have learned from the past few decades of software development it’s that you should not assume that current fashions, expectations or trends in programming are in any way better than prior ones. Most of them are just as bad (in different ways), and this is only properly realised years later.

        EDIT: If you’re writing something to be useful, then stick to tried and true development & design practices. Don’t equate “modern” to mean “tried and true”.

        1. 3

          I’m not a fan of using the word ‘modern’ to describe software in a good light

          Same. I roll my eyes whenever I read the phrase “modern programming language”.

          As for UTF-8: it can have just as graceful a fallback. Suppose we want to port ls -l. We’ll start by writing the same bytes as before, but they’ll be annotated by channel. For -rw-rw-r--, we could write that to mode. The next field is the number of hard links. But first comes a space, and there might be multiple spaces to pad out the field. They can be written into tab. Eventually we’ll reach a filename that we might want to color, so we’ll write the whole escape code to escape, and when we get to the \n, we’ll write it to “record-separator”.

          This gets us some slight benefit. You can use ≈jq instead of cut/sed, and feel less nervous about filenames with funny characters. There isn’t a whole lot of benefit until further incremental steps are taken:

          1. Annotate existing output.
          2. Write extra data to channels that are hidden by default. The writer must know if the read end is naive, or only interested in particular subset of channels.
          3. Separate data from presentation. A library can handle rendering to the naive format.
          4. The shell & terminal become type-aware. Their capabilities increase over time. Short filenames hide a full path, and your SSH client sticks the hostname in front. The output of ls doubles as a filemanager. wget & wget no longer makes a mess because each command requests a separate output pane. You can run find ~/Pictures/ | grep cat, pass the output to ImageMagick to make a collage, display it inline, and right click -> rerun on input change. Running program --help attaches a typed grammar for tab-completion. The shell is merely twice as terrifying as a spreadsheet.

          Each step can be reached from the previous. No worlds need to be rewritten. All that’s required is implementing a few syscalls and kickstarting some coreutils.

          1. 1

            nushell has thought about some of this a bit: https://book.nushell.sh/en/working_with_tables

        2. 3

          If you’re modernizing, why not just drop curses for drawing and just use a graphics library?

          1. 2

            How would you read this multi-channel pipe? Color channels must stay synchronized with data but it should be possible to read data without reading color channels. I’m afraid that this is a dilemma.

            1. 1

              “Channels” might be a bad name. When reading, the *buf is filled just like the usual read, but there is a second buffer that contains channel span information. Perhaps “labels” would be a better name. A high-level wrapper for this syscall in Rust might return a Vec<(ChannelName, Vec<u8>)>.

          2. 2

            I thought I was the only crazy person that found having to pair colours first to be annoying. I was literally building a lookup table (input: fg & bg, output: colourpair) to work around it; basically nullifying curse’s colourpair feature.

            Curses is somewhat standardised. I’ve used the “pdcurses” implem to get win32 support for one of my ncurses apps. Whilst new features are good, it’s also worth seeing some restraint and checking out whether or not this would cause problems with the other implementations.

            1. 1

              It shouldn’t cause issues; backwards compatibility should never hold back innovation. If you need compatibility with multiple curses implementations (there are several more than just those two), then you write targetting curses. Otherwise, it’s entirely reasonable to just target a single implementation of a single library. 99% of all libraries work that way; no one worries about compatibility with alternate gtk+ implementations. Besides, other curses implementations do tend to also pick up extensions added by ncurses; ncurses already has more features than the original curses. They currently advocate using attr_on() over attron(), even though only the latter is in the original curses.

            2. 1

              I’m squarely in that group that simply avoids using Ncurses. I find Ncurses to be a baroque API, from what I know, and as with many UNIX APIs it seems so often I learn of some new quality and can’t tell if it’s a joke or not, at first.

              My advice is to simply use ECMA-48; every terminal emulator you’re likely to come across supports a subset and it’s easy to use, being free from ’‘color pairs’’ and other nonsense. The only issue is sending different sequences to terminals that don’t support the subset being used, or finding a common subset, but there is a point where this ceases to be reasonable and one adopts a ’‘If you’re terminal doesn’t even support this, I don’t care if it works.’’ attitude.

              Writing a library in most any language that sends ECMA-48 codes is simple, so you can work in whichever language you’re actually using instead of binding with a decrepit C library. It’s also important to stress that people only really use Ncurses and whatnot still because the graphical APIs of UNIX are far worse to cope with.

              Now, to @Hales :

              If there’s is one lesson that we should have learned from the past few decades of software development it’s that you should not assume that current fashions, expectations or trends in programming are in any way better than prior ones. Most of them are just as bad (in different ways), and this is only properly realised years later.

              If one digs deep enough into the history of computing, one lears that what’s modern is distinctly worse than prior systems. There were easy graphical systems on computers decades ago, but UNIX is still stuck sending what it thinks are characters to what it thinks is a typewriter and using X11 lets you pick your poison between working with a directly (Wayland is coming any year now, right?) or using a gargantuan library that abstracts over everything. It’s a mess, I think, don’t you agree?

              Also, @Hales , you’re note on UTF-8 reminded me of something I found amusing from the Suckless mailing list, when I subscribed to it trying to get proper parsing of ISO 8613-6 extended color codes into st. Simply put, UTF-8 is trivially transparent in the same way color codes are transparent, in that there are many cases where invariants are violated and programs misbehave, commonly expressed as misrendering. There was an issue with the cols program, that shining example of the UNIX philosophy: it didn’t properly handle colors in columnating its output; to properly handle this, the tool would need to be made aware of color codes and how to understand them, as it would otherwise assume each would render as a single character would, ruining the output; the solution, according to one in the mailing list, was that colors are bloat and you don’t need them.

              Don’t you agree that’s amusing?

              1. 4

                I’m squarely in that group that simply avoids using Ncurses. I find Ncurses to be a baroque API, from what I know, and as with many UNIX APIs it seems so often I learn of some new quality and can’t tell if it’s a joke or not, at first.

                I agree with you. 100%. HOWEVER. Ncurses comes with pretty much every mainstream unix, making it a very attractive target for applications. I think that ncurses’ limitations are holding back innovation of TUI interfaces. If every unix comes with an adequate TUI widgeting library, this encourages quality TUI widget-based systems.

                If one digs deep enough into the history of computing, one learns that what’s modern is distinctly worse than prior systems. There were easy graphical systems on computers decades ago, but UNIX is still stuck sending what it thinks are characters to what it thinks is a typewriter and using X11 lets you pick your poison between working with a directly (Wayland is coming any year now, right?) or using a gargantuan library that abstracts over everything. It’s a mess, I think, don’t you agree?

                It’s important to understand things in context. At the time it came out, X was very useful and its complexity was every bit justified. Each bit of bloat was entirely justifiable when added, it’s just in hindsight, when looking at the whole, that it appears bloated. Yes, xorg is a mess. (Incidentally, wayland is also a mess and I hope that it doesn’t come at all. Much prefer arcan.) However, I think it’s a mistake to say that in the dawn of computing they understood simple, sensible interfaces and nowadays it’s all just pointless abstractions and bloat. Sure, Unix, BeOS, and C were all clean and well-made; while javascript, c++, and windows are all currently a mess. So what? There’s still significant amount of innovation and progress being made in the modern world of software and to ignore it would be a mistake.

                [cols/utf-8/colour/alignment-related stuff]

                Yes. The entire concept of a TTY is a mess. I am going to work on replacing that. Other people are working on replacing that. (The solution, obviously, is to make the code that generates the text and the code that displays the text work on the same level of abstraction.) That doesn’t change the fact that we’re stuck with ttys for the time being; is it wrong to try to improve our experience with them until we can really leave them?

                1. 3

                  However, I think it’s a mistake to say that in the dawn of computing they understood simple, sensible interfaces and nowadays it’s all just pointless abstractions and bloat

                  The stuff we’ve kept from the dawn of computing is basically all either ‘simple, sensible interfaces’, or ‘backwards compatible monstrosity’. Those are, after all, the two reasons to keep something - either because it’s useful as a building block, or because it’s part of something useful.

                  1. 3

                    The entire concept of a TTY is a mess. I am going to work on replacing that. Other people are working on replacing that. (The solution, obviously, is to make the code that generates the text and the code that displays the text work on the same level of abstraction.) That doesn’t change the fact that we’re stuck with ttys for the time being; is it wrong to try to improve our experience with them until we can really leave them?

                    Have you looked at the TUI client API for Arcan? and the way we deal with colour specifically? If not, in short:

                    1. Static set of semantic colour labels with some compile-time default palette.
                    2. Clients resolve on-draw the labels into R8G8B8(FG, BG).
                    3. Channel values are set as bg/fg attributes to the appropriate cells in the grid.
                    4. Server/WM side has commands for runtime replacing the colour values stored at some label index.
                    1. 1

                      Do you allow the client to set arbitrary 24bit colours to the grid?

                      Does the TUI API work with text-based but not-fixed-width interfaces (e.g. emacs, mlterm)?

                      Thank you for posting, I hadn’t heard about arcan until today but have just read a chunk of your blog with interest :)

                      1. 2

                        colors: both arbitrary fg/bg (r8g8b8) and semantic labels to resolve (such as “give me fg/bg pair for alert text”) shaped text: yes (being reworked at the moment to account for server- side rendering), but ligatures, shaped is there as a ‘per-window’ attribute for now, testing out per line.

                        1. 1

                          Thanks for the reply.

                          I think I would want to be able to use different fonts (or variations on a font) for different syntax highlighting groups in my editor. This looks quite nice in emacs and in code listings in latex. Perhaps you consider this to be an advanced use where the application should just handle their own frame buffer, though.

                          While I have your ear, what’s the latency like in the durden terminal and is minimising latency through the arcan system a priority?

                          1. 2

                            in principle, multiple fonts (even per line) is possible, and that’s how emojii works now, one primary font for the normal glyphs and when there is a miss on lookup, a secondary is used. There is an artificial limit, that’ll loosen over time. Right now, the TUI client is actually handling its own framebuffer, and we are trying to move away from that, which can be seen in the last round of commits. The difficulty comes from shaped rendering of ligatures, where both sides need to agree on the fonts and transformations used; doing it wrong creates round-trips (no-no over the network), as the mouse-selection coordinate translation needs to know where the cursor position actually became.

                            Most of this work is towards latency reduction, removal of terminal protocols fixes synchronization, moving rendering server-side allows direct-to-scanout buffer racing-the-beam rasterization, or at least, entirely on-gpu for non-fullscreen cases.

                            1. 1

                              The difficulty comes from shaped rendering of ligatures, where both sides need to agree on the fonts and transformations used;

                              When you say “both sides” do you mean the client on e.g a remote server and a “TUI text packing buffer” renderer on e.g. a laptop?

                              Sounds like you could just send the fonts (or just the sizes of glyphs and ligature rules) to the server for each font you intend to use and be done with no round trips. Then you just run the same version of harfbuzz or whatever on each side and you should get the same result. And obviously the server can cache the font information so you’re not sending it often (though I imagine just the sizes and ligatures could be encoded real small for most fonts).

                              Do you have any plan about what to do about the RTT between me pressing a key on my laptop, that key going through arcan, the network, remote arcan and eventually into e.g. terminal vim and then me getting the response back? I feel like mosh’s algorithm where they make local predictions about what keypresses will be shown is a good idea.

                              Most of this work is towards latency reduction, removal of terminal protocols fixes synchronization, moving rendering server-side allows direct-to-scanout buffer racing-the-beam rasterization, or at least, entirely on-gpu for non-fullscreen cases.

                              Sounds exciting! I don’t know what you mean by “moving rendering server-side”, though. Is the server here the arcan server on my laptop? And moving the rendering into arcan means you can do the rendering more efficiently?

                              Is arcan expected to offer an advantage in performance in the windowed case compared to e.g. alacritty on X11? Or is the benefit more just that anything that uses TUI will be GPU accelerated transparently whereas that’s more of a pain in X11?

                              1. 1

                                Right now (=master) I am tracking the fonts being sent to the client on the server side, so both sides can calculate kerning options and width, figure out sequence to font-glyph id etc. The downsides are two: 1. the increased wind-up bandwidth requirement when you use the arcan-net proxy for network transparency, 2. the client dependency on freetype/harfbuzz.

                                1. can be mitigated through caching and stream cancellation if there is a client side match already. Only solutions for 2. seems to be a roundtrip for the shaping offsets.

                                My first plan for the RTT is type-ahead (local echo in ye olde terminal speak), implemented on the WM level (anchored to the last known cursor position, etc.) so that it can be enabled for other uses as well, such as input-binning/padding for known-networked windows where side channel analysis (1 key, 1 packet kind of a deal) is a risk.

                                Sounds exciting! I don’t know what you mean by “moving rendering server-side”, though. Is the server here the arcan server on my laptop? And moving the rendering into arcan means you can do the rendering more efficiently?

                                Is arcan expected to offer an advantage in performance in the windowed case compared to e.g. alacritty on X11? Or is the benefit more just that anything that uses TUI will be GPU accelerated transparently whereas that’s more of a pain in X11?

                                Both performance and memory gains. Since the actual drawing is being deferred to the composition stage, windows that are partially occluded or clipped against the screen would only have its visible area actually being processed - while alacritty has to both render into an offscreen buffer (that is double buffered) that then may get composed. So whereas alacritty would have to pay for (glyph atlas texture, vertex buffer, front-buffer, back-buffer) on a per pixel basis in every stage, the cost here will only be the shared atlas for all clients (gpu mem + cache benefits), rest would be a ~12b / cell + vertex buffer.

                  2. 2

                    Wayland is coming any year now, right?

                    It has been here for a while. GNOME and KDE support it natively, a few popular distros ship with Wayland enabled by default. Firefox is a native wayland app. What makes you think it’s “any year now” ?

                    1. 1

                      :D

                      Colours are not the only invisible control codes that I’d expect cols to have to handle. Alas I can’t see a “simple” solution to this. You pretty much have three options:

                      1. Treat everything as 1 char wide (simplest option)
                      2. Use a full unicode library to determine your character widths (most “correct”, but much bigger)
                      3. Whitelist known ANSI characters and assume everything else is zero-width (allows colour support, but upsets international users with you being ‘unfair’)

                      Of the crappy options I can see: #1 does seem the most like something suckless devs would like.

                      Suckless makes some great stuff, but some of their projects are a bit too minimal for me. Take for example their terminal emulator st:

                      ## How do I scroll back up?
                      
                      Using a terminal multiplexer.
                      
                      * `st -e tmux` using C-b [
                      * `st -e screen` using C-a ESC
                      

                      I love my scrollwheel, whether it’s a real one or two-fingers on my cheap little laptop’s touchpad. That and being able to use Shift+PageUp/Down. I guess everyone draws the line somewhere differently, and I have more things about my current term (urxvt) that I could moan about.

                      or using a gargantuan library that abstracts over everything. It’s a mess, I think, don’t you agree?

                      I don’t have any raw X11 experience. I’ve primarily used SDL, which yes indeed abstracts that away for me.

                      I’m not completely convinced that wayland is going to be the answer: from everything I read it seems to be solving some problems but creating entirely new ones.

                      From a user perspective however: Xorg is wonderful. It just works. You have lots of choice for DE, WM, compositor (with and without effects), etc. The open source graphics drivers all seem to have a TearFree option, which seems to work really well. I’d love to see latency tested & trimmed, but apart from that the idea of having to change a major piece of software I use scares me. I don’t want to give up my nice stack for something that people tell me is more better (or more “modern”).

                      1. 2

                        You forgot one:

                        1. Interpret control codes as 0-width, and assume each UTF-8 codepoint is 1 char wide (composing characters and some alphabets will break this, but if you need this, then option 2 is your best bet).

                        Interpreting ECMA-48 for this isn’t that bad, I have Lua code that does such [1]. And a half-assed approach would be to assume all of C0 (0-31, 127) [3] as 0-width, all of C1 (127-169) as 0-width, with special handling for CSI (155) and ESC (27). For ESC, just suck up the next character (except for ‘[’) and for CSI (155 or ESC followed by a ‘[’) just keep sucking up characters until a character from 64 (@) to 126 (~) is read, and that will catch most of ECMA-48 (at least, the parts that are used most often).

                        [1] It uses LPEG. I also have a version that’s specific to UTF-8 [2] but this one is a bit easier to understand in my opinion.

                        [2] https://github.com/spc476/LPeg-Parsers/blob/master/utf8/control.lua

                        [3] 127 isn’t technically in C0 or C1, but I just lump it in with C0.

                      2. 1

                        If one digs deep enough into the history of computing, one learns that what’s modern is distinctly worse than prior systems.

                        Flying on a modern airliner is also distinctly worse than passenger jet travel in the 1950’s. But I can still afford to go anywhere in the world in a day.

                        1. 1

                          Worse in experience? I am almost certainly safer flying on a modern passenger jet now than in the 1950s.

                          1. 1

                            https://i.huffpost.com/gen/1840056/thumbs/o-56021695-570.jpg

                            Enough said. :-P

                            (I am aware this is not a conclusive statement.)

                            1. 1

                              Heh, you can still get that (actually, much better) on international business class. Hmm, I’m curious if the cost is comparable between 1950’s tickets like those in your picture (inflation adjusted) and international business class today…