1. 28

  2. 17

    This doesn’t entirely solve the problem for two reasons.

    You set it to xterm:yterm:zterm. How does the terminal know which option was selected? This only works if the list is a strict superset, with no conflicts. The potential for user error and mismatches is huge.

    It doesn’t really solve the problem of stale servers missing entries. Things work better perhaps, but still not great. What happens when I write uterm which has more features than xterm but not as many as yterm?

    Feature detection is the right way to do things. Then I can invent my own terminal, and all the remote servers will use all the features available. (Feature query would probably be a better name; I don’t mean literally try everything and see what sticks.)

    Bonus thought: any solution should also consider the situation of nested terminals. I’m running an xterm, I login to a server and run tmux. What is my TERM? A simple priority list doesn’t really answer questions like how many colors can be displayed.

    1. 10

      Agreed. I like the article because it raises the issue of pain associated with TERM, but feature querying is the better way to solve this morass.

      1. 3

        Feature detection is the right way to do things.

        I believe that the author did discuss both feature querying, as well as simply having a list of enabled features, as possibilities.

        1. 6

          I agree that a giant string, putting the termcap itself in the environment, is a mess.

          Dismissing online querying may have been dismissed too soon. It doesn’t have to slow down every terminal app, only those that use the requested extension. If TERM says xterm, and you don’t need 24-bit color, then you just go. It’s only kaleidoscope ls that needs to perform the query.

          1. 3

            One problem with feature-querying is that things are often more complex than “yes-or-no”. For example, gnome-terminal supports arbitrary (24-bit sRGB) colour. xterm supports the same escape sequence, but maps the requested colour to the nearest one in its 256-colour palette. Do they both “support” the feature? tmux, as a terminal emulator, supports arbitrary 24-bit colour, but if the terminal it’s running inside doesn’t support 24-bit colour, it obviously can only display 256 colours or less… but if you later reconnect to tmux from gnome-terminal, you’ll see them all. Does tmux “support” the feature? What if tmux is running inside xterm, so tmux can send the 24-bit colour escape sequences to the outer terminal, but the outer terminal then (unknown to tmux) does the wrong thing with them?

            A more practical problem with feature-querying is that there’s not really a good way to squeeze it into the existing de facto wire-protocol. There’s no way to guarantee that terminal responses will get back to the application that asked for them (the app could exit back to the shell, or exec something else), and so over time, just about every “answerback” feature in terminals has been neutered for security reasons: an app can’t read the title-bar, or the icon text, or the clipboard, or just about anything else.

            $TERMINALS is not a perfect solution to the problem, but at this late stage in the game, it’s probably the most practical one left.

            1. 2

              One problem with feature-querying is that things are often more complex than “yes-or-no”. For example, gnome-terminal supports arbitrary (24-bit sRGB) colour. xterm supports the same escape sequence, but maps the requested colour to the nearest one in its 256-colour palette. Do they both “support” the feature?

              I think that’s a non-issue. If you can spit out an escape sequence and the terminal accepts it instead of doing something weird and horrible, there is no problem using that sequence.

              So what if your terminal supported 24-bit sRGB but you’re actually forwarding it to a display with over a link using 16-bit YUV with 4:2:0 chroma subsampling and another layer of lossy quantization? You don’t get all the colours but it works.

              The real grief is (1) whether a feature is supported at all, and (2) whether different terminals use the same encoding for that feature. Problem (1) is unavoidable but it should be possible come up with a standard encoding for purely stylistic codes that may be ignored if not implemented. Apart from that, querying might be an acceptable choice. The gratuitous differences (problem 2) really need to go as far as I’m concerned. If a feature is supported, then all terminals claiming to support that feature should damn well use the same exact encoding. Otherwise it’s not the same feature. Then people need to work together a little so that we don’t get a dozen different “features” that all do the same thing, differently.

              1. 1

                If you can spit out an escape sequence and the terminal accepts it instead of doing something weird and horrible, there is no problem using that sequence.

                For some applications, asking for a 24-bit colour and getting a 256-colour approximation is perfectly reasonable. For other applications, it can be a deal-breaker (for example, if your foreground and background colours are distinguishably different in 24-bit colour, but happen to map to the same colour in the 256-colour palette).

                There’s also the problem that implementations have bugs. Imagine a terminal-emulator developer who wants to add 24-bit colour support. They design, implement and test the feature, it works beautifully, so they update the “feature query” code to report “yes, I support 24-bit mode”. The new version ships, it winds up in CentOS 9 so it’ll be around for a decade, and then somebody discovers that it segfaults if you try to use 24-bit colour mode in a locale that uses comma as the decimal separator. Now, that may seem like an unlikely sequence of events, but the web-browser community briefly experimented with having browsers self-report what features they supported, and this “client reports it supports a feature, but it turns out to be broken for some use-cases” sort of thing wound up happening all the time.

                The browser community eventually settled on “graceful degradation” as the philosophy of choice. That has occasionally worked for terminal emulation (consider the X11 mouse reporting mode versus the SGR mouse reporting mode), but there’s not a lot of potential control sequences guaranteed to be ignored by legacy terminals that we can use to signal new features.

                1. 2

                  That kind of “worst is best” mentality might have been inevitable in the world of web browsers where there is a sick amount of money involved and coming from people stubbornly (or forcedly) stuck with IE6 or whatever. And that same approach of catering for the lowest common denominator and adding complexity to work around bugs in legacy software that will never be patched gives us OpenSSL… honestly, I think we can do much better. Especially so in the world of terminal emulators which are multiple orders of magnitude simpler.

                  Yes, I advocate (see my other comments here) that most software flat out drop support for legacy terminals. People who want the complexity of them are free to bite the bullet and run a compatibility layer. So if you really end up in the unlikely situation that you’re forced to use an old broken terminal emulator that you may not patch, then you should use that compatibility layer first, and then escalate the issue with whoever is disallowing software updates in 2020. The problem internal to that organization should not leak out and affect the rest of the world. If that organization is just you, your problem is self-inflicted and I have no sympathy.

                  In a similar sense, worrying about the number of colours that can be displayed is imho a silly corner case problem that the whole system doesn’t need to cater for. First it makes me question the sanity of the application – why would it forcibly choose such close colors that they are distinguishable in true colour but blend together with a smaller colormap? Even if the terminal supported true colour, such poorly chosen colours may yet be indistinguishable to the viewer. What now, do we also feed data about my PC monitor and eyeballs in to the system? Alternatively, the program that purports to support true colour but then does a poor remapping of them is buggy. Fix it, use something else, or disable colours in it, or strip the colour capability flag. The solution doesn’t need to be so complicated.

                  I don’t love terminals, but one of their redeeming qualities is that they are, in the end, relatively simple. I think that we can keep it so, and even make them simpler than they are now, but the effort would be hindered if people worry too much about absurd corner case scenarios and broken unpatchable software that they absolutely must use in conjunction with newer software that rightly assumes the program on the other end isn’t broken.

              2. 1

                Unfortunately a list of terminal types doesn’t solve the nesting problem. Are you going to remember to reset the environment every time you connect to tmux from a different machine? On the other hand, querying allows a program to determine the current state of the actual display terminal. This seems easier than demanding tmux support a superset of all features and downmix on its own.

                1. 1

                  I don’t think the nesting problem is soluble, even if feature-querying were practical. An application would have to re-query every time it drew to the screen, and… I’m pretty sure a lot of applications wouldn’t bother. (also, how should tmux respond to feature requests when running detached, or when attached to multiple terminals with different capabilities?)

                  It’s no doubt a lot of work for the tmux people to support a superset of all features and downmix them at runtime… but that’s how tmux works today, so it wouldn’t be extra work.

        2. 10

          I like the article, but the facebook login popup obscures some of the beginning text.

          1. 6

            It is a fossilized world, and one that I would like to see go.

            Curses and its database of terminal incompatibility made some sense back when terminals were hardware, there were no standards for them, and rewiring & reprogramming them would’ve been harder than just slapping a layer on top of it, host-side.

            Now it’s just legacy and bloat that at best does nothing for you, and at worst causes trouble because your database doesn’t have the right entries or your curses implementation is cranky. Bloat – on OpenBSD, about 36k lines of C, plus the database – in a library that just about every application using terminal escapes will load.

            1. 2

              how would you propose that an alternative should work? (and yes, it’s a leading question but I’m fishing for ideas…)

              1. 8

                Start with a standard. Yes that means drop support for ancient crap. If someone really wants to play with old glass on new systems, they can take the hit and run that bloated compatibility layer.

                Although I’m not too fond of anything that is in use now, the deprecated ANSI standard or the de-facto xterm standard do work. I wrote a code editor that uses a tiny subset of the ANSI escapes and it ought to work on practically any terminal one might run into these days. If there are a few weird ones that just don’t support these escapes, I say fix them or nuke them. There are plenty of terminal emulators to choose from, and it’s not terribly difficult to make a new one.

                Long term, I’d like a modernized, forward-looking standard. For instance, I think it is really silly that there are so many keys and inputs that we cannot reliably use because people couldn’t agree on a way to encode arbitrary inputs. And then there’s hacks like waiting some arbitrary time to determine whether you pressed the ESC key or sent some escape sequence starting with that same character. This is a thing that can be solved.

                I’m not terribly concerned with extensions – after all terminals aren’t changing that much these days. And frankly most of the change seems to be in purely stylistic stuff like colours. So it is possible to write the standard such that these purely stylistic codes must be parsed but may be ignored. Thus, colour, italics, bold and such become entirely optional, and there may be space left to encode nonstandard stylistic instructions. Then applications may be written under the assumption that whatever style it wants to use, it can send, and the receiving terminal supports it – if they don’t, no harm done, they just won’t see the colour. No need to query about these.

                Then if there’s still a pressing need for it, I wouldn’t object to having some way to query the terminal about feature support. Actually, I don’t particularly care how the query is done – these flags could be passed in the environment, a config file, whatever. Or they may be given by the terminal in response to a code that asks for them. Just like you can query for the cursor location already on existing terminals. Implementation details. See cpuid, opengl extensions, etc.

                The important thing is that pointless differences just need to go away. If you support feature X, there should be exactly one standard encoding of the control codes that make up X. No database of silly to figure out how 300 different terminals have 700 encodings for the same basic feature X that has been around for decades. If there’s some non-standard extension Y, other people willing to implement that same feature must follow the same rules. Eventually non-standard extensions can be incorporated into the standard. Again, see OpenGL.

                1. 2

                  In terms of supporting arbitrary key inputs, have you seen the proposal from libtermkey’s author?

                  1. 2

                    I have seen it, yes. And it is a fair proposal that does a fair job of maintaining compatibility with historic behavior while requiring relatively small changes to support the new scheme in current terminal emulators.

                    However, I might be a little too idealistic to just embrace that idea as great. A pet peeve of mine is programs that confuse letters and keys, and then hold some keys special because they’re not letters. Applications really have two modes of taking input – one is text, which might have been typed out, or pasted, or produced by an IME. The other mode is just taking abstract inputs and binding them to actions. Most nontrivial applications kinda need both.

                    When the artificial confusion of letter inputs, modified letter inputs and non-letter inputs is maintained, you’ll find that your application is unable to bind an action directly to control or shift, or to use some letter key as a modifier. Or, indeed, any other key that wasn’t explicitly given the special role of control and alt..

                    And what about non-keyboard inputs? Terminals may support mice too. Again here I just see unnecessary artificial boundaries in that these things are just flat out unable to process anything that is not a keyboard, or perhaps one of the two big mouse buttons. In fact it is possible to encode any input – from a fancy keyboard, or the seventeenth thumb button of my mouse, or my belly button – all uniquely. It is not hard, if people only quit looking at the concrete and thought in abstract for a minute.

                    So it should be possible to encode any combination of inputs (thus allowing any of these inputs to act as a modifier). That same encoding may include the text (or single letter) that is associated with that combination. Then it is up to the application to treat it as text input or as something abstract.

                    Of course this would be a much bigger and more intrusive change – not so great in terms of backwards compatibility.

            2. [Comment removed by author]

              1. 6

                Except the problem isn’t the fact that it’s an environment variable, the problem is what the environment variable contains, which is what the author proposes changing. The reason for the new name name TERMINALS instead of TERM is simply to allow for backwards compatibility if necessary.

                1. [Comment removed by author]

                  1. 5

                    Like setting EDITOR and VISUAL.

                    1. 2

                      TERMINALS there is several problems de facto.

                      1. 3

                        TERMINALS is the new systemd, yo.

                  2. 4

                    Just put your termcap in an environment variable and the problem’s solved amirite?

                    1. [Comment removed by author]

                      1. [Comment removed by author]

                        1. 1

                          At least we can use most programs that interact with at terminal in the 9term-style, with TERM=dumb, or simply ignoring TERM.

                          I prefer the curses library as a program instead, like ACME (or Emacs and nvim with their built-in shell): One program that handle displaying multiple buffers in multiple windows.

                          All the other programs could then build their interface this way: stdin stdout and plain text flowing through (REPL). The PAGER and EDITOR/VISUAL would redirect the text to a new buffer.

                      2. 1

                        As someone who has written a terminal emulator (https://github.com/deadpixi/mtm), let me relay a bit of experience here.

                        For character cell applications, beyond basic things like color support, no user generally cares what character sequences are sent from the application to the terminal. Speeds are generally fast enough that it doesn’t matter if you need to send five bytes to move the cursor up versus three.

                        There are features that vary from terminal to terminal, most notably mouse support and alternate screen buffers, but if you don’t need mouse support you fall back to likely not caring what’s going on “under the hood”.

                        Writing mtm, I went to a lot of trouble to find the most basic possible terminal that had an existing entry in the terminfo database (I wanted mtm to work “out of the box” without having to install a new terminfo entry). I started out with mach, which worked well, but assumes that all terminals are white-on-black and (via the “op” capability) reset the terminal back to those colors when a curses application exited.

                        After that I decided to go the “emulate all of the common terminal escape codes that I could find” and ended up going with the generic “ansi” terminal type. This had a couple of problems, the most notable being that the default alternate character set assumed that you weren’t running UTF-8 or whatever. I got around that by turning off UTF-8 processing in ACS mode, but it felt hacky.

                        Interestingly enough, the “ansi” terminal type also doesn’t specify a “csr” (change scrolling region) capability. This is assumed to be present by a lot of applications (recent versions of apt, for example). So for a while there mtm had “ansi” plus “csr”.

                        Then I realized that it was massive overkill. So now mtm emulates the “eterm-color” terminal type, which is the terminal emulated by the Emacs AnsiTerm package. It’s a much simpler terminal to emulate (though it includes color and a definable scrolling region) and it works with everything I’ve experimented with and it’s been my primary terminal for months now.

                        (The smallest useful terminfo entry was the sun-color entry, but it was a little bit too simple.)