1. 81
  1. 11

    I never thought of it that way until now but I think I just found my favourite example for the longest-running, leakiest abstraction ever – non-printable characters.

    Non-ambiguity – there is a discernible difference between pressing the ‘ESCAPE’ key and the ESC-ASCII character that was used to mark the beginning of an escape sequence.

    Modifiers exist, CTRL+C is a symbolic C key with CTRL modifier, that does not equal ^C or it being magically translated to broadcasting SIGINT.

    It feels a little unsettling that what we now use, effectively, as a data encoding format (ASCII) continues to hold – and it’s up to us to handle when writing software, too – things that correspond not to textual data, but TTY key press and serial link events, and that they follow you everywhere, even in places where the are no teletypes, no serial links, and no keys. Like most people who poked at their innards I used to think that terminal emulators are nasty, but for the longest time I thought that’s just because we took the “emulator” part a bit too seriously and didn’t bother to question it because we’re intellectually lazy. Now I realize it’s legitimately hard to question it, because a lot of assumptions are actually baked into the damn data model – terminal emulators aren’t leaky abstractions, they’re quite literally the abstraction being leaked :-D.

    I once had the good fortune to meet one of the elderly chaps who worked on the IEC’s standard terminology, the International Electrotechnical Vocabulary. One of the things he was quite adamant about was that admitting there is such a thing as a “control character” effectively amounts to the younger generation taking klunky hacks that his generation resorted to due to a lack of vision and promoting them to engineering – they were meant as “control signals” and, for a variety of reasons, it made sense to encode them the same way data is encoded, but that should have never been taken to mean that they are data. Encoding transient events in the stream of data is sometimes called for, but counting them along with data – and, worse, committing them to storage – is a bad idea, because they’re really not even the same kind of thing. I honestly thought he was being pedantic (as you can imagine, people who work on standards about standard definitions are really bloody pedantic) but that was before I poked at xterm’s innards…

    Edit: I think the IEC had not just long settled on the idea that “control characters” are a thing – I think his colleagues were a little refractory to this idea. In all fairness, at the time, I nodded respectfully but otherwise thoroughly discounted his rant, as some engineers tend to be bitter and a little particular about the most trivial of things in their old age. Of course, I thought, “control characters” are a thing – they’re atomic items in an alphabet encoded according to specific rules or, as we call them when we’re not trying to be overly pedantic, characters (please bear in mind that I was really young, and therefore really stupid in some ways). Maybe the dichotomy is needlessly radical but I think it highlights a nice idea – that encoding signals along with the data is a good technical solution for some problems, but it should never be allowed to be formally reversed, not even when replaying previously signaled conditions is also a good technical solution.

    1. 2

      FWIW this is a primary reason Oil has QSN:

      https://www.oilshell.org/release/latest/doc/qsn.html

      So control characters in filenames can be properly escaped like '\x01\x02'.

      Oil is rigorous about not mixing data and code, which is a big design flaw of shell. Terminal control codes are a kind of code!

      In the doc I point to the concept of “in band signaling” in electrical engineering, which is basically just a word for what you’re describing:

      https://en.wikipedia.org/wiki/In-band_signaling

      1. 2

        In the doc I point to the concept of “in band signaling” in electrical engineering, which is basically just a word for what you’re describing: https://en.wikipedia.org/wiki/In-band_signaling

        Kiiind of – I mean, yes, that’s actually the term that I would use (I should probably clarify three things at this point – that this was not a majority-held opinion and I’m not quite holding it either, that both me and that guy were EEs (well, okay, I was EE-ish) so the common ground between us was closer to the physical layer rather than libc, and that his first exposure to information theory and telecom systems preceded the first edition of the ANSI X3.4 standard by a few years so he was there for the making of a lot of terminology that I’d inherited, and was therefore far more open to debating it).

        The concept of in-band signaling applies a little uneasily to some digital communication systems. It predates layered and packet-switching networks, and it’s specifically about sharing the channel, not necessarily the encoding scheme, among data and control streams. For example, one can imagine a (horribly unpractical) wired serial link which encodes printable characters in ASCII, but uses variable DC offsets for control signals (extremely bad idea, don’t do it). This rigorously meets an inflexible definition of in-band signaling, as signals and data do share the same channel, although people who, like me, learned about these things at a time when OSI layers were already basically indistinguishable from the story of creation and are inclined to think in terms of layers and logical channels would say it’s actually a form of channel-associated, out-of-band signaling.

        IMHO, in hindsight, the original bad idea for Unix TTYs specifically was not so much that they mixed control codes with character codes, but that they mixed transient runtime- and hardware-specific signals with textual control codes. It makes some sense to encode, say, EOF or LF in-band with a text stream, and to use a control code for it (even by redefining them – ^D and ^J aren’t used the way ANSI thought they’d be used). You can certainly signal them out-of-band if you go out of you way but it makes a lot of things uselessly coplicated. Encoding runtime signals in-band, on the other hand, like SIGINT, was proooobably not as good an idea. That’s why, much like my old interlocutor, I went out of my way to avoid calling it in-band signaling, and just called it “encoding transient signals along with the data”, the idea being that transient signals should probably be out-of-band.

      2. 1

        things that correspond not to textual data, but TTY key press and serial link events, and that they follow you everywhere, even in places where the are no teletypes, no serial links, and no keys.

        Then enter CLK. Think of the buffer-window boundary timing component and find the other dimensions that is leaking. What happens if your transfer stalls at the ‘last observed character’ being ESC? Now vim, zsh and others have something akin to ‘set timeoutlen=1000 ttimeoutlen=0’ heuristics to balance between the latency feel of the escape key in your typing, vs the risk of interpreting the next printable characters as keypresses. Then someone at the local-remote boundary need to chunk this accordingly, and not play the timing game too close to the edge because the jitter from the local display system may well push it over.

        I’m not set on in which context these things should even be brought up and taught to the kids as cautionary tales, but “lifetimes of machine instruction encoding schemes and dispatch routines” seem close to where these concepts meet for a very saucy orgy, even if it doesn’t roll of the tongue that easily.

      3. 7

        Yes, finally something which matches what I want the shell to be!

        I have on question/user request. I really love what I am seeing in the first video, and I would love to start using that (both the GUI niceties, and the underlying absence of terminal emulator craziness) immediately. However, my understanding is that arcan as a project is quite monumental, and I am not quite ready to leave the comfort of my KDE/Plasma environment. What would it take to make cat9 installable via a Linux package manager, and making the ./cat9 binary just start an X11 window? I don’t care much if that window is backed by gigabytes of arcan stack and horrible bridging to X11, but I do care a lot about being able to install a single command and just run it.

        1. 8

          The next article on the demonstrated shell(s) is mostly written, and contains just the stuff you want to know/see (ok some waxing about the cool bits not yet shown, but that is just dessert).

          The threshold for publication is basically me daily driving the shell for a week without issue or reaching for another; enough distance to this one to not seem spammy or self-promoting and enough re-iterations that it alienates about 99.9% of the audience that could benefit by appearing too complicated, i.e. hiding in plain sight.

          Had I not had the release cadence of “once a year, and hopefully not even that” the void-linux package is pretty much turnkey. There’s some talk on the discord of a nightly-build repository, if that materialises (out of my hands, I’m not paying me enough to CM) that (or some insert-your-distribution translation of the template) would be your answer.

        2. 7

          Wow this is very cool … I might have missed it, but what is the IPC protocol between the textual shell and the graphical shell? It is defined to be polyglot and not a Lua API?

          FWIW I have purposely divorced Oil from the terminal via “headless mode”. (Slogan: The shell UI should have a terminal, it shouldn’t be a terminal.)

          That was partly based on a conversation with you a couple years ago, and then last year a contributor came around and we actually ironed out a protocol, and he wrote a Go client for it.

          http://www.oilshell.org/blog/2021/06/hotos-shell-panel.html#oils-headless-mode-should-be-useful-for-ui-research

          (I really should have published the screenshots … there are some on https://oilshell.zulipchat.com/)

          We called the protocol “FANOS” – file descriptors and netstrings over (Unix domain) sockets. That work has been dormant for awhile (since the contributor moved on) but I hope people can revive it. If it can be somehow hooked up to Arcan’s IPC that would be cool.


          A main constraint of the protocol is that I don’t want to break ls --color and isatty(). So a file descriptor is passed to the shell for EVERY command, and then shell spawns processes like ls that inherit that descriptor.

          And then isatty() will work return true if the GUI passed it a terminal, as intended. It will return false if the GUI passed it a pipe.

          Does Arcan’s IPC address this issue? I’m wondering if all programs like ls have to be modified, or if they can be used “as is”.

          I found that replacing the shell is a huge amount of work so I don’t want to mess with ls, let alone vim :)

          1. 3

            Wow this is very cool … I might have missed it, but what is the IPC protocol between the textual shell and the graphical shell? It is defined to be polyglot and not a Lua API?

            Long Story condensed: The TUI API (C) is layered on Arcan-SHMIF (C), SHMIF is much “Arcade hardware designs of the 80ies revisited” - shared memory (ringbuffers for events (one in, one out), larger resizable regions for audio/video/complex-meta like HDR and VR) semaphores for synch, socket for descriptor passing and integrating with i/o multiplexed designed client event loops.

            Event data model is static (128b/event because cache lines), match C-ABI to avoid pack/unpack and looks very syscall:y) and lockstepped to Arcan. There is, conservatively speaking, about 10 years of accumulated details in how the above is actually structured to do all I need, but suffice it for now that the extra nuances of TUI specifically is a packing format for the pseudo-rendered screen (BLINK attribute goes away, shaping hints join in) as I or P frames.

            This only works locally (of course), for the distributed case, the protocol is the ongoing A12 with all the horrors replacing SSH entails – but there is a server that translates between the two.

            There are half-finished bare minimum Python bindings that was used for a PoC in a prior engagement – but a lot of care was put into the Lua bindings to make sure that they also covered for the ‘shell needs’ asynch-I/O deficiencies Lua normally has. Speaking of vim

            A main constraint of the protocol is that I don’t want to break ls –color and isatty(). So a file descriptor is passed to the shell for EVERY command, and then shell spawns processes like ls that inherit that descriptor.

            (Insert something about omelettes and breaking eggs…) Two strategies. The one which manifests in a new window:

            https://github.com/letoram/cat9/blob/sub/cat9/base/jobctl.lua#L309

            1. shell asks display server for a window that will be delegated to another.
            2. display server checks with wm, wm says shell has been a good boy, make it so.
            3. display server allocates (shmpage+semaphores+socket) and passes it over existing connection.
            4. shell takes this, makes a note of some information (e.g. uuid) to be able to reposition/embed the window.
            5. spawn new process (in this case a terminal emulator), pass the (shmpage+semaphores+socket).
            6. new process maps these resources and draws into it.

            The other is a bastardised version of popen.

            What is missing there for Cat9 specifically still, is assigning per-job decoders so there can be an ECMA-48 light decoder memory safe - but also used for much more interesting things.

            1. 2

              Hmm I can see why that kind of protocol would be useful for GUI apps to communicate with the display server, but I feel like the shell <-> UI deserves a special case that is small and well specified.

              And I’m not sure if you can pass a file descriptor between two processes over shared memory. Seems like probably not.

              I think I need to publish those screenshots …

              I can see a two level hierarchy where a shell + GUI uses the FANOS protocol, and then the GUI communicates with X or another display server in some other manner …

              Although it is true that this gives you the “multiple multiplexer” problem which I am annoyed by. I have been discussing that with Oil users – shell has job control which is an extremely primitive non-GUI multiplexer. And then you also have screen / tmux at the terminal level, and then something else at the GUI level. So I definitely see the motivation in unifying them.

              1. 4

                And I’m not sure if you can pass a file descriptor between two processes over shared memory. Seems like probably not.

                The IPC system design was to fit the set of primitives available to Win/Linux/BSD/OSX/Android with as little “per-OS” details as possible (up until 0.4 a Windows port was also maintained, after the n’th compiler bug on atomics/alignof/… I stopped). On Windows you can DuplicateHandle into another process, and then pass a shared identifier over some other channel (like shm). This is infinitely better than the nowadays defunct STREAM (pipe++) fd passing or the aux data of POSIX sockets.

                Hmm I can see why that kind of protocol would be useful for GUI apps to communicate with the display server, but I feel like the shell <-> UI deserves a special case that is small and well specified.

                The trap is Simplicity is Systemic - although the description is extremely dense (being an overview and something to expand when I have the time). I argue it is the main culprit to the situation the desktop is currently stuck with on most fronts. The amount of ‘almost’ the same thing being reimplemented again and again in infinitely many simple ways is staggering, and the ugliest of bugs.

                If you write down just the requirements / expectations from an advanced user (matklad’s link in this topic is a fair one) and use that as a rough checklist to verify any designs against. Add some reasonable non-functionals:

                • Responsive - not once should it stall or fail to respond to a user input in a timely manner because the shell, the communication channel or any if its children are busy with other processing.
                • High throughput - (Paste a 10GiB file into shell CWD should not be problem).
                • Network transparent - locally and remote.

                I think you’ll be hard pressed to find something “simple” that doesn’t also invite sidebands that bring back the complex legacy you just tried to get away from.

          2. 3

            I heart everything Arcan in theory(that I’ve seen/read), one of these days I’m going to have to put it in to practice and actually run it. I just don’t think it’s gotten to the point where I can be lazy and just pkg install arcan or apt install arcan and everything just works?

            1. 9

              FWIW, you can do apt install nix && nix-env -i arcan
              (there are a few commands in-between to initialize Nix which I left out)

              1. 5

                Better yet:

                $ nix-shell -p arcan.all-wrapped
                $ arcan welcome
                

                not sure the newer flakes way of doing it with nix develop or whatever…

                1. 3

                  Awesome! I use nix! Thanks!

                2. 2

                  I know of a few projects that look into it, though I’ll leave communication to the parties involved – when they think it appropriate. For what I want (principles -> design) the user accessible thing will have to be a tool for building custom live-images with a handful of UX presets for specific device classes (htpc, mobile, embedded, desktop, …).

                  Excluding sidetracks, the remaining posts before that:

                  • The Day of a new Command-Line interface: Curses!
                  • Bringing X along for the ride: Window Management
                  • Arcan Explained: A Browser for a Different Web
                  • Daybreak: Arcan as OS Implementation

                  Excluding one really hard problem, the rest is figured out and it is more “getting it done”. Time will tell if I run out of steam before then.

                  1. 2

                    I hope your steam keeps you going for a good while! Thanks for all of your hard work and your continued postings!

                3. 1

                  For starters you need an Arcan build that comes straight from the source. The things needed here are really new, not covered by any release, and is actively worked on. Arcan is a pain to build from source and, if you want it to replace your entire display server, also a pain to setup. Twice the fun.

                  So, I wonder what the viability of an Arcan based distro would be? I get that maintenance is probably the bottle neck here, but what would be truly amazing, and would open the door for all of your average Joe’s to try this out, would be to release a preconfigured Linux distribution that runs Arcan out of the box. Sure, this might be asking a lot, but this would be the ticket in for many faint hearted onlookers.

                  1. 1

                    I think the bigger problem is IMHO, coming up with introductory materials. Everything I read about Arcan feels like a cipher, and I should definitely understand windowing systems (even if most of my experience is elsewhere).