1. 32

    Personal opinion: The GNU tools are superior to all other implementations for the USER. It is called “userland” because it is about the users, not about some mysterious POSIX compliance. I even install them on macs, because they are just better. I do not care one bit about the FSF or RMS or whatever, I use the GNU tools because they are better. I think back to the dark times I had a Solaris 8 box at work and everything was oh-so-POSIX compatible. Thank you, but no thank you.

    1. 21

      Yeah, what this author seems to be unaware of is that POSIX does not say that you can’t implement extensions, and in fact that’s basically expected. POSIX codifies things that are already widespread practice. Features go into code and then into POSIX, not the other way around. If you try and get some new feature into POSIX, they’ll tell you to go away because it isn’t in any implementations yet. So this:

      POSIX certified means that these OSs not only comply with POSIX but don’t add any compatible extensions. GNU, on the other hand, is POSIX compatible, but adds extensions (e.g. long options, options and behaviors that aren’t defined by the specs, etc.)

      is not correct, for multiple reasons. POSIX certification does not mean that you can’t add extensions. On the flip side, GNU is not POSIX compliant! The mere existence of the POSIXLY_CORRECT environment variable proves this.

      I found that the further I got into this article, the more inaccuracies like this there were. By the end I was pretty frustrated because it appeared that the author was speaking with authority (and a poorly hidden emotional undertone) on a subject that they actually were not an authority on.

      1. 13

        Indeed, the post links to Drew DeVault’s post about glibc isalnum segfaulting, and the pointed glibc weirdness is there exactly because of localisation, which is falls under the umbrella of, usability.

        The GNU tools put emphasis on usability above all else and that is a good thing. Alpine, borne out of BusyBox, doesn’t even come close. Granted, Alpine/BusyBox have their uses, places where humans aren’t often needed, but on user machines the GNU userland is superior.


          The linked article was discussed here.


            These days human usability is mostly about the GUI though. And the corresponding high-level tooling. For low-level tools, portability/simplicity/consistency are becoming bigger concerns. Who even expects all the localization things to be in libc?! I always thought that any localization stuff is the domain of libraries like ICU and standard libraries of higher-level languages, it never even occurred to me to look in libc for that. Every time a low-level library tries to do that kind of thing, it just results in nasty surprises like the infamous “Turkish İ” problem.

            1. 10

              from that comment I would guess that your native language is English, so you have no use for localisation in your shell. For many, many people that is not the case and they benefit greatly from the support of even the lowest level tools. It makes it more accessible to more people.


                As was discussed on this site a year ago (see my comment above), glibc supports locales because locales are part of the C89 standard.

                You might design your software to move locales support “up the stack” but glibc does not have that luxury.

                And a libc that does not support locales is strictly speaking not standards-compliant, hence probably not portable.


                  Depends on how you use your computer I guess. I do most things from the shell, so usability of the shell and shell tools pretty much is usability for me. This is of course not how most people use their computer these days, but there’s still a fairly decent number of us, and when GNU started out it’s how the majority of people used their computers.

                  The shell has always served two somewhat conflicting purposes: scripting and direct usage.


                  It’s ironic that you use localisation as an example because I recently got bitten by GNU cut not actually being encoding-aware: the --characters and --bytes options do the exact same thing! BSD cut doesn’t have this problem.


                    Alpine, borne out of BusyBox, doesn’t even come close

                    Busybox doesn’t seem so bad these days; I’ve ran some systems on busybox, and I have to say I hardly notice the difference. It implements POSIX and most useful extensions. My biggest gripe is that it doesn’t allow configuring the ls colours without a recompile (the defaults don’t work well on light backgrounds).

                  2. 10

                    Yeah the statements about POSIX in this post are silly – one of many serious misunderstandings. This is a low quality post.

                    POSIX certified means that these OSs not only comply with POSIX but don’t add any compatible extensions.

                    Huh? That’s not what POSIX means. Also, POSIX is generally old, and outdated, and people need to do things with their computers. It makes zero sense to implement only POSIX. This is a weirdly religious idea.

                    A recent example I gave is that GNU awk supports match() for submatch extraction, but original awk, mawk, and POSIX don’t. You should use this; it will make your awk scripts better.

                    Another one is the paper about how modern Unixes don’t actually use POSIX for IPC and such because it doesn’t do what they need (binder on Android, DBus on Linux, something else on OS X).

                    Open source extensions are good because they’re … open source. Vendor lock-in is bad, but the best you can do is give the user the right to modify, fork, redistribute, etc.

                    That said, I do think GNU code is “inexplicably bad” (a statement I heard somewhere else). There are many collaborative open source projects like CPython, LLVM, etc. which have higher code quality, not to mention the ones done by fewer authors (sqlite, Lua, etc.)


                      Personally, I used non-POSIX commands such as head --bytes N (the example in the article) from the shell so many times and they saved me a great deal of time.

                      Would I use head --bytes N in a script that is going to be published? No. I’ll use cut -b -N or something that is, if not 100% POSIX compliant, at least widely compatible.

                      Am I happy that non-POSIX features such as head --bytes N exist? Yes. I can use them while prototyping or in some quick-n-dirty pipeline.


                        My reasons for disliking GNU are the exact opposite of this article: I don’t think they do nearly as much as they could to improve UX. It’s “good enough” I guess, but that’s not the same as “good”.

                        POSIX was always intended as a “minimum feature set that works the same everywhere” type of specification, originating during the great Unix wars. It was never and will probably never be at the forefront of innovation. It’s a dictionary: it recorded what “language” people were using.

                        These days I just write most scripts in /bin/zsh, because it’s just so much more useful. Want to run it on OpenBSD? No problem, install zsh! Why zsh and not bash? Because zsh is clearly the better shell in almost every way and always has been really. A lot of “bash features” were added to zsh first (and both of course borrowed liberally from the then-proprietary ksh) and the only reason bash seems to be the de-facto default is “because GNU”.

                        The article complains about head adding -c. The way head and tail works is just weird, I can never remember how it works. If I were to write a “coreutils” package then I’d add POSIX head and tail for compatibility, and then move on to a much more useful lines or something which has a UX I can actually remember and which includes all the features from head and tail except better.

                        Or in short: POSIX is stuck in 1980. GNU is stuck in 1990.

                      1. 4

                        So between in-band information (like ANSI escape codes) and out-of-band information (like signals and ioctls), GOOD LUCK WRITING A TERMINAL EMULATOR.

                        Actually you don’t have to mess with the signals when writing a terminal emulator. The kernel does that. You just send the ioctl to inform the kernel that things changed (TIOCSWINSZ); Though you might send the signal yourself additionally. What’s annoying is if there’s another foreground process the signal goes to it then your other ones won’t necessarily be informed. My terminal emulator tries to hack this by resending the thing when the process changes… but that’s hard to do too since the terminal just gets pipes without necessarily having all the info.

                        The linux terminal is a mess. Frankly I wish it would die. But it also does useful things and has a huge ecosystem sigh.

                        1. 2

                          What is “the Linux terminal”? Honest question. Are you referring to terminal emulation? In which way is that Linux specific?

                          What would you do different? Do you mean that a text repl as an interface tot he system is a bad idea per se? Why? What do you suggest instead?

                          1. 18

                            What is “the Linux terminal”?

                            The abstract concept of terminals as used by Linux programs. So the fact that it is random sequences mixed in to a pipe where neither side knows much about the other, with the kernel in the middle doing its own thing.

                            Of course it isn’t actually Linux specific but I thought this was an easy and convenient way to communicate the ideas used there as opposed to the ideas used in a Windows console application or an X application, etc.

                            What would you do different?

                            Almost literally everything. It is awful from top to bottom. Everyone loves to hate the Windows console subsystem (mostly because the end-user ui was questionable until kinda recently, not even allowing convenient resizes (you had to go to the properties sheet instead of dragging the window like everything else? Bah!)), but the technical foundation was much nicer, with a considerably less error-prone input API, though the output API became a bit of a pain to use, it was also at least more reliable.


                            I guess I’ll start there. Linux terminal input api is “read bytes from pipe”. The interpretation of those bytes is extremely complicated since it is technically undefined; it can change based on what terminal you have. Nowadays though most everyone emulates the xterm (which itself kinda emulates the VT100 among other things, but it is really xterm that sets the standard today), so that simplifies things a little. A little… rxvt is still pretty respectable but has a slightly different input sequence. And until recently, the Mac OS Terminal was slightly different too. And then different people’s xterm clones also can offer or not offer their own things.

                            But still, let’s say you just want to read xterm and call that good enough. The sequence is still a massively complicated state machine with very little consistency! The byte 27 means expect extra info… except when it doesn’t because the user just hit esc. Those sequences are variable length and the end character depends on what the next few characters were, so if you try to implement just a few and skip unknown ones… there be dragons. You can do it but even skipping unknown things takes a lot of preprogrammed knowledge. Once you do read a sequence successfully, their interpretation may depend on global settings in the terminal which can be left behind from the previous application. Fun.

                            Then once you decode it, various pieces of information are not even present. Want a key release event? Too bad. Want to know if it was enter or shift enter? Need an extension (and this is why most thing emulate xterm since at least then you can kinda standardize on extensions… but there’s no guarantee it is present and querying this is a pain even when it is possible, which it often isn’t).

                            Ever had a program crash then moving the mouse across the terminal outputs spam? Or find you type and nothing happens? now you know why. The terminal can’t tell the program crashed and reset itself, no, it is stuck with whatever random global state it was in then and the other application may not know how to interpret this data stream.

                            Contrast to the Windows console where input events come as structured data, creatively named an INPUT_RECORD struct. The length of these is told to you so you can skip them easily if you’re not interested. Extensions can now be easily added and applications won’t be lost. Much much nicer. Most gui things work like this too. So I’d take structured input data in my new system over the escape sequences on input. (Of course, it does burn a few extra bytes on the wire. Realistically they could have afforded this on the input even back in the physical hardware at 300 baud days but you can understand things were different then.)

                            Speaking of input events, the general terminal paradigm when programming the client application is to block for input. You know, scanf("%d", &a); printf("%d\n", a);. This isn’t necessarily bad - sure can be convenient - but it really does suck for anything more interactive. Most linux programs do some kind of event loop even with the terminal anyway. But then scanf() stops working. Now I hate scanf anyway, but again Windows indicates things could be better… on Linux, unless you work the terminal yourself, e.g. GNU readline’s implementation, input sucks. User types in that scanf prompt, makes a mistake. Backspace… just adds more spam to it, digging them in deeper. Unless you change to “raw mode”, the kernel, not the application, and usually not the terminal either (though it actually kinda can be there, this is not the standard) is doing that input sequence and it is the stupidest possible thing that can work. On Windows, by contrast, when you ask it to just read a line in cooked mode, backspace works, arrow history works (though it is janky, it is there), even tab complete is available in the standard api (though again … not great and fairly rarely used by third party applications, at least they tried).

                            I honestly don’t know why Linux cooked mode doesn’t offer at least something in terms of user experience, since it is all line buffered external to the application anyway. But like I probably would change this too.


                            Outputting data to the linux terminal is just as problematic in theory as input, but in practice it isn’t so bad since most of that is de-facto standardized….. but not all of it. Like the link mentioned here, deciding to add color or not is often done with a isatty() check or a random switch. The sequences themselves are still a state machine that can be corrupted by a program crash or whatever, and the terminal implementation to read them is still a pain.

                            The Windows way of communicating out-of-band metadata brings its own problems. You write text, then in a different function call, change color, then write more text. Bleh.

                            What I’d do is make it take structured commands. This might be length+command+data sequences (which you can imagine they didn’t do in the 300 baud days for bandwidth reasons, but nowadays we’d be generally ok on it) or heck maybe even something like XML just to trigger the haters, but in all seriousness, XML would be quite suitable for the task, given its standardized extensible markup and metadata capabilities, allowing receivers - not just terminals anymore, but any receiver including programs in a pipeline - to strip things they don’t care about. Writing <span color="red">Hello world</span> to either a pipe or a terminal would work since the receiving program can simply strip out that color data. (Of course you could in theory do that with ansi escape sequences too but you’d need to know they are there - which is communicated out of band right now - and the rules for stripping them which is again fairly complex since they come in several forms, but to be fair, parsing XML is not any easier. Anyway my real point here is to put more structure on it, the exact form of that structure isn’t terribly important.)

                            META COMMUNICATION

                            If you want to ask for the size of a terminal, or the current cursor position, or other similar things now as an application, it is a bit of a pain. You need to flush your buffers, write out the request in the same output stream as everything else, then read back - possibly buffering and/or still processing unrelated input that was already in the stream in the mean time - the answer, which comes as some of those escape sequences. If it comes at all…. maybe the recipient doesn’t understand the request or doesn’t care to answer. So you need to work in some kind of timeout, but not too quick or it will mysteriously fail on ssh connections, then when the esc sequence does come and you aren’t expecting it, you’ll perhaps read it as the user hitting other keys. What a mess.

                            If your input and output channels are structured data though, you can work it in here with less trouble. Even piping it into less could take the data and discard it for you, simplifying some application development.

                            Currently, if the terminal wants to know about the application, it might be able to ask the operating system - list open files of your pty or use other extension things. Or maybe the shell will be configured to inform you. But there’s no real hope doing it in-band, since applications wouldn’t expect such a message to come. You could, in theory, send a pseudo key sequence but the other side needs to understand it too, so unless you control the whole stack it won’t work (and now you know why I wrote my own terminal stack lol. ok not really this in particular but it is nice to have the option sometimes). Structured input and output may not necessarily give you an answer - the application can still ignore the request (like odds are the implementation of scanf would ignore anything that isn’t a “give me a line of text” input answer) - but at least you could more confidentially try without worrying about corrupting the stream.

                            I would standardize the terminal getting knowledge about what application is talking to it. Then if it crashes, it can at least reset to the previous state automatically and carry on.

                            UI PARADIGM

                            Most of the above is about the “bottom” end of the system, the implementation of the terminal and communication with it. But I said things suck top to bottom and I meant it. There’s three main kinds of linux terminal programs: full screen interactive ones (e.g. vim, mutt, irc client), interactive shells (bash, python, mysql), and batch processing pipelines (grep, awk). Some are a little bit of both, like less is full screen interactive and tends to be part of a pipeline, or you can run python interactively or part of a pipeline, etc. These generally branch on the value of isatty on either input or output explicitly, thus you might need to either openpty or --color or whatever to override its decision. Applications that fail to do this don’t work very well.

                            Moreover, the very concept of command line arguments gets dicy. touch -- -R; rm * is the classic example - rm doesn’t know if -R is a user-provided switch or a shell-provided filename. Windows, again does this differently - over there, the command line is just a string and the program is responsible for interpreting it. On both systems, pipes need special care and the interpretation of that data gets iffy. On Windows, a pipe might be UTF-8 whereas the console output is UTF-16. How do you know this? Read a couple bytes and guess. lol but not kidding.

                            Neither way is really great, but the Linux way has had pretty good real world success in being both scriptable and usable interactively, with surprisingly nice extensions offered by innovative shells, all the while being fairly easy to add your own programs to it, so we don’t want to throw all that away.

                            And that’s the real problem to making a big change here…. realistically, you’d still have to support the old ways anyway.

                            But if we’re in fantasy land, what I’d love to do is to standardize a number of flags and some of the output. For example, make --help mandatory and define it to output a certain structure so people and machines can read it. Then you can tell what the possible arguments are and what formats they take… then make the interpretation of arguments standardized for the program, so no more rm * confusion. These things together means you can script it in various forms - shell can automatically offer auto-complete by parsing --help in the background. GUI programs could even pop up a dialog box for it automatically!

                            The output, being structured for the terminal already, can be structured for other pipelines too. Metadata like color ignored by pipes while other metadata might be consumed by them. Windows Powershell does this kind of thing, though I think I’d keep some kind of primarily text output - hence my earlier suggestion of XML. XML with tags stripped is plain text again and, if done right, still readable bare in a zero-feature terminal (just like how websites can be easily read by the most basic text browser. right? RIGHT?).

                            The batch pipes and interactive shells can both be used this way. To support pipes or interactive, you can just output <user-interactive-prompt-for-line>Prompt $</user-interactive-prompt-for-line> and then read lines, whether from the prompt or from a pipe. Let the pipes strip the user-interactive part themselves.

                            So I’m trying to embrace higher level the whole way. If you want pipeline batch programs, embrace that and make them explicitly known. If you want fullscreen, that’s similar to what it is now, just easier to program since you don’t need 5000 lines of library code to parse the stream. You could end up a system that still does what Linux terminals do in practice but better.

                            But alas it is all just fantasy anyway.

                            EDIT: I wrote readf when I meant scanf so changed that over (readf is the D language version of scanf but since this isn’t for a D audience I meant to stick to the C names. oops.)

                            1. 3

                              Thanks for the fantastic write up.

                              I would standardize the terminal getting knowledge about what application is talking to it. Then if it crashes, it can at least reset to the previous state automatically and carry on.

                              Note there’s another dragon there, because the terminal supports multiple processes talking to a single terminal concurrently despite having a single global state machine. I think fixing the state issue also implies single terminal ownership. Also, in addition to resetting state on process exit, it should ensure default state on entry; currently child processes inherit the parent’s state, so we have things like processes expecting cooked mode on entry despite a shell process not using that mode.

                              over [on Windows], the command line is just a string and the program is responsible for interpreting it.

                              That’s true, but it’s become a total mess. On Linux, a file name with a space can be parsed by a shell and put in a single argv entry. On Windows, it must be quoted so the child process can parse it. But that means the parent really needs to understand the child’s parsing rules, and the parsing rules are inconsistent.

                              The Windows way of communicating out-of-band metadata brings its own problems. You write text, then in a different function call, change color, then write more text. Bleh.

                              Somewhat meta, but I’d rephrase it and say that the Linux terminal was designed for a VT100 serial line (so everything is in-band) whereas Windows was designed around an RPC model where different message types exist. If you squint a bit, the real difference is ordinary text/input/output being encapsulated in some kind of message structure - this isn’t obvious because Windows doesn’t expose a single stream of data, but it’s the obvious way to turn those function calls into a single stream of data.

                              The big problem in the Windows model is now the terminal emulator is its own process rather than the kernel, you write text (and synchronously context switch to the terminal and back before your program can continue), change color (and synchronously context switch to the terminal and back before your program can continue), write text (and…

                              It’s a shame that the Windows functions weren’t converted into a single stream (on the application side) which would have made SSH type protocols simpler, and eliminated the context switch madness.

                              Input events come as structured data, creatively named an INPUT_RECORD struct…Extensions can now be easily added and applications won’t be lost.

                              In theory, yeah. Except it’s Windows, compatibility is everything, and you don’t know what assumptions some random program made. So what actually happened is console windows became resizable but no new event is generated to indicate that. Instead the existing “buffer size changed” event is overloaded, even though the buffer size isn’t changing, just to poke client apps to know something happened. Back when resizing was via the properties dialog, applications got no notification at all. (I have code that polls to detect resizes.)


                                I think fixing the state issue also implies single terminal ownership.

                                Yeah, or at least put the state per-connection if you do let output be combined.

                                But that means the parent really needs to understand the child’s parsing rules, and the parsing rules are inconsistent.

                                Indeed. What I’d actually like with the standardized metadata is that the parent DOES understand the child’s parsing rules at least to some extent, but then it is consistent. Ultimately it would be nice if the argv itself was separated a bit for options vs operands at least (could standardize on the -- thing most things already do).

                            2. 2

                              The interface you have to work with in order to implement a terminal ‘emulator’. The REPL or whatever your ‘shell’ provides doesn’t really factor into it much (more than it blocks possibilities for improvement) though it has evolved around the limitations of the forest of terminal protocols so are in many ways tainted.

                              There are subtle, special semantics for how all the infrastructure is setup to allow your command to talk to your shell to talk to your emulator to talk to this ghastly reanimated corpse of 60ies tech. There are subtle and breaking nuances between behaviours between the different UNIX derivatives. There are functions and interfaces to be used that never or rarely appear elsewhere and talked about hardly ever. Every attempt at a layer has hacks around misbehaviours in what is perceived to be above it and what is perceived to be below it.

                              For just a “fun” one out of many. Take ‘ncurses’ - the popular TUI library. It still has a ‘bandwidth estimator’ that tries to work with the ‘baudrate’ (stty(1)) of your virtual serial port talking to the pseudoterminal device deep down in the kernel to figure out the cost of ‘draw commands’ (cursor to this position, …) so that it can try and buffer just the right amount for the window to be drawn without the cursor stopping half-way through. This estimator needs to know what each draw command translates to and that is dependent on the flavour of terminal protocols available and the quirks of the emulator used, so it checks an environment variable for a terminal emulator name so that it can look those up in a database (termcap(5), terminfo(5)) of the set of hacks to enable/disable.

                              That is for moving the cursor around and writing characters with varying foreground, background and formatting (bold, italic around). There is also signal propagation, flow control, character sets, mouse cursor, clipboard, different screen modes, … There are reasons why xterm is 65+k lines of code and you don’t want to know about any of them.

                              1. 2

                                There are reasons why xterm is 65+k lines of code and you don’t want to know about any of them.

                                Well to be fair a lot of this can be simplified if you look at the de-facto standard today. My terminal emulator core is only 5,000 lines of code and my ncurses replacement is 8,800 lines of code (though a good chunk of that is my gnu readline replacement which comes in at about 2,300 lines, but like since cooked getline is so bad a terminal program without something like readline is borderline unusable, hence i integrated it there, and then there’s the Windows implementation and some integrated emulator stuff that pads it out further).

                                Though I guess xterm speaks X directly too and I put that off in a separate library… which is 20,000 lines but well that does a LOT of things and has bundled bindings and Windows and Mac support too. So it could be much smaller if it was just what the terminal needs.

                                It is still a lot of messy code on both ends though.

                                references for my versions: github.com/adamdruppe/arsd terminalemulator.d is the terminal emulator core that does the emulation based on pipes and in-memory (the front-ends are actually in a separate repo here https://github.com/adamdruppe/terminal-emulator but they’re relatively simple since the messy part is in the core lib). terminal.d is my client api. and simpledisplay.d is my window library.

                                Having done it all though on both sides and high and low level has given me a lot of knowledge that I kinda wish I didn’t have.

                                But still a de-factor simplification is < 10k lines and still works for 99% of what you’d actually do nowadays, so a fraction of all the things xterm and ncurses have implemented over the years.


                                  try and buffer just the right amount for the window to be drawn without the cursor stopping half-way through

                                  Wow. Guess things like that is why ncurses is so well… cursed big and stuff.

                                  Meanwhile good terminals are starting to support sync sequences so we can do things like Kakoune’s smart diff based rendering reliably :)


                                    Where is the list of clients implementing it, and what trigger should one use to probe? toggle altscreen is an obvious one, but that wouldn’t be sufficient for tmux to vim – for instance. I experimented with “vsyncing” through XON/XOFF, but as with everything else it just becomes heuristics that break in some or many edge cases.

                                    From that issue: This would indicate the linux console renders every intermediate state instead of what most terminals do: render once there is no more terminal output pending.

                                    Just using pending output to vsync would make find / a very unhappy command (and anything with buffer backpressure really but that one is the obvious test target).

                                    Others go galaxy brain with https://github.com/cipharius/kakoune-arcan – and in return can fullscreen a window to get back to drawing single ‘racing the beam’ buffer with glyph cache shared between all clients using the same typeface, pt size and dpi.


                                      list of clients implementing it

                                      I’m not aware of one, only a list of terminal emulators.

                                      what trigger should one use to probe

                                      I’m not sure what this means. I’m not aware of any test cases where sync solved a practical problem (glitch) other than the kakoune one.


                                        Wall-of-text warning to avoid assuming background, paragraph 2 and 3 from the end are the meat.

                                        The problems are the same between terminals and other display servers. It’s one of those overarching pesky things in the same vein as ‘colour correct workflow’ - synch is quite hard; almost always the hardest one.

                                        You have a producer working with a command-stream (cursor_to:dx,dy, write:abcdef) (cursor_to:dx2,dy2, write-bold:ghjkl) … and a consumer (server side) which interprets the stream and translates it to the native language of some output, possibly mixes these ‘signals’ together (composition).

                                        If the output imposes a deadline (old printer outputs typically didn’t, just flow control/small-to-no buffers - avoid ‘back pressure’ from a producer sending faster than the chain can handle and you are as good as you can get. With monitors you get binned intervals to fit (vertical/horizontal blank and sync) and then stalls if you miss your timing window.

                                        Terminal protocols adapted, so now you have two different screen modes - the line oriented one and the ‘altscreen’ that works just like a row-resolution tilemap, the tiles just corresponds to glyphs from fonts. As the protocol accumulated more and more features, the processing cost for each of the cells varied and you started to miss the timing window or the command buffers were even larger than what can be processed in one update. Here you get tearing, and you have no other option. It is just less noticeable with a slowly changing signal so user annoyance in terminals is less than other forms of graphics.

                                        Worse still, you also get input to display latency from a typical ‘render loop’ on the producer side as a while(ev = wait_for_input()){process(ev); draw_what_changed(); } format when the ‘draw’ part impose higher costs and block (serial link and all that) while input is accumulated and released in bursts (“embarrassing pause”, “jank”, …) with possibly noticeable variations in timing between the two (jitter) attenuating the effect. This is the problem from the user perspective, it is a QoS one - if you don’t stress or test outside your ‘normal use’ you might not see the problem and cannot properly evaluate the solution.

                                        Not having agency to fix clients, hacks are added to the compositor side to determine if and when it should forward. The “counting characters” trick from ncurses is such a hack, and makes somewhat more sense when you have slow/variable latency connection (so networked). The “double buffer” trick from standard graphics makes more more sense there - but both are hacks.

                                        Other ones were to check which mode the terminal emulator state machine itself is in and guess how many characters you have left to work with; If there is a small burst of data coming you likely have a smaller region update, a larger one - break when you run out of time to the next synch deadline or after about as many character updates as there are cells on the screen. The you need to add a refresh deadline as well because there aren’t always natural synch points in the data stream, find / being a simple example.

                                        So I that is implicit synch and first stage explicit synch – next up is fencing as you ramp up asynchronous processing. You insert command markers to indicate when it is safe to process and forward. The consumer waits for those markers. This is across a privilege boundary, so if you don’t have a timeout you have a denial of service vulnerability as the producer can always “forget” to mark the end.

                                        That is what the protocol change you link suggested. Here comes legacy and feature detection. The producer asks if a mode is set (DECRQM) and gets back something that says ‘zwp_linux_explicit_synchronization_v1’ is in the ‘wl_registry’. Now comes nested composition, which is the default in the terminal stack. The terminal emulator only sees one client in a possibly deep chain or one where the end keep being swapped out. If one producer dies; is killed; shuts down; detached; backgrounded etc. after a (CSI ? 2026 h) the denial of service part kicks in and I see nothing in the suggestion to detect that has happened.

                                        The follow up in heavier system graphics is ongoing now and has been chugging along for close to a decade. Fences aren’t granular enough -> timeline semaphores. Display timing keeps drifting -> cancellation of submitted buffers.

                                  2. 1

                                    But all those predate Linux, no?

                                    Nonetheless, Interesting reply. Thank you. I wonder if there have been attempts at creating a new simpler and more robust ‘wire protocol’ for text terminals.

                                    1. 3

                                      But all those predate Linux, no?

                                      Yes, but there are new generations of Linux users/developers that know of little to nothing else. The author of the link is either one from that group or speaking directly to it. Network effects pushing out diversity.

                                      Thank you. I wonder if there have been attempts at creating a new simpler and more robust ‘wire protocol’ for text terminals.

                                      It’s a 2D problem – you want better solutions for both the vertical (command to shell to desktop) and the horizontal (between commands, cat foo | grep bar). Solving for the horizontal is more intuitive and approachable (typed data exchange), so there are several of those, but you are still bound by a ton of restrictions. Eventually trying to work around those will again add ‘inband signalling’ with desktop interaction mixed in with the data you want to transfer, and anything in the pipes and filters chain can screw it up royally - much akin to what cat /dev/urandom does to your terminal emulator today. That will still not be enough for the ‘resize’ case and sideband signalling gets thrown in again in some form. Say hi to the new boss that looks strangely like the old one but with more JSON.

                                      Solving for the vertical takes you into display server protocols (desktop IPC) with the added twist of special needs for text rendering and grid-of-cells/list of lines. There is even more OS cruft here and even more divergent quirks (hello systemd…). I outlined one possible solution several years ago (The dawn of a new command-line interface) – it has been working for quite a while, I just forgot to tell about it and now don’t really want to.

                                      1. 2

                                        Yes, and it predates Unix as well. When people say “Linux terminal” they just mean “the terminal emulator I am running on my Linux system”.

                                1. 4

                                  Yes, you read that correctly—you mount the array using the name of any given disk in the array. No, it doesn’t matter which one

                                  What the actual f… how was this deemed acceptable by anyone ever?!

                                  1. 6

                                    Why do you think it’s a problem specifically?

                                    1. 4

                                      It’s extremely confusing. If a disk device is used in an array, mounting it should return EBUSY. Mounting a disk and mounting an array with the disk are semantically different. A disk != the array containing it.

                                      1. 2

                                        This idea is not set in stone. It’s just something we’re used to. Fuse started to change that - you can mount a config with credentials to talk to a database, you can mount a key file to publish DNS records, etc. Mount just takes a file descriptor and the selected driver and says “now kiss”.

                                        I agree it’s not common, but it can be explained in one sentence, so not that confusing. It saves you from an extra utility otherwise needed to create the array device, which is nice.

                                    2. 2

                                      This is one of the least offensive parts in an article filled with WTF; seems like an odd thing to focus on.

                                    1. 5

                                      Why “unix”? No windows allowed?

                                      1. 30

                                        This sort of desktop screenshotting is a uniquely unix phenomenon in my experience, because there are so many possible window manager configurations that are possible. It’s useful to see the variety of options available to you.

                                        Sharing Windows screenshots would be like showing up to a car show with a stock 2016 Toyota Camry.

                                        1. 7

                                          Sharing Windows screenshots would be like showing up to a car show with a stock 2016 Toyota Camry.

                                          I don’t think this is true, or at least it used to not be true.

                                          Back in the 98-XP days, Windows used to have a thriving ecosystem of alternate shells (I ran BlackBox on my machine for a while), as well as tools like WindowBlinds (and other stuff from StarDock, I think it was) that let you mess with window decorations and other fun stuff.

                                          1. 7

                                            Having actually used WindowsBlinds (not back in the day, but on my old systems for hyucks), it is super jank and caused a LOT of instability. I’ve found it was better just to stick with colour themes for the normal rendering, or deneutering uxtheme. (And the real reason someone would want to do that is so you could use Watercolor :) )

                                            Of course, tweaking/ricing is a lot less appealing than other stuff now that I’m no longer a teenager…

                                            1. 3

                                              Those days are pretty long gone. I know you can hack it but it’s nowhere near as common as it is on Unices.

                                              1. 3

                                                Yeah, this - even without running alternate shells, XP was very skinnable. Really miss those days, to be honest. Windows is so boring now.

                                                1. 2

                                                  I find it strange how quickly people forgot it. “Windows XP looked silly”. No, it looked like you wanted it to look, and you could change the theme in two clicks.

                                              2. 2

                                                As the others already mentioned, there was a time when no one (statistically) posted these screenshots from unix because enlightenment was the only fancy looking thing.

                                                But I’ve not seen customized windows for many years and I also stopped doing it (on any OS). Mostly using the default from i3 now, sometimes changing the bar color.

                                                1. 4

                                                  Compiz was the golden age of fancy looking unix. The desktop cube was so popular…

                                                  1. 3

                                                    Have a look at Wayfire, a 3D Wayland compositor inspired by Compiz and based on wlroots: https://github.com/WayfireWM/wayfire

                                                    1. 6

                                                      I am in fact one of the contributors to Wayfire :)

                                                  2. 1

                                                    I distinctly remember GNUStep desktops being quite popular to screenshot in the early 2000s. That was released around the same time as the first E release.

                                                    1. 1

                                                      Wikipedia agrees with my memory that I used it in 98, but I don’t claim any authority on what was around, was just getting started with Linux.

                                                      1. 1

                                                        There was a phase where the blackbox/fluxbox family of window-managers was very popular in the forums I was hanging out in back then. Good times!

                                                1. 3

                                                  In French and German it makes more sense: Informatique/Informatik. An instance of how language can shape your perception.

                                                  1. 2

                                                    This always seemed excessively focused on the object of the discipline (information) rather than the discipline itself. Just as I think the term “computer science” focuses on the means used by the discipline rather than the discipline itself.

                                                    1. 1

                                                      “Informatics”, then, perhaps?

                                                      I kind of like that.

                                                      1. 3

                                                        It’s kind of already a thing - I work at a “School of Informatics”. Most university departments in anglophone countries still call themselves computer science though

                                                        1. 1

                                                          It’s kind of a silly term – one of the most overloaded ones ever:


                                                          1. 1

                                                            Engineering informatics (not to be confused with informatics engineering, Southern European term)

                                                            That made me laugh. In general though, I think it doesn’t really matter what it’s called. Computer science or Informatik here in Germany, people know what you are talking about and that is good enough.

                                                      1. 0

                                                        Wait, Windows has had automatic root updates since XP? And nobody else does? Why is everything worse than Windows XP?!

                                                        1. 13

                                                          I’m not sure what you mean by automatic updates, but ~every Linux distro has a ca-certificates package which gets updated all the time. (And even when it’s EOL, you can use it from newer versions since it doesn’t have dependencies)

                                                          1. 4

                                                            Except for Android, I guess, which doesn’t get them updated until you get a whole new image from the manufacturer.

                                                            1. 2

                                                              But as stated in the official LE post, you can (and probably should) add the root cert to your app-local trust storage (and probably even use cert pinning). That won’t help with browsers, though I’d guess you just install firefox for android (or something like that) and they ship their complete own TLS stack (and certs). Because you can’t run any relevant TLS on android 4.4, which you want to support for many apps..

                                                              1. 2

                                                                FYI: you can still order stuff from Amazon and use Google Search on Android 4.4. Mozilla’s own website works, too, even though they suggest yours shouldn’t.

                                                                If your own website, and your employer’s website, don’t work on Android 4.4 because TLSv1.0 iNsEcUrE — it sounds like you’ve been sold the snake oil!

                                                          2. 0

                                                            Windows has automatic root updates?

                                                          1. 4

                                                            Most audio applications really need floating point performance. Not, mind you, modern floating point performance, but nevertheless, more than is usually available in soft-fp systems.

                                                            Can we ask what your intended use is?

                                                            1. 3

                                                              Soft fp is also less power-efficient by definition, in addition to old soft-fp-only chips being less efficient due to, uh, being old – made on an older process.

                                                              1. 1

                                                                The intended use is a mobile personal computer (what we used to call a PDA) for blind people. So no screen, but a physical keyboard (probably a braille one) and audio output, including text-to-speech. It was possible to do good text-to-speech on 90s PCs, so I’m not worried about having enough processing power for that. Being able to play lossily compressed audio formats like MP3 is also important. Encoding audio in a low-bit-rate format such as Opus would also be needed for some applications, such as voice calls or recording voice memos.

                                                                Edit: Being able to run one of the big three web browser engines, or a GUI of any kind, is specifically not a goal. I’m curious about how good battery life can be if we don’t start with the assumption that the device has to be compatible with the mainstream web.

                                                                1. 1

                                                                  OK, so your use case is:

                                                                  • Stereo audio out, microphone in (mono or stereo)
                                                                  • 2+ USB2 ports for braille display and keyboard
                                                                  • wifi for networking
                                                                  • long battery life
                                                                  • no need for video display or touchscreen digitizer
                                                                  • nice if it had a haptic buzzer, perhaps

                                                                  This sounds like a job for one of the power-optimized ARM SOCs from RockChip or NXP. You can get all of that on a test board for cheap to ridiculously cheap, usually with 256MB - 1 GB RAM and some MMC storage, everything else on breakout pins. They all run Linux, though often with silly bootloaders. I can’t specify an exact one because everybody is out of everything right now…

                                                              1. 9

                                                                Is there some write up which spells out Alpine+Rust technical problem exactly? From this article, I infer the following:

                                                                • Alpine has two year support cycle, and they need to stick to the same version of the compiler throughout the cycle.
                                                                • Rust, however, releases every 6 weeks, and only officially supports the latest stable. Eg, a security issue found in the compiler will be backported to the current stable, but not to the stable from two years ago.

                                                                Would this be a fair summary of the problem?

                                                                1. 31

                                                                  It sounds like it. Note that Clang has the same problem: LLVM has a release every 6 months and upstream supports only the latest version. In FreeBSD, we maintain our own backports of critical issues and move the base system’s toolchain to newer versions forward in minor releases (a major release series has about a five-year support lifecycle).

                                                                  This is not ideal from a stability perspective because sometimes a newer clang can’t compile things an older clang could but this also needs to be balanced against the desire of people to actually compile stuff: If we didn’t ship to a base system compiler that supported C++17 in FreeBSD 12 (EOL 2024) then by the end of its support lifecycle the base system compiler would be a waste of space and everyone would use one from ports (most of the stuff that I work on has been C++17 for a while and is moving to C++20 at the moment).

                                                                  Note that the reason that we have clang in the base system at all is that POSIX requires a cc binary. Without that, we’d probably move to supporting clang only in ports (where we carry multiple versions and anyone who wants to install an old one can, putting up with the bugs in the old one instead of the bugs in the new one).

                                                                  I don’t have a huge amount of sympathy with a lot of these complaints because they often come from policies that the packaging system is enforcing. You want to maintain a complete set of packages for two years with security backports? That means that you are committing to back-porting security fixes for two years. If you’re RedHat then you might have customers who are willing to do that (so that they can then run Alpine / Debian / Ubuntu in containers because nothing in their host OS is sufficiently recent to be useable) but if you don’t then you need to ask yourself why you want to make that commitment. Can you relax it and say that you’ll bump the version of the Rust compiler?

                                                                  Packaging policies should first be driven by what is possible, then by what users want. Promising to do the impossible or promising to stick to some arbitrary standards that users don’t actually care about doesn’t help anyone.

                                                                  1. 17

                                                                    I don’t have a huge amount of sympathy with a lot of these complaints because they often come from policies that the packaging system is enforcing. You want to maintain a complete set of packages for two years with security backports? That means that you are committing to back-porting security fixes for two years.

                                                                    Exactly! As someone who’s been on the receiving end of, “Bbbbut but we promised client X we would provide these guarantees even though we have no way to enforce them and it’s not our lane! We (and by ‘we’ I mean you) have to find a way to provide said guarantees!”

                                                                    Note that in the OP, and unlike in my anecdote, it looks like Alpine is doing the right thing here by moving those packages they can’t support into community, so kudos on making the hard uncomfortable calls.

                                                                    1. 3

                                                                      I don’t have a huge amount of sympathy with a lot of these complaints because they often come from policies that the packaging system is enforcing. You want to maintain a complete set of packages for two years with security backports? That means that you are committing to back-porting security fixes for two years.

                                                                      Of course it’s a policy that someone made up. The point is that it’s a useful policy for users. I think software developers in general often have quite different interests from the people who run their software—they want to move fast and break things, while as a sysadmin I just don’t have the bandwidth to keep up with everything breaking all the time. Distros don’t backport just for the sake of it; they’re bridging a gap between what FOSS developers want to make and what users need.

                                                                      But also, in general, they do backport. I think it’s reasonable to complain when upstream projects make that intractably difficult, though, to the extent that it’s reasonable for anyone to complain about something that’s free. As a user who relies on stable distros, if they weren’t complaining for me, I’d be complaining myself.

                                                                      1. 5

                                                                        The point is that it’s a useful policy for users

                                                                        No, the point is that it may be useful in some cases for some users in some contexts and probably was useful for a large subset of visible users at the time that the policy was created.

                                                                        I think software developers in general often have quite different interests from the people who run their software—they want to move fast and break things, while as a sysadmin I just don’t have the bandwidth to keep up with everything breaking all the time

                                                                        As a developer, I want access to the latest versions of my tools and libraries. That’s easy.

                                                                        As a sysadmin, I don’t want security vulnerabilities in the things that I deploy. I also don’t want unexpected changes that break things for my users. In the LTS model, these two constraints very often come into conflict. Security vulnerabilities are easy to fix in the mainline branch but if you want back-ports then someone needs to do that work. If you want volunteers to do that, then you’re asking them to do work. If you’re RedHat (IBM) then you’ve got a load of customers who are paying you to pay engineers to do that. If you’re FreeBSD or Alpine? You have users demanding it but not being willing to do the work or pay for it, so you have to ask why you’re devoting effort to it (in the case of FreeBSD, most of the big companies that use it run -HEAD so don’t care about this at all).

                                                                        As a user who relies on stable distros, if they weren’t complaining for me, I’d be complaining myself.

                                                                        How much are you / your employer paying (and who are they paying) to ensure that you have support for a stable distro?

                                                                      2. 2

                                                                        the reason that we have clang in the base system at all is that POSIX requires a cc binary

                                                                        I thought it has more to do with the base/ports split and the “base builds base” tradition. It’s not impossible to just ship system images with a pkg-installed llvm :)

                                                                        (Also who cares about that aspect of POSIX. Not the Linux distros where you have to pacman -Sy gcc, haha)

                                                                    1. 2

                                                                      What is needed to achieve something like what node or go does? It has venvs for separate library environments, it has pipenv for project local environments, it has venv for project local environments with a specific version of python.

                                                                      What specific UX changes are needed to assemble these to something similar to what node or go does?

                                                                      What is done in these other communities to handle the other half of the problem which is the health of library dependencies? Automated testing of all permutations seems impractical for large packages.

                                                                      1. 3

                                                                        pipenv/poetry/pdm/etc do achieve npm/cargo/stack/etc style “project” UX. It just took so long… and all the layers are just hidden inside, it’s an abstraction waiting to leak.

                                                                      1. 7

                                                                        Similar criticisms apply to Telegram… funnily enough, down to a damn cryprocurrency integration attempt!

                                                                        To avoid [federation related metadata stuff in Matrix], you can either disable federation

                                                                        At that point you might as well use rocket chat or that new discord clone thing or whatever…

                                                                        1. 8

                                                                          Does anyone else have the problem that Matrix is incredibly slow? I have a top of the line desktop PC and it takes 20-60 seconds from double clicking the icon until the Element client finishes with all the spinners and freezing and becomes usable. Also, how can I extract conversations in a usable format (e.g. SQLite)? These are my two biggest pain points with Matrix and the reason why I don’t use it.

                                                                          1. 9

                                                                            You’re describing problems with Element, which is fair. It’s an electron app, and it’s memory hungry. So is Signal.

                                                                            The difference is that you can use Matrix without using Element; there are 3rd-party clients out there that work great with much better performance footprints. If you don’t want to use the Signal electron app, or if Signal decides your platform isn’t worth supporting (like arm64) then your only option is to not use Signal.

                                                                            1. 8

                                                                              If Element is showing spinners for tens of seconds, that’s all time spent waiting for the server to respond.

                                                                              Synapse is (at least was) slow. Back when I’ve tried running it, there even was a postgres query you had to run occasionally to clean up things that brought it to a complete halt. Thankfully it seems that Conduit is a server project that will actually be good.

                                                                              1. 2

                                                                                The server stack isn’t much better performance wise, sadly.

                                                                                1. 1

                                                                                  It’s an electron app, and it’s memory hungry. So is Signal.

                                                                                  Signal Desktop on my laptop has been running for a couple of days. It is using 23.1 MB of RAM (between 3 processes, two using about 1 MB) and is responsive. Restarting it takes a few seconds and it’s usable as soon as it presents the UI.

                                                                                  1. 3

                                                                                    That experience is completely different from mine, but arguing about it is academic since I don’t even have the option of running it on my machine any more even if I wanted to.

                                                                                2. 2

                                                                                  It seems fine as a Weechat plugin for me

                                                                                  1. 1

                                                                                    Which server do you use? I run my own with (usually nearly) the latest versions of Synapse, and I don’t see any such problem. Starting up the client on my iPhone just now took 4 seconds, on my Linux laptop it took 12 seconds (but normally it’s always running and available as a panel icon).

                                                                                    1. 3

                                                                                      Same here. A friend of mine chats with me over the Matrix “default” home server and I’ve seen his client freeze like that too, while for me it’s always been fast (I’m using a self-hosted home server). I think there need to be more alternative servers and awareness of how to use such alternatives.

                                                                                      1. 1

                                                                                        Yeah, I think Matrix.org is slow, because they haven’t been able to scale in proportion to their users. Synapse is overly resource-hungry, but it’s not actually slow unless it’s starved of resources. Small homeservers are pretty much always faster than Matrix.org.

                                                                                  1. 15

                                                                                    Packaging is usually people’s #1 problem with the Python ecosystem. I do empathise with people struggling but often when I read the specifics of problems people have they often seem to be doing what I would consider “ambitious” things, like mixing package managers (conda and pip in one example in TFA), trying to do data science on a musl libc ddocker container or trying to take their app to prod without packaging it up.

                                                                                    In Pythons case I think the packaging system is solid overall and does work, though there are beartraps that unfortunately they aren’t labelled well. That said I think poetry does provide the right UX “affordances” and I think it should be the default choice for many.

                                                                                    1. 14

                                                                                      My latest problem with Python packaging was with setting up JupyterHub at $JOB. Here is what happened:

                                                                                      • building a notebook docker image FROM jupyter/scipy-notebook
                                                                                      • RUN conda update
                                                                                      • environment completely broken
                                                                                      • change conda update to only update the specific package we need a bug fix for
                                                                                      • environment completely broken
                                                                                      • ADD fix_a_specific_bug_in_that_specific_package_myself.diff /tmp/patch.diff
                                                                                      • WORKDIR /opt/conda
                                                                                      • RUN patch -p3 < /tmp/patch.diff

                                                                                      I did nothing ambitious, unless updating packages is “ambitious” in Python. The entire packaging ecosystem seems like a huge dumpster fire. Why are there so many different package managers? I can only imagine because the authors of each package manager thought all the other ones suck too much to bother using, which seems completely true to me.

                                                                                      Ruby just has gem. Rust just has cargo. They both just work. Python has really dropped the ball here.

                                                                                      1. 6

                                                                                        Have you tried just using pip? pip install jupyterlab just works for me.

                                                                                        To be honest I see a lot of people complain about conda which makes me suspect it doesn’t work. Like I have gotten a lot of mileage out of pip and python -m venv (and even that one, if you’re in docker you don’t need to mess with virtual envs) (Though maybe you’re on windows?)

                                                                                        1. 2

                                                                                          The official Docker image we based on uses Conda, and mixing Conda and pip is asking for even more pain. And I was under the impression Conda has packages that pip doesn’t, but maybe that’s not true anymore.

                                                                                        2. 4

                                                                                          @ $WORK, For the past decade+ our policy was: whatever our OS packages for dependencies only. If we need more than what our OS(typically Ubuntu LTS) packages for python dependencies, then the only option is to import the package into the tree, as a direct dependency and we now OWN that code. This is the only sane way to do it, as the python packaging stuff after many decades is still mostly broken and not worth trying to fight with.

                                                                                          We’ve since started moving to playing with Nix and Docker containers, which mostly solve the problem a different way, but it’s still 90% saner than whatever python packaging people keep spouting. Note, the technical issues are basically 100% solved, it’s all community buy in and miserable work, nothing anyone wants to do, which is why we are stuck with Python packaging continually being a near complete disaster.

                                                                                          Maybe the PSF will decide it’s a real problem eventually, hire a few community organizers full time for a decade and a developer(maybe 2, but come on it’s not really a technical problem anymore) and solve it for real. I’m not holding my breath, nor am I volunteering for the job.

                                                                                          1. 2

                                                                                            My solution has been the opposite: do not rely on brew or apt for anything. Their release strategies (like “oh we’ll just upgrade python from 3.8 to 3.9 for you”) just don’t work, and cause pain. This has solved so much for me (and means I’m not breaking the system Python when doing “weird” stuff). Python.org has installers, after all!

                                                                                            Granted, I’m not on Windows but everything just works for me and I think it’s partly cuz package maintainers use the system in a similar way.

                                                                                            I kinda think Python needs a “don’t use anything from dist-packages and fully-qualify the Python version everywhere” mode, to prevent OS updates from breaking everything else.

                                                                                            1. 1

                                                                                              There is def. work involved to “port” our software to the next LTS branch, but it’s usually not miserable work. Plus we only have to do it every few years. The good thing is, by the time the python package makes it into a .deb in -stable branch the python code is also pretty stable, so upgrades and bugfixes are usually pretty painless.

                                                                                          2. 1

                                                                                            Yes I’m afraid I would consider blithely updating lots of software to the latest version very ambitious. Only a loon would update a number of different pieces of software at once with the serious expectation that everything would continue working, surely you can’t mean that?

                                                                                            I can’t speak to other package managers in general except to contradict you on rubygems. Last time I was running ruby in prod we had to increase the vm size to run rubygems in order to run puppet. Our Rubygem runs were unreliable (transient failures ala npm) and this was considered common. Perhaps this is an out of date view, it was some years ago.

                                                                                            1. 15

                                                                                              That’s insane. apt-get upgrade works fine. gem update works fine. cargo update works fine. Each updates the packages while maintaining dependency version invariants declared by those packages. Conda seemed to be trying to do that, but couldn’t do it correctly, and then trashed the system. This is not a failure mode I consider acceptable.

                                                                                              Sure, sometimes upgrading to newer software causes issues with backwards compatibility, hence stable and LTS release channels. But that’s not what happened here. After conda update completed, it was impossible to do anything else with conda at all. It was utterly broken, incapable of proceeding in any direction, no conda commands could successfully change anything. And the packages were all in completely inconsistent states. Stuff didn’t stop working because the newer packages had bugs or incompatibilities, they stopped working because package A required B version 3+, but version 2 was still installed because conda self-destructed before it finished installing B version 3. Has that ever happened to you with gem?

                                                                                              If updating my packages in the Python ecosystem makes me a loon, what is normal? Does anyone care about security patches? Bug fixes?

                                                                                              I can’t comment on gem‘s memory usage, I’ve never run it on such a small machine that I’ve had that problem. But I haven’t had transient failures for any reason other than networking issues. I have used gem to update all my --user installed gems many times. Conda literally self-destructed the first time I tried. Literally no issue with gem comes remotely close to the absolute absurdity of Python packaging.

                                                                                              1. 4

                                                                                                apt-get upgrade does work pretty reliably (but not always, as you imply) because it is a closed universe as described in TFA in which the totality of packages are tested together by a single party - more or less anyway. Cargo I have not used seriously.

                                                                                                Upgrading packages does not make you a loon, but blindly updating everything and expecting it to work for you arguably does - at least not without the expectation of debugging stuff, though I wouldn’t expect to debug the package manager. That sounds like you also ran into conda bug on your first (and apparently sole?) experience with python packaging. I can’t help you there except to say that is not the indicative experience you are extrapolating it to be.

                                                                                                I don’t want to get deep into rubygems tit for tat except to repeat that it has been a huge problem for me professionally and yes including broken environments. Rubygems/bundler performance and reliably were one of the reasons that team abandoned Puppet. Ansible used only system python on the target machine, a design I’m sure was motivated by problems having to bootstrap ruby and bundler for puppet and chef. I’m sure there were other way to surmount that but this was a JVM team and eyes were rolling each time DevOps was blocked on stuff arising from Ruby packaging.

                                                                                                1. 2

                                                                                                  your first (and apparently sole?) experience with python packaging

                                                                                                  Latest experience. Previous experiences have not been so egregiously bad, but I’ve always found the tooling lacking.

                                                                                                  having to bootstrap ruby and bundler for puppet

                                                                                                  That’s the issue. As a JVM team, would you run mvn install on production machines, or copy the jars? PuppetLabs provides deb and rpm packages for a reason.

                                                                                                  Regardless, even though gem has problems, I can still run gem update on an environment and resolve any problems. What about Python? Apparently that’s so inadvisable with Conda that I was a loony for trying. It’s certainly not better with pip, which doesn’t even try to provide an update subcommand. There’s pip install -U, which doesn’t seem to provide any way to ensure you actually end up with a coherent version set after upgrading. Conda, though it blew up spectacularly, at least tried.

                                                                                                  Seriously, how are you supposed to keep your app dependencies updated in Python?

                                                                                                  1. 2

                                                                                                    how are you supposed to keep your app dependencies updated in Python?

                                                                                                    In the old days, there were scripts that told you what’s outdated and even generated new requirements.txt. Eventually pip gained the ability to list outdated deps by itself: https://superuser.com/questions/259474/find-outdated-updatable-pip-packages#588422

                                                                                                    State of the art — pyproject.toml projects — your manager (e.g. pdm or poetry or whatever else because There’s One Way To Do It™) just provides all the things you want.

                                                                                              2. 8

                                                                                                Yes I’m afraid I would consider blithely updating lots of software to the latest version very ambitious. Only a loon would update a number of different pieces of software at once with the serious expectation that everything would continue working, surely you can’t mean that?

                                                                                                You have been Stockholm syndromed. Yes, in most languages, you just blithely do upgrades and only run into problems when there are major version bumps in the frameworks you are using and you want to move up.

                                                                                            2. 5

                                                                                              I’ve been in the Python ecosystem now for about six months after not touching it for 15+ years. Poetry is a great tool but has its warts. I worked around a bug today where two sources with similar URLs can cause one to get dropped when Poetry exports its data to a requirements.txt, something we need to do to install dependencies inside of the Docker container in which we’re running our app.

                                                                                              As Poetry matures, it’ll be great.

                                                                                              1. 2

                                                                                                Poetry is already three years old.

                                                                                                1. 2

                                                                                                  A young pup!

                                                                                              2. 2

                                                                                                I agree with your post, but thisnis a stretch:

                                                                                                In Pythons case I think the packaging system is solid overall and does work

                                                                                                You define you dependencies but they will update their dependencies down the tree, to versions incompatible to each other or with your code. This problem is left unsolved by the official packaging systems.

                                                                                                But in all frankness… This of pulling dependencies for something that takes otherwise 5 minutes to implement, and being ok with having dozens of moving targets as dependencies, is something that just can’t be solved. I rather prefer to avoid it. Use fewer and well defined dependencies.

                                                                                                1. 7

                                                                                                  If you ship an app, this can be solved by “pip freeze” which saves the deep dependencies as well. If you ship a library, limit the dependency versions.

                                                                                                  This is not really a python problem - you have to do the same thing in all languages. In the similar group of languages Ruby and JS will suffer the same issue.

                                                                                                  1. 2

                                                                                                    Pip freeze helps, but you’re still screwed if you rely on some C wheel (you probably do) or Python breaks something between versions (3.7 making async a keyword in particular was brutal for the breakage it caused).

                                                                                                    1. 1

                                                                                                      I’m not sure what you mean - wheels are precompiled, so get frozen like everything else. What do you think breaks in that situation?

                                                                                                      1. 2

                                                                                                        If I knew, it wouldn’t be broken, would it? All I can tell you is that wheels routinely work on one machine but not another or stop working when something on a machine gets upgraded, whether deliberately or accidentally. Why? I have no idea.

                                                                                              1. 31

                                                                                                Python is as messy, if not messier, than Node

                                                                                                I’d say Node is, like, not messy at all. Due to Node being younger, its ecosystem developed and matured in an age where per-project isolated local dependencies were becoming the norm.

                                                                                                Python is arguably uniquely messy due to the invention of virtualenv. I’m not aware of any other language community where the default way to make isolated projects for ages was a thing that made a whole local prefix with symlinks to the interpreter and whatnot and a script you’d have to run to bring it into $PATH. Python’s closest cousin (in terms of 2000s web dev evolution anyway), Ruby, did it all correctly relatively early on — the Bundler experience is just like npm/cargo/stack/etc.

                                                                                                But forget virtualenv, the real messiness of Python packaging is the freaking layering. Distutils, setuptools, pip, virtualenv (or the new project-based managers) — it’s all there and which part is handled by which layer gets confusing quickly.

                                                                                                1. 10

                                                                                                  The thing that worries me is that I don’t think the core Python devs get it: the packaging situation is so bad that it very well may kill the language. It should be all hands on deck, this is the most important thing for us to fix. But I don’t see that happening at all, and instead there’s fiddling with esoterica like walrus and match…

                                                                                                  1. 13

                                                                                                    So one interesting twist here is that many core devs work for big companies that use their own build and deployment systems for Python. (e.g. I worked with probably a dozen core Python devs at Google many years ago.) So they may not feel the issue on a daily basis. I certainly feel it more now that I do more open source, although I was aware of it back then whenever I had to install NumPy, etc.

                                                                                                    From what I hear Jane St. is in a similar situation with OCaml and OPAM. They sponsor the open source package manager, but they don’t actually use it themselves! Because they use a monorepo like Google. monorepo means “no version constraint solving”, which simplifies the problem drastically (it’s an NP complete problem).

                                                                                                    I also think the problem is more complicated than “core devs don’t get it”. It’s more like the solutions being very constrained by what happened in the past. For a long time the import system / algorithm itself was a very big wart and there was a huge effort to clean it up.

                                                                                                    I will say that I think most of these problems were known before Python 3, and I wish there was a hard break then, but that was already 12-15 years ago at this point. And then people would have probably liked the 2->3 transition even less, etc.

                                                                                                    1. 5

                                                                                                      So one interesting twist here is that many core devs work for big companies that use their own build and deployment systems for Python. (e.g. I worked with probably a dozen core Python devs at Google many years ago.) So they may not feel the issue on a daily basis. I certainly feel it more now that I do more open source, although I was aware of it back then whenever I had to install NumPy, etc.

                                                                                                      This is definitely the case with my cloudy overlords, although I will say that this may be changing. I think some people are recognizing that there is wisdom in not rolling their own and allowing devs to leverage familiar interfaces for packaging.

                                                                                                    2. 4

                                                                                                      For what it’s worth, this is the #1 reason I keep not trying Python. It’s just a huge headache and I can’t care enough about it.

                                                                                                      1. 4

                                                                                                        I’ve been using Python professionally since 2010, I used to absolutely love it, and I’ve finally just reached the point where I no longer consider it an acceptable choice for a greenfield project of any scale, no exceptions. Someone will always come around to suggest that it’s open source, and you should fix what you don’t like, but it’s been my experience that the community culture has settled into a state that no project or ecosystem ever recovers from, which is when issues start being responded to with justifications along the lines of “this can’t change because of fundamental technical deficiency X, which because of its magnitude we’ve decided to stop treating as a technical deficiency in favor of treating it as a fundamental invariant of the universe, in spite of ample evidence that there are better ways to do it.” Either that technical deficiency is truly unsurmountable, in which case the tool is de jure broken and I have no good reason to use it, or that technical deficiency is surmountable but there will never be any will to fix it, in which case the tool is de facto broken and I have no good reason to use it. I feel deep sadness about this, but at this point there is too little that is exceptional about Python to justify putting effort into fixing what’s broken. No tool lasts forever, maybe we should just accept that it’s time to sunset this one.

                                                                                                        1. 0

                                                                                                          Thats a great point, and a good answer to “Why don’t you just fix it yourself” that is thrown anytime you complain about any open source project, especially established ones like Python.

                                                                                                          Python especially has this cultish mentality “All if perfect how dare you suggest otherwise”

                                                                                                      2. 3

                                                                                                        The thing that worries me is that I don’t think the core Python devs get it: the packaging situation is so bad that it very well may kill the language. It should be all hands on deck, this is the most important thing for us to fix. But I don’t see that happening at all, and instead there’s fiddling with esoterica like walrus and match…

                                                                                                        Python’s governance is pretty transparent. Do you have concrete suggestions for improvement? If you do, consider coming up with even a proof of concept implementation and creating a PEP.

                                                                                                        Be the change you want to see in the world :)

                                                                                                      3. 8

                                                                                                        the real messiness of Python packaging is the freaking layering. Distutils, setuptools, pip, virtualenv (or the new project-based managers) — it’s all there

                                                                                                        Yup +100 to this … This is why I sometimes download tarballs with shell scripts and use “python setup.py build” instead. That’s only one layer :)

                                                                                                        That approach doesn’t work all the time, e.g. if you have a big NumPy stack, or a big web framework with transitive dependencies.

                                                                                                        On the other hand, if it works, then you know exactly what your dependencies are, and you can archive the tarballs somewhere for reproducible builds, etc.

                                                                                                        1. 3

                                                                                                          the Bundler experience is just like npm/cargo/stack/etc

                                                                                                          Thatk’s mostly splitting hairs, but for me there’s a big difference with cargo when it comes to experience: cargo run / cargo build just work, while bundle exec, npm run require running install command manually.

                                                                                                          1. 1

                                                                                                            There’s a lot that node gets right simply by virtue of being a younger language and community than Python or Ruby by a long shot, so it could benefit from observing the effect of critical decisions over time.

                                                                                                            Unfortunately that lack of maturity can express itself in various ways, some merely cosmetic, some less so.

                                                                                                            Overall even as someone like myself who doesn’t love the language, having now learned it I can appreciate that there’s a lot of interesting work going into that community that bears watching and learning from, whatever your first choice in programming languages may be.

                                                                                                          1. 6

                                                                                                            Damn. That “reveal” slider thing is just amazing. I mean, it kinda shouldn’t feel that surprising, there was the 3D view in Firefox devtools some years ago and whatnot, but something about that particular simple demo just felt mind-blowing even though I’m well aware of 3D transforms already.

                                                                                                            unlike box-shadow, the filter property is hardware-accelerated in Chrome, and possibly other browsers

                                                                                                            WebRender actually got GPU-rendered box-shadow first, before the shadow filter :)

                                                                                                            1. 3

                                                                                                              Fun fact, I just clicked through the iPad comparator page and found out that the iPads released today in their cellular editions have dropped support for GSM/EDGE (2G). Good riddance I guess, though words like “GPRS” fill me with nostalgia…

                                                                                                              1. 2

                                                                                                                That’s interesting. There are still places, especially when I’m on the road, where my phone loses 4g for a while and I’m on EDGE. I always figured, since we’ve already phased out 3G, that EDGE would remain the back-up for a while for areas without coverage from the higher-frequency cell networks, but apparently not.

                                                                                                                1. 3

                                                                                                                  I think the expectation is that those frequency bands will be reallocated to 4G over time. You can have large cells with 4G - just at lower data rates (still faster than EDGE though!).

                                                                                                                  1. 1

                                                                                                                    Hmm, I see. Still sounds like there’s going to be a lot of years in between though, where lots of locations which currently have coverage won’t have any.

                                                                                                              1. 5

                                                                                                                New phone with slightly faster CPU (possibly; no numbers) and possibly better camera system (possibly; no numbers). Again. These iPhone news have been kind of boring for a while haven’t they.

                                                                                                                1. 3

                                                                                                                  The iPad mini update is huge though. I’ve been kinda wanting one, but the dated design, old processor, and worst of all the Lightning port (UGHHHH) were putting me off. Now all those complaints are gone!

                                                                                                                  Now I can complain about… 60Hz, I guess. Why is “ProMotion” reserved to “Pro” models? (At least it’s now finally also on the Pro phone, not just the Pro tablet!) High refresh rate is super common by now in the Android world, affordable mid range devices like OnePlus Nord come with 90Hz…

                                                                                                                  1. 1

                                                                                                                    Yeah, the Mini upgrade is huge. I’ve always been a huge fan of it. I currently have a Pro (bought in the period when nothing really happened to the Mini and it seemed dead), but if I were getting an iPad today, I’d very seriously consider the new Mini.

                                                                                                                    1. 1

                                                                                                                      I just bought an iPad mini last year, but the screen got cracked and hey, my kids need their own electronic babysitter, don’t they?…

                                                                                                                  1. 19

                                                                                                                    could use a collision-resistant hash, like perhaps SHA256

                                                                                                                    No, what, why are you going straight to the full-on expensive cryptographic hash functions — there’s things like SipHash designed specifically for this.

                                                                                                                    1. 4

                                                                                                                      I do agree that the backgrounded buttons of current Adwaita can feel too heavy. But I’ll miss them nonetheless. They remind me of the Mac OS X Leopard - Lion era so much. The nostalgic value is real :)

                                                                                                                      1. 35

                                                                                                                        Unlike say, VMs, containers have a minimal performance hit and overhead

                                                                                                                        Ugh. I really hate it when people say things like that because it’s both wrong and a domain error:

                                                                                                                        A container is a packaging format, a VM is an isolation mechanism. Containers can be deployed on VMs or they can be deployed with shared-kernel isolation mechanisms (such as FreeBSD Jails, Solaris Zones, Linux cgroups, namespaces seccomp-bpf and wishful thinking), , or with hybrids such as gVisor.

                                                                                                                        Whether a VM or a shared-kernel system has more performance overhead is debatable. For example, FreeBSD Jails now support having per-jail copies of the entire network stack because using RSS in the hardware to route packets to a completely independent instance of the network stack gives better performance and scalability than sharing state owned by different jails in the same kernel data structures. Modern container-focused VM systems do aggressive page sharing and so have very little memory overhead and even without that the kernel is pretty tiny in comparison to the rest of a typical container-deployed software stack.

                                                                                                                        Running everything as root. We never let your code run as root before, why is it now suddenly a good idea?

                                                                                                                        This depends entirely on your threat model. We don’t run things as root because we have multiple security contexts and we want to respect the principle of least privilege. With containerised deployments, each container is a separate security context and already runs with lower privileges than the rest of the system. If your isolation mechanism works properly then the only reason to run as a non-root user in a container is if you’re running different programs in different security contexts within the container. If everything in your container is allowed to modify all state owned by the container then there’s no reason to not run it all as root. If you do have multiple security contexts inside a container then you need to think about why they’re not separate containers because now you’re in a world where you’re managing two different mechanisms for isolating different security contexts.

                                                                                                                        1. 22

                                                                                                                          I think you mean an image is a packaging format, whereas a container is an instance of a jail made up of various shared-kernel isolation mechanisims (including the wishful thinking) as you mentioned.

                                                                                                                          Yes, the terminology is unfortunate. My reimplementation of Docker calls it an “instance” rather than “container”.

                                                                                                                          1. 3

                                                                                                                            yeah, the “’never run as root in your container” thing kills me

                                                                                                                            1. 9

                                                                                                                              IIUC that’s all because the way Linux isolates users (with the whole UID remapping into a flat range thing) is weird and there’s way too many security bugs related to that.

                                                                                                                              1. 1

                                                                                                                                I don’t know if this is still true, but part of where this advice comes from is that it used to be that running as root meant running as root on the host (i.e. the mechanism you’re talking about was not used by Docker). In theory this was “fine” because you could only get at stuff on the container environment, but it meant that if there was a container breakout exploit you were unconfined root on the host. So running as non-root in the container meant that you’d have to pair a container breakout with a privilege escalation bug to get that kind of access.

                                                                                                                                In other words: the isolation mechanism did not work properly.

                                                                                                                            2. 1

                                                                                                                              That’s interesting. I haven’t actually bench tested the two in years. I’ll have to revisit it.

                                                                                                                              1. 1

                                                                                                                                You might want to have a look at NVIDIA’s enroot or Singularity for some lower-overhead alternatives. I’ve briefly looked at enroot after I saw the talk about Distributed HPC Applications with Unprivileged Containers at FOSDEM 2020, but sadly haven’t gotten a chance to use them at work yet.

                                                                                                                                1. 2

                                                                                                                                  Have you tried https://github.com/weaveworks/ignite to just run a docker image in a VM instead of a container?

                                                                                                                                  1. 1

                                                                                                                                    No, haven’t stumbled across that before. Thanks, that looks very interesting!

                                                                                                                                    1. 1

                                                                                                                                      That seems interesting. I wonder what benefit it provides compared to the shared-kernel isolation mechanism used by docker run <container>. Do I get stronger isolation, performance boost, or something else?

                                                                                                                                      1. 2

                                                                                                                                        I think there are always tradeoffs, but a VM may be easier to reason about than a container still. It’s a level of abstraction that you can apply thinking about a single computer to.

                                                                                                                                        I do think that you get stronger isolation guarantees too. You can also more easily upgrade things, so if you have a kernel vulnerability that affects one of the containers, you can reload just that one. There are many issues that affect hypervisors only or guests only.

                                                                                                                                        At launch we used per-customer EC2 instances to provide strong security and isolation between customers. As Lambda grew, we saw the need for technology to provide a highly secure, flexible, and efficient runtime environment for services like Lambda and Fargate. Using our experience building isolated EC2 instances with hardware virtualization technology, we started an effort to build a VMM that was tailored to run serverless functions and integrate with container ecosystems.

                                                                                                                                        It also seems like a compromise between the user interface for a developer and an operations deep expertise. If you have invested 15 years in virtualization expertise, maybe you stick with that with ops and present a container user interface to devs?

                                                                                                                                        For me, one of the big things about containers was not requiring special hardware to virtualize at full speed and automatic memory allocation. You’re never stuck with an 8GB VM you have to shut down to prevent your web browser from being swapped out when you’re trying to open stack overflow. You know 8gb was suggested, but you also see that only 512MB is actually being used.

                                                                                                                                        Most hardware these days has hardware acceleration for virtualization and firecracker supports the virtio memory ballooning driver as of Dec 2020, so many of the reasons I would have used containers in 2013 are moot.

                                                                                                                                        As an ops person myself, I find containers to often have an impedance mismatch with software defaults. Why show a container that is limited to two cores that it has 64 cores? Haproxy will deadlock itself waiting for all 64 connection threads to get scheduled on those two cores. You look in there and you’re like ‘oh, how do I hardcode the number of threads in haproxy now to two…’. It’s trivial with haproxy, but it’s not default. How many other things do you know of that use nproc+1 and will get tripped up in a container? How many different ways do you have to configure this for different runtimes and languages?

                                                                                                                                2. 1

                                                                                                                                  Containers can be deployed on VMs

                                                                                                                                  OT because I agree with everything you said, but I have yet to find a satisfying non-enterprise (i.e. requiring a million other network services and solutions).

                                                                                                                                  Once upon a time, I was sure VMware was going to add “deploy container as VM instance” to ESXi but then they instead released Photon and made it clear containers would never be first-class residents on ESXi but would rather require a (non-invisible) host VM in a one-to-many mapping.

                                                                                                                                  1. 2
                                                                                                                                    1. 1

                                                                                                                                      We use this at my work (Sourcegraph) for running semi-arbitrary code (language indexers, etc.), it works really well.

                                                                                                                                1. 2

                                                                                                                                  The problem is an unfortunate incompatibility of licenses: perf is GPLv2, not GPLv2+, and libbfd is GPLv3+

                                                                                                                                  That’s really stupid. Wait, why do people think that license compatibility matters for dynamic linking again? Isn’t it completely nuts to think that linking things dynamically creates a derivative work? Who does the “derivation” exactly? rtld in memory? By that line of thinking, running a GPLed program on Windows would imply that Windows needs to comply with the GPL.

                                                                                                                                  (Also, regardless of that, why don’t the maintainers of these projects see this and think, “I’ll add the + or at least a special exception to fix this”?)

                                                                                                                                  1. 4

                                                                                                                                    The real question is why isn’t libbfd LGPL?

                                                                                                                                    1. 2

                                                                                                                                      libbfd isn’t intended for use by third-parties; it’s a holding place for code shared between binutils, gdb, and gcc.

                                                                                                                                    2. 2

                                                                                                                                      It’s only a problem for distros that only let privileged developers submit binary packages, because of redistribution problems. I’m currently on NixOS and my version of perf is linked with libbfd.

                                                                                                                                      1. 1

                                                                                                                                        Could you elaborate on this? How does a privileged group of developers relate to distribution problems?

                                                                                                                                        1. 2

                                                                                                                                          If a distro only uses ports trees, then distributing binaries is an optimization rather than an essential part of software delivery. Ports trees generally work around composition issues because end users are allowed to run nearly any combination of software on their personal computers.

                                                                                                                                          1. 1

                                                                                                                                            I see, that definitely makes sense when you explain it that way.

                                                                                                                                            Yet it still feels weird to me, as I can easily download the deb src package to build and install manually. Even though it’a different from a ports tree in structure (deb src packages are tools for making the binary distribution, ports packages are optimizations of a source distribution), from an end user perspective it’s just different ergonomics.

                                                                                                                                            1. 2

                                                                                                                                              The key difference is probably how options are specified. As I understand it, a deb src package contains a recipe for building a .deb binary package. In contrast, a port typically contains a recipe for building a set of binary packages. Some of these may permit binary redistribution, others don’t. When the FreeBSD package build infrastructure runs, it will not build configurations that don’t allow binary distribution, but if you run poudriere locally then you can enable these options.

                                                                                                                                              Of course, perf could just link to LLVM’s Symbolizer library and get this functionality without any problems, but that’s probably a hard sell at Red Hat, which employs most of the GNU toolchain developers.