Threads for jclulow

  1. 3

    Kitty’s shell integration allows you do many cool things like opening the output of the last command in a pager. I think it should be possible to use that mechanism to get your last output in a shell variable.

    https://sw.kovidgoyal.net/kitty/shell-integration/

    1. 2

      I’m curious about how that works. If you’re able to get it into the shell, is that driven by the shell emitting escape codes that are interpreted by kitty to cause it to send the last output as input to the terminal? It feels like that could be a serious security issue if you can cause someone to, say, accidentally emit such an escape sequence from untrusted user input.

      1. 2

        The protocol is described at the bottom of this page: https://sw.kovidgoyal.net/kitty/shell-integration/

        1. 1

          The part of the protocol described there seems to be how the shell should delimit sections of output, which makes sense. I’m more curious about how you would retrieve those sections, to get the contents of one of them (such as the output of the last command) in a pager or a shell variable from wherever it is stored inside kitty.

          1. 2

            Gotcha. For common cases (like if you wanted a way to easily copy the last command output), you would do something like this: https://github.com/kovidgoyal/kitty/discussions/4477#discussioncomment-2077030

            If you wanted to do something more complex, you can use kitty’s scripting interface: https://sw.kovidgoyal.net/kitty/kittens/custom/#passing-the-contents-of-the-screen-to-the-kitten

    1. 1

      There’s no Upstairs daemon or anything like that, it all stays in-process.

      How does this work? If the replication and everything is managed by the application, what happens if multiple application instances write to the same data concurrently? How do they coordinate?

      1. 4

        Considering that the use-case for Crucible in oxide is VM backing storage, which would normally have only one writing instance, that isn’t considered a problem that needs to be solved by them.

        1. 9

          I have been pretty involved in the design of Crucible, and you’re totally correct. We’ve been able to avoid a whole lot of distributed systems issues by having just one client, the hypervisor to which the virtual disk is attached. We’ve been able to focus mostly on the crash safety properties of the storage format without needing a complex consensus or coordination strategy.

      1. 20

        I usually prefer to use \r (carriage return), without a newline: this preserves any text before. You’ll need to explicitly flush the output, though, since it’s usually line buffered:

        print(f"Processing {line}...", end="\r", flush=True)

        https://asciinema.org/a/CMgdWJCK5gT0DGfRrWd2z04pP

        1. 4

          oh that’s actually a really nice way of doing this!

          Of course for counting that works as-is, but I think with this you gotta actually print over all the output since you’re not clearing the results.

          In [3]: print("Processing 10000...\rProcessing 1..")
          Processing 1..00...
          
          1. 8

            Yes. Tracking the line length sounds good, but sometimes people type stuff on accident into the terminal, so unless you’re also disabling echo (or processing it) you might prefer end="\x1b[K\r"

            In general, before doing anything like this, I check if stdout is a tty (i.e. sys.stdout.isatty() in python) and then use fun graphics if they are, but just regular newlines if they’re not. This way the user can save your log output in a file “just” by piping it or redirecting the output and do other things with the log data instead of just watching it fly by.

          2. 3

            I think it’s slightly preferable to put the CR on the front so that the text always starts in the right place, even if something else has interfered with the terminal in the meantime.

          1. 9

            I never really got into dotfiles. My problem is:

            • If I’m working on a remote system, it’s likely not mine, and I’ll only be there for a short time, or worse, on someone else’s account over a screenshare, so investing in knowing the defaults and quickly changing the intolerable is more convenient than trying to pull dotfiles on a client’s system.
            • If it’s my own system, I rarely configure a primary system (it’s usually like 5-7 years), and by the time I do get around to it, there’s a good chance I’ve switched platform, my own needs and tastes have changed, and I want to experiment with something new. Each new system is an opportunity to approach things with a clean slate.
            1. 5

              Pulling dotfiles on to a machine can be simple. Installing chezmoi is a one line command that only requires a shell and curl/wget, nothing else. You don’t need to install a scripting language runtime, or even git (chezmoi has a builtin git client using go-git if no git binary is present on the machine).

              Even if you rarely configure a primary system, I’m sure that you use a version control system for your code. Using any dotfile manager (there are many to chose from) brings the benefit of version control to your dotfiles, including easy rollback/undo (great for experimenting) and backups.

              1. 2

                That would be nice if the platform I work on supported Go :) - in all seriousness, while git is likely installed, the fact of the matter is it’d be ridiculous to install my configuration environment onto a client’s machine, let alone account on their own system.

                I don’t really change my configuration or deviate much from defaults either. It’s massive overhead for like what, 5 lines of .vimrc? Not to mention for i.e. GUI stuff, the config files in the XDG dir are possibly fragile across versions and not worth including in a scheme.

                1. 1

                  if the platform I work on supported Go

                  What platform is it?

              2. 2

                I’m in a pretty similar boat. I could use VS Code with a vi keybindings plugin, which is pretty great, or I could get better at vim/neovim, which is installed on literally every machine I have to work with, which often have to get worked on remotely or ad-hoc or through weird VPN’s. And I’m now even the person who writes the default .vimrc file, as long as other people are happy with it.

                The fish shell is worth it though. I wish I could install fish as default on all our work machines, but our infrastructure involves too many terrible bash scripts and people keep adding more. (Don’t say “use fish for interactive use and bash for scripting”, there’s (bad) reasons we can’t do that.)

                1. 2

                  Well, at least you can use VSC on the local system and usually use it as the frontend for a remote system, almost like what the TRAMP people on Emacs do. (I’ve never had good luck with TRAMP, but the VSC remote editor works pretty well the few times I’ve tried it…. except none of my remote servers are Linux 🙄)

                  1. 1

                    Yeah I’ve used VSC a few times. It’s pretty great, but again, I’m often doing this on a generic service-terminal-we-provide-with-our-hardware Linux laptop or something of that nature. I haven’t yet had to hack a client’s workstation or a hotel courtesy PC to fix something, but I could see it happening.

                    1. 1

                      Curious what issues you’ve had with tramp

                      1. 1

                        Two issues from my list:

                        • one of my customers has configured the prompt of all his servers with colors in the bashrc. Tramp checks for a specific regex (I believe) to figure out it had a successful login. I can’t find how to connect without editing those. Took me a long time to understand what was going on
                        • I am too thick to understand how to use sudo
                        1. 1

                          I can’t help with the first.

                          For me, sudo is :sudo-edit

                          1. 1

                            Ahhh interesting.. thanks. I’ll be experimenting immediately if that cover also org remote execution. Thanks

                1. 3

                  Intel Processor Trace is a way to do this for arbitrary software and no compiler tricks, with magic trace providing a nice frontend.

                  1. 1

                    dtrace?

                    1. 1

                      Yes, DTrace has the pid provider for tracing entry and exit of things that look, in the ELF binary, like C functions, or arbitrary instructions within those functions. It also has the profile provider that you can use to do stack sampling, which obviously has a lower overhead than tracing with pid.

                      The older truss(1) tool also provides for system call tracing and even function boundary tracing a bit like the article, which I believe works by adjusting the way the runtime link editor does dynamic linking for libraries.

                      1. 1

                        Right, the advantage with Intel PT is that there is no need to overwrite instructions with breakpoints. In comparison with the post above, uprobes/dtrace also needs to execute the overwritten instructions since there won’t be NOPs there in general.

                    1. 25

                      Yeah yeah, mention Rust. Rust is too complicated to implement by one person.

                      I’m not sure that’s a practical metric by which to judge a tool. The C compilers that provide a practical foundation for modern software development were not implemented by one person either.

                      In general Turing completeness is necessary but not sufficient: it’s just one facet of what makes a language practically useful. There are many other properties that end up resulting in costs someone has to pay to use a language; e.g., is it memory safe, or will engineers and users alike be on the hook for an unknown number of egregious memory safety bugs?

                      1. 12

                        Also mrustc has been implemented mostly by one person.

                        1. 2

                          I knew this would be brought up; you know the effort they’ve had to do to achieve this? An incredible amount.

                          1. 8

                            It’s 100K lines of code, and majority of it was developed over a 2-3 year period (with ongoing development to catch up with evolution of Rust). The number of commits and lines of code happens to be close to TCC:

                            It does take a couple of shortcuts: it’s a Rust-to-C compiler (no machine code generation) and it doesn’t perform borrow checking (the Rust language is carefully designed to make it optional. Lifetimes are purely a compile-time lint, and don’t affect generated code or its behavior).

                            I think overall in terms of implementation difficulty Rust is somewhere between C and C++. Parsing of Rust is much simpler than C++, and Rust has fewer crufty language features than C++ (there’s one way to initialize a variable), but some features are big-ish (borrow checker, type inference).

                            How hard it is to implement mainly depends on how good quality of implementation you want to have. For example, LLVM is 85× larger than mrustc and tcc, with over 130× more commits. It’s a 20-year collaborative effort, likely not possible to do by a single person. The main rustc project is also maximalist like that, because it isn’t merely an effort to get Rust working, but to make it fast, efficient, user-friendly, well-documented, reliable, portable, etc., so much much more work went into it beyond just the language implementation.

                            1. 2

                              I cannot speak for mrustc, but 100k loc for tcc is bullshit. Just counting sources and headers in the top level, I get 55k loc (the remainder is taken up by tests and win32 headers). Close to 40k is taken up by target-specific code. The core compiler is about 10k loc.

                              1. 1

                                openhub stats I’ve quoted are for the whole repo, and I see 57K .c and 38K .h in there. This includes tests, so it’s indeed more than just the compiler.

                                1. 2

                                  If I run a word count on everything in the ‘src’ directory of mrustc, I get about 130k loc. I therefore conclude that mrustc’s rust compiler is approximately 10x larger and more complex than tcc’s c compiler. Recall that tcc also includes assemblers and linkers, and supports many targets.

                              2. 0

                                I mean if 3 years is not a lot of effort then cheers to you! You must be an absolute coding beast.

                                1. 15

                                  I feel like this is a fairly disingenuous and dismissive argument - your original post stated that “Rust is too complicated to implement by one person.” The comment you were responding to was making the point that not only is there an implementation of Rust by primarily one person, but a single-contributor C implementation is a comparable size and would theoretically take a similar amount of effort to implement. People here aren’t trying say it’s not a lot of effort, but that it does exist and you may be trivializing the amount of effort needed for a C implementation.

                                  1. 3

                                    Sorry, I didn’t mean to dismiss anything! Isn’t the statement still true if it’s been mentioned they still got help?… Regardless the general sentiment is right. I should have said instead that it’s not reasonable!

                                    I may very well be trivializing the effort for a C implementation. In my mind C’s type system, lack of borrow checker, and other features make its implementation maybe a magnitude easier. I could be completely wrong though and please elaborate if that’s the case!

                                    1. 4

                                      A non-optimizing C89 or C90 compiler is relatively simple to implement, with only minor inconveniences from the messy preprocessor, bitfields, parsing ambiguities of dangling else and typedef (did you know it can be scoped and nested and this affects syntax around it!?). The aren’t any things that are hard per-se, mostly just tedious and laborious, because there’s a lot of small quirks underneath the surface (e.g. arrays don’t always decay to pointers, sizeof evaluates things differently, there are rules around “sequence points”).

                                      There are corners of C that most users don’t use, but compiler in theory needs to support, e.g. case doesn’t have to be at the top level of switch, but can be nested inside other arbitrary code. C can generate “irreducible” control flow, which is hard to reason about and hard to optimize. In fact, a lot of optimization is pretty hard due to aliasing, broken const, and the labyrinth of what is and isn’t UB described in the spec.

                                      1. 3

                                        There are corners of C that most users don’t use, but compiler in theory needs to support, e.g. case doesn’t have to be at the top level of switch, but can be nested inside other arbitrary code

                                        It’s worth noting that, since you said ‘non-optimising’ these things are generally very easy in a non-optimising compiler. You can compile C more or less one statement at a time, including case statements, as long as you are able to insert labels after you insert a jump to them (which you can with most assembly languages). Similarly, sequence points matter only if you’re doing more than just evaluating expressions as you parse them.

                                        The original C compiler ran on a computer that didn’t have enough memory for a full parsed AST and so the language had to support incremental code generation from a single-pass compiler.

                          2. 9

                            LLVM was originally just Chris Latner. I think the question isn’t “Can one person build it?” It’s “Can one person build it to the point where it has enough value for other people to work on it too?”

                            1. 5

                              LLVM was originally just Chris Latner

                              Several of the folks in / formerly in Vikram Adve’s group at UIUC would be quite surprised to learn that.

                              1. 1

                                I actually looked at Wikipedia first before my comment, but that made it seems like it was Latner’s project under Adve’s mentorship. I’ll take your word for it that it was a group effort from the start.

                            2. 3

                              This was my first thought as well. There are a lot of very useful things that are too complicated to be implemented by one person - the current state of Linux probably falls into that category, and I know that at least I wouldn’t want to go back to even a version from 5 years ago, much less back to a version that could have been implemented by a single person.

                              1. 2

                                …And there are a lot of useful things that are simple enough for one person to implement! :D

                                1. 3

                                  Ha, I agree with that, was mostly just highlighting that I don’t feel like “too complicated to implement by one person” is a good reason to dismiss Rust’s potential usefulness.

                                  For myself, I originally got frustrated with Rust not allowing me to do things; eventually, I realized that it was statically removing bad habits that I’d built in the past. Now I love when it yells at me :)

                              2. 1

                                [Tool] is too complicated to implement by one person.

                                I’m not sure that’s a practical metric by which to judge a tool

                                I am. Short term, that means the tool will cost much less: less time to make, fewer bugs, more opportunities for improvement. Long term it means other people will be able to rebuild it from scratch if they need to. At a lower cost.

                                1. 3

                                  The flip side of this is that the tool will do much less. A wooden hammer is a tool that a single person can make. A hammer with a steel head that can drive in nails requires a lot more infrastructure (smelting the ore and casting the head are probably large enough tasks that you’ll need multiple people before you even get to adding a wooden handle). An electric screwdriver requires many different parts made in different factories. If I want to fix two pieces of wood together than a screw driven by an electric screwdriver is both easier to use and produces a much better result than a nail driven by a wooden hammer.

                                  1. 1

                                    Obviously I was limiting my analysis to software tools, where the ability of a single person to make it is directly tied to its complexity.

                                    One fair point you do have is how much infrastructure the tool sits upon. Something written in Forth needs almost nothing besides the hardware itself. Something written in Haskell is a very different story. Then you need to chose what pieces of infrastructure you want to depend on. For instance, when I wrote my crypto library I chose C because of it’s ubiquity. It’s also a guarantee of fairly extreme stability. There’s a good chance that the code I write now will still work several decades from now. If I wanted to maximise safety instead, I would probably have picked Rust.

                                    1. 6

                                      Obviously I was limiting my analysis to software tools, where the ability of a single person to make it is directly tied to its complexity.

                                      My point still applies. A complex software tool allows me to do more. In the case of a programming language, a more complex compiler allows me to write fewer bugs or more features. The number of bugs in the compiler may be lower for a compiler written by a single person but I would be willing to bet that the number of bugs in the ecosystem is significantly higher.

                                      The compiler and standard library are among the best places for complexity in an ecosystem because the cost is amortised across a great many users and the benefits are shared similarly. If physical tools were, like software, zero marginal cost goods, then nail guns, pillar drills, band saws, and so on would all be ubiquitous. If you tried to make the argument that you prefer a manual screwdriver to an electric one because you could build one yourself if you needed then you’d be laughed at.

                                      For instance, when I wrote my crypto library I chose C because of it’s ubiquity. It’s also a guarantee of fairly extreme stability

                                      It also gives you absolutely no help in writing constant-time code, whereas a language such as Low* allows you to prove constant-time properties at the source level. The low* compiler probably depends on at least a hundred person-years of engineering but I’d consider it very likely that the EverCrypt implementations of the same algorithms would be safer to use than your C versions.

                                      1. 2

                                        I reckon amortized cost is a strong argument. In a world where something is build once and used a gazillion times the cost analysis is very different from something that only has a couple users. Which is why by the way I have a very different outlook for Oberon and Go: the former were used in a single system, and the cost of a more powerful compiler could easily outweigh the benefits across the rest of the system; while Go set out to be used by a gazillion semi-competent programmers, and the benefit of some conspicuously absent features would be multiplied accordingly.

                                        Honestly, I’m not sure where I stand. For the things I make, I like to keep it very very simple. On the other hand, If I’m being honest with myself I have little qualms sitting on a mountain of complexity, provided such foundation is solid enough.

                                        Do you have a link to Low*? My search engine is failing me.

                                        1. 2

                                          Do you have a link to Low*? My search engine is failing me.

                                          This paper is probably the best place to start

                                2. 1

                                  The C compilers that provide a practical foundation for modern software development were not implemented by one person either.

                                  Right but there are many C compilers which were written by one person and still work. To me, that’s the important part. Thank you for your thoughts!

                                  1. 2

                                    Why is that important?

                                    1. 1

                                      It’s important because fast forward 300 years and no one uses your language anymore. It must be reasonable the future humans can write a compiler on their own if they want to run your program.

                                      I’m really trying to encourage people thinking beyond their lives in the software realm lately, just as we need to do the same for the ecosystem.

                                      1. 3

                                        trying to build software to last 300 years seems like it would limit hardware development
                                        and everyone implements C compatibility in their new hardware so that people will use it
                                        if people can figure out quantum computers and computers not based on binary, they’ll probably need to figure out what the next C will be for that new architecture
                                        if you want your software to last 300 years, write it in the most readable and easy-to-understand manner, and preserve it’s source so people can port it in the future

                                        1. 3

                                          And this is why C is not good for longevity, but languages which are more abstracted. Thank you for that! Completely agree with what you’re thinking here.

                                          1. 3

                                            i don’t think the biggest blockers to software longevity is language choices or even hardware, it’s the economy/politics of it… long lasting anything doesn’t fit in well with our throw-away society, and since it can’t be monetized, the capitalist society snubs it’s nose at it

                                            1. 2

                                              Hehe, an interesting thread of thought we could travel down here. I’ll just say I agree to a degree.

                                        2. 3

                                          It’s important because fast forward 300 years and no one uses your language anymore. It must be reasonable the future humans can write a compiler on their own if they want to run your program.

                                          If you’re considering a person 300 years in the future then you should also consider that they will have tools 300 years more advanced than ours. 30 years ago, writing a simple game like space invaders was weeks worth of programming, now it’s something that you can do in an afternoon, with significantly better graphics. In the same time, parser generators have improved hugely, reusable back ends are common, and so on. In 300 years, it seems entirely feasible that you’d be able to generate a formal specification for a language from a spoken description and synthesise an optimising compiler directly from the operational semantics.

                                          1. 1

                                            You’re right, I haven’t considered this! I don’t know what to say immediately other than I think this is very important to think about. I’d like to see what others have to comment on this aspect too…!

                                            1. 1

                                              you should also consider that they will have tools 300 years more advanced than ours.

                                              Unless there has been a collapse in between. With climate change and peak oil, we have some serious trouble ahead of us.

                                              1. 5

                                                In which case, implementing the compiler is one of the easiest parts of the problem. I could build a simple mechanical computer that could execute one instruction every few seconds out of the kind of materials that a society with a Victorian level of technology could produce, but that society existed only because coal was readily accessible. I’ve seen one assessment that said that if the Victorians had needed to use wood instead of coal to power their technology they’d have completely deforested Britain in a year. You can smelt metals with charcoal, but the total cost is significantly higher than with coal (ignoring all of the climate-related externalities).

                                                Going from there to a transistor is pretty hard. A thermionic valve is easier, but it requires a lot of glass blowing (which, in turn, requires an energy-dense fuel source such as coal to reach the right temperatures) and the rest of a ‘50s-era computer required fairly pure copper, which has similar requirements. Maybe a post-collapse civilisation would be lucky here because there’s likely to be fairly pure copper lying around in various places.

                                                Doping silicon to produce integrated circuits requires a lot of chemical infrastructure. Once you can do that, the step up to something on the complexity of a 4004 is pretty easy but getting lithography to the point where you can produce an IC powerful enough to run even a fairly simple C program is nontrivial. Remember that C has a separate preprocessor, compiler (which traditionally had a separate assembler), and linker because it was designed for computers that couldn’t fit more than one of those in RAM at a time. Even those computers were the result of many billions of dollars of investment from a society that already had mass production, mass mining, and large-scale chemistry infrastructure.

                                                C code today tends to assume megabytes of RAM, at a minimum. Magnetic core storage could do something like 1 KiB in something the size of a wardrobe. Scaling up production to the point where 1 MiB is readily available requires ICs, so any non-trivial C program is going to have a dependency on at least ’80s-era computing hardware.

                                                TL;DR: If a society has collapsed and recovered to the point where it’s rediscovering computers, writing a compiler for a fairly complex language is going to be very low cost in comparison to building the hardware that the compiler can target.

                                                1. 1

                                                  Well, I wasn’t anticipating such a hard collapse. I was imagining a situation where salvage is still a thing, or where technology doesn’t regress that far. Still, you’re making a good point.

                                                  1. 4

                                                    That’s an interesting middle ground. It’s hard for me to imagine a scenario in which computers are salvageable but storage is all lost to the point where a working compiler is impossible to find. At the moment, flash loses its ability to hold charge if not powered for a few years but spinning rust is still fine, as is magnetic tape, for a much longer period, so you’d need something else to be responsible for destroying them. Cheap optical storage degrades quite quickly but there are archive-quality disks that are rated for decades. If anything, processors and storage are more fragile.

                                                    In the event of a collapse of society, I think it’s a lot more likely that copies of V8 would survive longer than any computer capable of running them. The implicit assumption in the idea that the compiler would be a bottleneck recovering from a collapse of society is that information is more easily destroyed than physical artefacts. This ignore the fact that information is infinitely copyable, whereas the physical artefacts in question are incredibly complex and have very tight manufacturing tolerances.

                                                    Of course, this is assuming known threats. It’s possible that someone might create a worm that understands a sufficiently broad range of vulnerabilities that it propagates into all computers and erases all online data. If it also propagates into the control systems for data warehouses then it may successfully destroy a large proportion of backups. Maybe this could be combined with a mutated bacterium that ate something in optical disks and prevented recovering from backup DVDs or whatever. Possibly offline storage will completely go out of fashion and we’ll end up with all storage being some form of RAM that is susceptible to EMP and have all data erased by a solar flare.

                                                    1. 1

                                                      It really depends on what we can salvage, and what chips can withstand salvage operations. In a world where we stop manufacturing computers (or at least high-end chips), I’d expect chips to fail over the years, and the most complex ones will likely go first. And those that don’t will be harder to salvage for various reasons: how thin their connection pins are, ball arrays, multi-layer boards requirements, and the stupidly fast rise times that are sure to cause cross-talk and EMI problems with the hand made boards of a somewhat collapsed future.

                                                      In the end, many of us may be stuck with fairly low-end micro controllers and very limited static memory chips (forget about controlling DRAM, it’s impossible to do even now without a whole company behind you). In that environment, physical salvage is not that horrible, but we’d have lost enough computing power that we’ll need custom software for it. Systems that optimise for simplicity, like Oberon, might be much more survivable in this environment.

                                                      C code today tends to assume megabytes of RAM, at a minimum.

                                                      In this hypothetical future, that is relevant indeed. Also, I believe you. But then the first serious project I wrote in C, Monocypher, requires only a couple KB of stack memory (no heap allocation) for everything save password hashing. The compiled code itself fits requires less than 40KB of memory. Thing is, I optimised it for simplicity and speed, not for memory usage (well, I did curb memory use a bit when I’ve heard I had embedded users).

                                                      I suspect that when we optimise for simplicity, we also tend to use less resources as a side effect.


                                                      Now sure, those simple systems will take no time to rebuild from scratch… if we have the skills. In our world of bigger and faster computers with a tower of abstraction taller than the Everest, I feel most of us simply don’t have those skills.

                                                      1. 4

                                                        Now sure, those simple systems will take no time to rebuild from scratch… if we have the skills. In our world of bigger and faster computers with a tower of abstraction taller than the Everest, I feel most of us simply don’t have those skills.

                                                        While it’s an interesting thought exercise, but I think this really is the key point. The effort in salvaging a working compiler to be able to run some tuned C code in a post-apocalyptic future may be significantly higher than just rewriting it in assembly for whatever system you were able to salvage (and, if you can’t salvage an assembler, you can even assemble it by hand after writing it out on some paper. Assuming cheap paper survives - it was very expensive until a couple of hundred years ago).

                                                        Most of us probably don’t have the skills to reproduce the massive towers of abstraction that we use today from scratch but my experience teaching children and young adults to program suggests that learning to write simple assembly routines is a skill that a large proportion of the population could pick up fairly easily if necessary. If anything, it’s easier to teach people to write assembly for microcontrollers than JavaScript for the web because they can easily build a mostly correct mental model of how everything works in the microcontroller.

                                                        Perhaps more importantly, it’s unlikely that any software that you write now will solve an actual need for a subsistence level post-apocalyptic community. They’re likely to want computers for automating things like irrigation systems or monitoring intrusion sensors. Monocypher is a crypto library that implements cryptosystems that assume an adversary who had access to thousands of dedicated ASICs trying to crack your communications. A realistic adversary in this scenario would struggle to crack a five-wheel Enigma code and that would be something that you could implement in assembly in a few hours and then send the resulting messages in Morse code with an AM radio.

                                                        1. 1

                                                          Most of us probably don’t have the skills to reproduce the massive towers of abstraction that we use today from scratch but my experience teaching children and young adults to program suggests that learning to write simple assembly routines is a skill that a large proportion of the population could pick up fairly easily if necessary.

                                                          I feel a little silly for not having thought of that. Feels obvious in retrospect. If people who have never programmed can play Human Resource Machine, they can probably learn enough assembly to be useful.

                                                          Perhaps more importantly, it’s unlikely that any software that you write now will solve an actual need for a subsistence level post-apocalyptic community.

                                                          Yeah, I have to agree there.

                                            2. 2

                                              Today’s humans were able to create Rust, so I don’t see why future humans wouldn’t. Future humans will probably just ask GPT-3000 to generate the compiler for them.

                                              If you’re thinking about some post-apocalyptic scenario with a lone survivor rebuilding the civilisation, then our computing is already way beyond that. In the 1960’s you were able to hand-stitch RAM, but to even hold source code of modern software, let alone compile and run it, you need more technology than a single person can figure out.

                                              C may be your point of reference, because it’s simple by contemporary standards, but it wasn’t a simple language back when the hardware was possible to comprehend by a single person. K&R C and single-pass C compilers for PDP-11 are unusable for any contemporary C programs, and C is too complex and bloated for 8-bit era computers.

                                              1. 1

                                                If GPT can do that for us then hey, I will gladly gladly welcome it. I’m not thinking about a post-apocalyptic scenario but I can see the relationship to it.

                                              2. 2

                                                But why one person? I think we’ll still write software in teams in 2322, if we write software at all by that point instead of flying spaceships and/or farming turnips in radioactive wastelands. The software was written by teams today, and I think, if it needs to be rewritten, it will be rewritten by teams in the future.

                                                1. 1

                                                  I would also be careful about timespans here. computers haven’t been around for a century yet, so who knows what things will be like 100 years from now? I don’t even know if it’s possible to emulate an ENIAC and run old punch card code on modern hardware, that’s the sort of change we’ve seen in just 75y. maybe multicore x86 machines running windows/*nix/BSD will seem similarly arcane 300y from now.

                                              3. 1

                                                Wouldn’t a published standard be more important to future programmers? Go might be a wonderful language, but is there a standards document I can read from which an implementation can be written from?

                                            1. 53

                                              This article is a cautionary tale about dogma and religious belief in computing, and the role of “serverless” is only ancillary. Teams making hundreds of repos, copy pasting code, chasing shiny new thingies configured with YAML/JSON/EDN/… - this can happen with almost any underlying deploy technology in the name of “Microservices”. The underlying issue is not the addictive nature of the cloud provider’s overhyped abstractions - it is this:

                                              When it’s not okay to talk about the advantages and disadvantages of [The Way] with other engineers without fear of reprisal, it might be a cult. Many of these engineers say [The Way] is the only way to [Be] anymore.

                                              In this article, The Way is “AWS lambda”, but it could easily be “Kubernetes”, “Event sourcing”, “Microkernels”, “Microservices”, “Cluster framework (akka/erlang/…)” or any other technology. Anything that can be a panacea that becomes a poison when dogma is too strong.

                                              I enjoy cautionary tales, but I’d be more interested to hear how to solve the underlying cultural issue of dogma within an org, since changing minds and engineering practice is so much harder than changing code.

                                              1. 17

                                                That’s one takeaway, but an equally important one is that serverless has inherent limitations when compared with “non-scalable” solutions … like hosting your app on a Unix box. That includes the first one (testing), “microservice hell”, and “inventing new problems”.

                                                The last one is particularly important. In more time-tested stacks there are “idioms” for each kind of problem you will encounter. Serverless only provides a very small part of the stack (stateless, fine grained), and if your application doesn’t fit (which is very likely), then you will need to invent some custom solution.

                                                And what he’s saying is that, after experience, those custom solutions (“hacks”) ended up worse than the time-tested ones.

                                                I agree the abuse of HTTP falls in the other category though. That is not inherent to serverless – if you got it wrong there, you would get it wrong everywhere.

                                                1. 9

                                                  You could make a similar cautionary tale about Enterprise software set in the early 00s with the rise of Java and XML.

                                                  1. 3

                                                    Yep, if you squint, AWS serverless is similar in outcome to IT-managed WebSphere. Although, at least WebSphere tried to be J2EE-compliant. Lambda and other services are purely proprietary, with no attempt to provide open standards.

                                                    1. 13

                                                      Lambda is just Greek for “bean”.

                                                  2. 4

                                                    I think this is symptomatic of the problematic relationship we have with senior engineers. Software engineering has a tendency for neophilia, where if someone criticizes a new idea they’re often dismissed as an “old timer” who’s afraid of change and whose opinions are irrelevant. I have the impression at least that in other fields the opinion of seniors are taken very seriously because they have experience that juniors lack. Certainly seniors are less prone to be wooed by new technology and have an abundance of past experience that the technology can be evaluated against. It’s really hard to ask critical questions like “How would new technology X handle scenario Y?” if you’ve never had to deal with “scenario Y” before.

                                                    One idea would be to have some kind of “senior engineer advisory board” or “council of elders” that could weigh in on technical decisions. At Pixar they famously had a “brain trust” of experienced employees that would vet important decisions at various stages of the production. The point would be to institutionalize some kind of sanity checking with more experienced developers so that we don’t have to make the same mistakes twice, both as an organization and as a field.

                                                    I’m not advocating for letting senior engineers rule by fiat, just that we should look more to seniors for guidance and counsel, just like pretty much every other industry seems to be doing.

                                                    1. 5

                                                      Microkernels

                                                      That item does not belong in that list.

                                                      1. 3

                                                        Why not? The point is that in a healthy engineering culture, no technology choice should be canon law, unable to be consciously weighed against alternatives, whether it’s your favourite pet or not.

                                                        1. 7

                                                          It lacks the cargo cult; Everybody seems to hate ’em. For no good reason.

                                                          1. 6

                                                            I think that’s uncharitable, and also untrue. Uncharitable because clearly lots of people are interested in microkernels one way or another, and because like all technical choices it represents a trade-off between competing concerns; your expectations of system behaviour may well not match the expectations of somebody less enthusiastic about microkernels.

                                                            It’s untrue because I hear loud positive noises about microkernels, even just on this site, all the time; e.g., your own comment over here!

                                                            1. 1

                                                              Yes, it is pretty much just me, and that is sad.

                                                            2. 4

                                                              I think that’s untrue now but it’s alternated. In the early ’90s, there was a lot of dogma around microkernels. OSF/1 was going to change the UNIX landscape completely. Minix was a demonstration of how operating systems should be written. Linux was a terrible design, *BSD was a legacy thing that would eventually just become a server on a microkernel (or, ideally, ripped apart and bits of it used to build multiple servers).

                                                              Then people got performance data on Mach. System call overheads were insanely high because Mach did port rights checking on every message and a system call required at least two messages to handle. The dogma shifted from microkernels are unconditionally good to microkernels are unconditionally bad.

                                                              In the background, systems like L4 and QNX showed that microkernels can outperform monolithic kernels but that didn’t really take off until multicore systems became common and shared-everything concurrency in a single kernel proved to be a bottleneck. Most of the debate moved on because microkernels quietly won by rebranding themselves as hypervisors and running Linux / *BSD / Windows as a single-server isolation layer.

                                                              These days, the L4 family is deployed on more devices than any other OS family, most monolithic kernels are run on top of a microkernel^Whypervisor, and anyone writing a new OS for production use is building a microkernel. Even monolithic kernels are adding userspace driver frameworks and gradually evolving towards microkernels. There’s a lot less dogma.

                                                      1. 14

                                                        “A new chapter begins for X” always reads like “things will get miserable for X”.

                                                        1. 4

                                                          In general I suspect that’s true, but having worked for Samsung in the past I would venture that this can literally only improve the situation for people working on and for people consuming the software.

                                                          1. 2

                                                            With you and many other former Joyent people at Oxide doing your own Illumos things.

                                                            Do you know how many people are still working on SmartOS and/or SDC/Triton still?

                                                          2. 3

                                                            “our incredible journey”

                                                          1. 6

                                                            Tailscale’s free tier is not very feature rich. You can’t do user ACLs with it, for example. And yes, you’re locked into a Google/Microsoft identity. If you don’t like that, pay for more. If you can’t, go use another service or roll it on your own.

                                                            Articles like this are why I’ll probably never make a service with a free tier. Yikes.

                                                            1. 21

                                                              Articles like this are why I’ll probably never make a service with a free tier. Yikes.

                                                              Because someone might write a polite and even-handed criticism? I have certainly read and indeed written substantially less charitable things about companies that are not giving their stuff away for free.

                                                              1. 12

                                                                Tailscale employee here. +1, my reading of iliana’s post is absolutely not “wow, those ingrate free users”. It’s a very well constructed critique of one of our product choices, and my reaction is a web of different things (mostly “I agree, and have complex thoughts on why we haven’t done the thing yet”), but I’m very grateful for this feedback.

                                                                1. 10

                                                                  Another Tailscale employee here. +1, iliana means very well by this. Xie means very will by this and xer feedback is already in the awareness of the product side of things at Tailscale. I was thinking out ideas on how to work around this with xer and presented them alongside that article. Overall I’m fairly sure that this will end up with positive change.

                                                                  1. 6

                                                                    Talking about ICE and union busting in this context doesn’t feel even-handed to me. 🤷🏽‍♀️

                                                                    I have certainly read and indeed written substantially less charitable things about companies that are not giving their stuff away for free.

                                                                    I’m confused, isn’t that exactly the point? If Tailscale only offered Google/MS login on their paid corporate plan this article would make much more sense?

                                                                    1. 8

                                                                      Talking about ICE and union busting in this context doesn’t feel even-handed to me

                                                                      A footnote explaining why the OP doesn’t want to work with Microsoft and Google seems entirely relevant, no? Google and Microsoft do work with immigration authorities, and they do engage in union-busting; these are facts, not imputation or speculation.

                                                                      1. 1

                                                                        But the OP isn’t forced to work with them. They can use the free service Google/MS provide or pay Tailscale to bring a different identity provider.

                                                                        That makes Google/MS business with immigration extremely irrelevant to an article about Tailscale.

                                                                        1. 9

                                                                          I’m really not sure what you mean. The complaint here is that the free tier of Tailscale requires people to use Google or Microsoft’s services. Explaining why Google and Microsoft are parties the OP doesn’t want to work with is at least somewhat relevant.

                                                                1. 11

                                                                  Didn’t expect Oxide to go with a home-grown Illumos-based OS. I can appreciate the desire to control the entire stack, but on the other hand, Oxide is now maintaining an OS all by themselves (I would be surprised if the Illumos community is very large/active, but I could be wrong). There is also the question of who will want to commit their servers to a niche OS that only one company can realistically do anything about (say, fix something like Spectre/Meltdown).

                                                                  1. 26

                                                                    It’s less problematic than it probably sounds since:

                                                                    1. They control the hardware and can probably ignore other hardware while following upstream.
                                                                    2. They have some serious OS experience in the team already who are deeply familiar with Illumos.
                                                                    3. Illumos is only the host OS. Their product is all about on-prem cloud. So their users will mostly not know that Illumos is under the hood. They’ll be launching VMs via API and running mostly Linux.
                                                                    1. 22

                                                                      users will mostly not know that Illumos is under the hood. They’ll be launching VMs via API and running mostly Linux.

                                                                      This is a key observation. Consider both AWS and VMware, where people may suspect details of the underlying proprietary implementation but are really only interacting with the explicitly exposed API surface area which doesn’t expose those details to the consumer.

                                                                    2. 20

                                                                      Didn’t expect Oxide to go with a home-grown Illumos-based OS.

                                                                      Solaris is probably mothers’ milk to people like Cantrill, who know it deeply (enough to certainly maintain it) and have a low opinion of Linux.

                                                                      1. 15

                                                                        There is something to be said for an operating system community that moves slowly. You don’t get to take advantage of a large community, but you also won’t get washed away by that large community. Keeping up with a firehose has its own overhead, and your priorities probably won’t match the community’s.

                                                                        It’s a lesson I learned watching node.js go mainstream. The emphasis on observability and debuggability went right out the window, because the community at large cared far more about new features than dealing with hard problems in production.

                                                                        I’m guessing Oxide went with illumos more out of familiarity, but the above reasoning doubtless played a non-trivial role too.

                                                                        1. 8

                                                                          I agree, hitching yourself to the wrong community is a consideration. But in this case, I think the goals of Linux (kernel, qemu/KVM, etc) would be pretty well aligned with Oxide’s goals and I don’t see a reason why they would change going forward.

                                                                          FWIW, we’ve made a similar decision about 5 years ago by creating BuildOS which was heavily inspired by SmartOS (a PXE-booted in-RAM OS that is essentially a VM runner) but was based on Linux. So far no complaints.

                                                                        2. 7

                                                                          Nerd cred? :)

                                                                          Seriously, The Solaris descended operating systems have an incredible rep with a good reason. For literally decades, Sun hardware and software were THE gold standard in enterprise grade stability, and even in our industry where sometimes it feels like we forget everything we’ve ever learned every few years, things like that don’t just evaporate.

                                                                          1. 7

                                                                            Yes. They did a lot of things super right and it feels like we’re still playing catch up in some ways.

                                                                            SMF vs SystemD

                                                                            Dtrace vs eBPF

                                                                            ZFS vs BTRFS

                                                                          2. 3

                                                                            I had the same thought. I shouldn’t be surprised given Bryan’s involvement but I still didn’t expect it—especially with the Rust project’s lack of interest in making anything but Linux, macOS, and Windows tier 1.

                                                                            1. 4

                                                                              Tier 1 doesn’t matter that much. The only difference from Tier 2 is that automated runtime testing is run continuously.

                                                                          1. 16

                                                                            I can just recommend to everybody which starts with rust: Start with building a simple single threaded program, e.g. a CLI.

                                                                            Concurrency makes lifetimes much harder to understand. And async by far.

                                                                            Don’t be fooled by simple looking async/await examples. Start without it.

                                                                            1. 6

                                                                              Agreed. I love Rust and use it basically daily now, but when first learning it I simply bounced off hard, multiple times, until finally I scaled my ambitions down to something small enough I could actually figure out what was going on. It was a Conway’s Life implementation that just printed results out to the terminal. Start smaller than you think you need to.

                                                                              1. 3

                                                                                advent-of-code type of stuff looks okay to just pick up the syntax and specifics.

                                                                                1. 3

                                                                                  If you’re going to do AOC problems, I recommend becoming comfortable in short order with Clone, and not worrying about the overhead of doing so. I would also investigate some ways to store graphs and trees without expecting to be able to use references (e.g., by using maps with integer node IDs as keys) as it can quickly become complex or impossible to structure a correct program in the way you might in a less strict language.

                                                                              2. 2

                                                                                Cannot agree more. Async in Rust brings in a lot of complexity and it’s definitely not necessary when one is getting started.

                                                                              1. 4

                                                                                I saw that this was posted before by someone else, but it’s been quite a while and the project wasn’t very mature back then, so I’d like to share it again in case it may be interesting to other people, especially the *BSD folks, as it’s one of the few native clients for those platforms.

                                                                                It makes use of librespot, an awesome project with the goal of reverse engineering the Spotify protocol. In the past years some other cool projects were started with a similar intention, e.g. psst (which even has its own protocol implementation) or Spot.

                                                                                1. 1

                                                                                  A colleague put me onto it recently. For whatever reason I couldn’t get used to some of the default key bindings, but it’s quite customisable and after I changed those ncspot has been delightful! Robust and responsive.

                                                                                  1. 1

                                                                                    Thank you, glad to hear that! They’re mostly based on ncmpc bindings, but may be quite opinionated especially if you’re not used to them. Finding bindings that suit everyone is quite hard I learned! :)

                                                                                  2. 1

                                                                                    I’m not sure if it’s librespot or spotifyd specifically, but I’ve had almost no luck getting spotifyd working on FreeBSD. It tries to bind a port that’s already in use, doesn’t gracefully recover, and even when it does connect and register, most of the time the iOS Spotify client doesn’t see it.

                                                                                  1. 1

                                                                                    The package manager he is describing is Hermit!

                                                                                    1. 2

                                                                                      These use cases are exactly what Hermit solves - it doesn’t do any packaging, library installation, dependency management, etc. - it just download binaries and makes them available. Its stubs can be checked into Git, and for users Hermit doesn’t require installation, it bootstraps itself.

                                                                                      OK, I’ve taken a quick look at Hermit, and for once it is itself a “single binary executable” (written in Go).

                                                                                      However I don’t think it’s the right solution:

                                                                                      • it works mainly with projects, although I think one could “convince” it to install globally;
                                                                                      • it supports only Linux and OSX, although I don’t see anything that technically would stop it being ported to BSD’s; (especially OpenBSD which seems to have very little “love” from many tool developers;)
                                                                                      • it supports only bash and zsh; other shells I assume won’t usually work due to the “activation” requirement (see below);
                                                                                      • it depends on a centralized repository for available “packages”; I’m sure one can change the URL, but still in order for one to be able to install something there must be a “recipe” in that repository; (at the moment there are only 141 such “packages”;)
                                                                                      • it also depends on a working internet connectivity; (at least for the initial install;)

                                                                                      But most importantly, it relies heavily on “touching” my shell environment, which personally I’m very sensitive of. For example hermit install something doesn’t even work without “activation”, which implies sourcing a bunch of bash in my environment… (I will never do that!)

                                                                                      Also, a deal breaker for me, the shims that Hermit installs in my project’s bin folder are bash scripts… Just looking at them I see one quoting issue (if my HERMIT_STATE_DIR contains spaces I’m doomed):

                                                                                      export HERMIT_EXE=${HERMIT_EXE:-${HERMIT_STATE_DIR}/pkg/hermit@${HERMIT_CHANNEL}/hermit}
                                                                                      

                                                                                      But even more worryingly, as part of the normal execution (if the tool is not installed) it just curl’s and runs a bash script:

                                                                                      curl -fsSL "${HERMIT_DIST_URL}/install.sh" | /bin/bash 1>&2
                                                                                      

                                                                                      Say GitHub has a bad day and just spits out some 404 HTML… I’ll all of a sudden start executing HTML with bash

                                                                                      So overall, although it’s a nice and pragmatic tool, I don’t think it’s the right choice, especially given the security implications with all that curl | bash-ing…

                                                                                      1. 3

                                                                                        You seem to be fixated on a specific approach you have in mind, which is fine - each to their own - but I’ll just correct a few invalid observations.

                                                                                        various limitations such as shell support, OS support

                                                                                        Shell support is only required for “activation”. Executing Hermit binaries directly works in any shell. For OS etc., it’s just a matter of a) package availability and b) someone to test and support it.

                                                                                        export HERMIT_EXE=${HERMIT_EXE:-${HERMIT_STATE_DIR}/pkg/hermit@${HERMIT_CHANNEL}/hermit}

                                                                                        This one is interesting, we run shellcheck on our scripts so it’s not clear to me why it’s not picking this up. I’ll have to take a look.

                                                                                        it depends on a centralized repository for available “packages”

                                                                                        This is incorrect - repositories can define their own manifest sources, including bundling their own in a sub-directory.

                                                                                        For example hermit install something doesn’t even work without “activation”, which implies sourcing a bunch of bash in my environment… (I will never do that!)

                                                                                        Incorrect. It is not necessary to activate an environment to use the stub scripts, you can just run them eg. ./bin/make

                                                                                        it also depends on a working internet connectivity; (at least for the initial install;)

                                                                                        Quite a bizarre complaint given that basically everything has to come from the Internet, including the repository the user presumably cloned. Of course it needs to be bootstrapped from somewhere. Once bootstrapped, packages are shared.

                                                                                        curl -fsSL “${HERMIT_DIST_URL}/install.sh” | /bin/bash 1>&2

                                                                                        Say GitHub has a bad day and just spits out some 404 HTML… I’ll all of a sudden start executing HTML with bash…

                                                                                        Also completely incorrect, for two reasons which I’ll leave as an exercise for the reader.

                                                                                        1. 3
                                                                                          export HERMIT_EXE=${HERMIT_EXE:-${HERMIT_STATE_DIR}/pkg/hermit@${HERMIT_CHANNEL}/hermit}
                                                                                          

                                                                                          This one is interesting, we run shellcheck on our scripts so it’s not clear to me why it’s not picking this up. I’ll have to take a look.

                                                                                          Though I would probably put the quotes in anyway because it can’t hurt and is easier to remember, I don’t think it’s actually unsafe – which is probably why shellcheck is not complaining. In the specific context of assigning a variable, the kind of word expansion that injects spaces between arguments does not actually occur:

                                                                                          $ cat /tmp/trial.bash
                                                                                          a='a b c'
                                                                                          export b=$a
                                                                                          export c=${C:-$b}
                                                                                          
                                                                                          set | grep '^[a-c]'
                                                                                          
                                                                                          $ bash /tmp/trial.bash
                                                                                          a='a b c'
                                                                                          b='a b c'
                                                                                          c='a b c'
                                                                                          
                                                                                          $ C='value with spaces' bash /tmp/trial.bash
                                                                                          a='a b c'
                                                                                          b='a b c'
                                                                                          c='value with spaces'
                                                                                          
                                                                                    1. 5

                                                                                      I still feel like there’s some synthesis out there on the static linking/dynamic linking (early binding/late binding) question. Static is better for production, no doubt about that. In basically every scenario, it’s better to have a system that is tested and known to work than an untested glued together conglomeration. But dynamic has its pull, and there must be reasons for that too. I think until we find a way to do static with the benefits of dynamic, people will keep trying to use dynamic in production and getting burnt.

                                                                                      1. 10

                                                                                        Static is better for production, no doubt about that

                                                                                        It’s not quite so clear cut. Static linking has some big advantages: it makes updates easier, it enables whole-program optimisation, it avoids indirection for cross-library calls, and so on. It has a couple of big down sides, one technical and the other legal:

                                                                                        • There is no code sharing between processes that link the same library.
                                                                                        • Distributing statically linked binaries that use LGPL libraries is a headache.

                                                                                        The first has three levels of impact. You need more space on disk for the duplicated code but that doesn’t matter very much with modern disk sizes. You need more space in RAM for the duplicated code. This can impact performance directly if it leads to swapping, indirectly if it causes less RAM to be available as file cache. This is typically only an issue if there’s a lot of code shared between processes. Finally, it increases instruction cache usage, causing cache misses in both programs that call different copies of the same function. This is more of an issue and affects systems with multiple processes in the runnable state, though I don’t know to what degree. It’s less of an issue on server systems where the trend is towards a single program in a single container on a single VM.

                                                                                        The second is a mess because you must provide recipients of your program the ability to relink the resulting program against their own version of the LGPL’d dependency. With dynamic linking, this is trivial (just replace the .so), with static linking it depends on providing the .o files for your application and a build script.

                                                                                        There’s also the auditing issue. Unless your statically linked binary comes with an SBOM, it’s difficult to tell if it links against a vulnerable dependency, whereas with dynamic linking you can see the library that it depends on and update it when there’s a security fix.

                                                                                        Apple has a somewhat hybrid model where they link a load of core libraries together into a massive blob and inject that into all processes, so that they can have cross-library optimisation within their own code and can avoid the start-up costs of applying dynamic relocations into their code and get good code sharing. More than 90% of the (used) code in a typical Cocoa app is in these libraries (there’s also a load of code that is dead in any given program). They also do a neat trick of reusing the leaf page-table pages for this entire section between all processes, which means that they’re always hot in the cache (and the higher-level pages typically are for the running program) which significantly reduces the cost of a TLB miss in this region.

                                                                                        This works for Apple because they have a rigid separation between system libraries (i.e. things bundled with macOS/iOS) and third-party libraries (i.e. things bundled with an app / installed via Homebrew or whatever). They don’t get much benefit from sharing in the second category because you’re unlikely to have very many apps that all ship their own copy of the same third-party library but they get a huge benefit from sharing the first category because everything links that. Generally, apps can statically link third-party dependencies if they need to and use rpath linking for LGPL (or similar) .dylibs that are stored within the app bundle (note: I believe this is probably illegal with signed binaries, because the end user can’t replace the .dylib without preventing the application from running but I’m not aware of anyone testing that in court).

                                                                                        In *NIX systems, the library providing the web view is probably the largest one that multiple programs link and keep resident at the same time. For security, this is increasingly a fairly thin wrapper library that runs a child process that does all of the rendering and just provides event injection and rendered output. This model works fine with static linking (as long as you have a stable RPC ABI between the child process and the parent).

                                                                                        I liked the old PC-BSD PBI model, which installed every application in a separate directory tree, with all of its libraries in ${APPROOT}/lib and so on. It then used hard links to deduplicate redundant copies of the same library. Everything used $ORIGIN rpaths for linking. It wasn’t fantastic for download speed (every KDE app required you to download 300+ MiB of KDE libraries, for example, which were then thrown away because you had a copy locally) but that would be avoidable fairly easily. This gave you the separate update ability but with the total code-size advantages of dynamic linking.

                                                                                        Vikram Adve also has a project calls ALLVM that looks promising. The idea here is to distribute everything as IR and make a decision at install time which bits belong in a static binary and which in shared libraries. This (in theory) lets you identify functions that are common to multiple programs and pull those out into dynamic libraries but keep specialised implementations of any of them privately, and redo this decision at any point.

                                                                                        1. 3

                                                                                          There is no code sharing between processes that link the same library.

                                                                                          Unfortunately I don’t think this is actually an issue currently. For example there have been quite a few writings about this subject, especially:

                                                                                          […] it’s difficult to tell if it links against a vulnerable dependency, whereas with dynamic linking you can see the library that it depends on and update it when there’s a security fix […]

                                                                                          Unfortunately this also isn’t usually true, especially when using pre-built binaries, because many projects often ship their own version of well known libraries. A good example is zlib that is “vendored” quite a lot; another example is openssl. Thus these vendored libraries don’t appear on a simple ldd or readelf invocation.


                                                                                          All in all thanks for all the insight into what Apple is doing and PC-BSD used to do.

                                                                                          About the LGPL legality issue, I wasn’t aware of that requirement, although given most tools are open-source, legally I don’t think its actually an issue as the user can just ditch the pre-built binary and use the source original code.

                                                                                          1. 10

                                                                                            https://drewdevault.com/dynlib – by Drew DeVault, written in June 2020;

                                                                                            Looking at the list, it appears that Drew ran this on a system that had almost no modern GUI applications installed. The only GUI libraries that show up are in the low-level X ones. You’ll see something very different on a system with KDE or GNOME installed, but even then he’s measuring the wrong thing. It doesn’t matter how many programs link shared libraries, it matters how many are running at the same time and how big the shared libraries are. On a Windows / macOS system, a typical GUI app is linking tens or hundreds of MiBs of shared libraries and those same libraries are used by all graphical apps. Something similar is true if you’re running a suite of KDE of GNOME apps, though the fact that things like Firefox and OpenOffice use their own toolkits alters this somewhat.

                                                                                            The claim about 4.6% of symbols is ignoring transitive dependencies. For a trivial example, if I call printf then I’m calling a libc function that is half a dozen instructions. Definitely smaller in static linking, right? Well, no, because printf is a thin wrapper around vfprintf, which calls localeconv and ends up pulling in a huge amount of object code in total. I probably hit more than 4.6% of the symbols in libc from a single printf call, which makes me deeply suspicious of the rest of this claim. Note that the ‘a good linker will remove unused symbols’ is true only in languages with no dynamic dispatch. In C, if you take the address of a function but never call it, it (and everything reachable from it) has to be in the final binary. The linker can’t do any analysis here (go can do a lot more in the front end), it doesn’t know that you’ve assigned the function pointer to a structure but you never actually call that function.

                                                                                            https://lore.kernel.org/lkml/… – reply by Linus Torvalds, written last in May 2021;

                                                                                            The key bit in Linus’ comment is this:

                                                                                            Yes, it can save on disk use, but unless it’s some very core library used by a lot of things (ie particularly things like GUI libraries like gnome or Qt or similar), the disk savings are often not all that big - and disk is cheap. And the memory savings are often actually negative (again, unless it’s some big library that is typically used by lots of different programs at the same time).

                                                                                            Note the qualification. Dynamically linking a small library, especially one where you use few symbols, is a net loss. Dynamically linking a large library that provides you with a lot of services and that a lot of things link is generally a net win.

                                                                                            The specific context of this thread is dynamically linking clang. This is well documented as a bad idea. There are two situations when you should do it:

                                                                                            • If you want a short compile-test-debug cycle, because building a dynamically linked clang is a lot faster than building a statically linked one. Don’t do this if you run the whole test suite because it takes longer to run the test suite and dynamically link than it takes to run the test suite and statically link.
                                                                                            • If you are in an environment where disk space is very constrained and you are installing all of the LLVM tools.

                                                                                            LLVM is currently exploring a crunchgen-like approach for the second case, where clang, lldb, lld, and so on are all in the same statically linked binary that dispatches based on the value of argv[0]. The follow-up post points out that dynamically linked LLVM is under half the size of statically linked LLVM and neither is particularly small. This is a lot worse for some of the binutils replacements, which are a few KiB of code per utility and which depend on a few MiBs of LLVM library code, than it is for clang which has a large chunk of clang-only code (though shared with libclang, which a load of other tools such as clangd use).

                                                                                            even your arguments are heavily qualified with conditions;

                                                                                            Because it’s not a clear-cut call. If static linking were universally better, we’d use it everywhere. We don’t because it’s definitely worse in some cases and evaluating which is better is a fairly non-trivial decision.

                                                                                            Unfortunately this also isn’t usually true, especially when using pre-built binaries, because many projects often ship their own version of well known libraries. A good example is zlib that is “vendored” quite a lot; another example is openssl. Thus these vendored libraries don’t appear on a simple ldd or readelf invocation.

                                                                                            I guess that depends a bit on your corpus as well. The packaged versions of these often use the system-provided one, for precisely the auditing reason.

                                                                                            About the LGPL legality issue, I wasn’t aware of that requirement, although given most tools are open-source, legally I don’t think its actually an issue as the user can just ditch the pre-built binary and use the source original code.

                                                                                            Assuming that most tools are open source is a very big assumption. It’s definitely true for things in open source package repos (by definition) but it’s far from true in the general case.

                                                                                        2. 2

                                                                                          Though it’s clearly a trendy perspective, I don’t think this is really an absolute truth:

                                                                                          Static is better for production

                                                                                          I think part of your justification is not unreasonable, i.e.,

                                                                                          it’s better to have a system that is tested and known to work than an untested glued together conglomeration

                                                                                          But that’s not really a question of linking, it’s a question of testing and of packaging and distribution.

                                                                                          If you test a particular combination of dynamic libraries and executables and then ship exactly those files to production, then your test coverage is just as good as with a static binary. This is arguably what people have been doing for quite some time with Docker and other container image formats.

                                                                                          1. 2

                                                                                            At $WORK, we have a few services written in Lua. To ease installation, the program is basically a simple C program where Lua, the Lua modules we use (both written in C and Lua) and the main program written in Lua, are all statically linked into an executable. That way, ops just has to install a single executable when we update the code, instead of the platforms package format (we use Solaris in production—does Solaris even have something like RPMs? I honestly don’t know).

                                                                                            1. 3

                                                                                              we use Solaris in production—does Solaris even have something like RPMs? I honestly don’t know

                                                                                              Sure! It depends which version. Solaris 10 had older packaging tools that are similar in principle to RPM. You generally had to shepherd the packages onto a system yourself or use NFS, and dependency management was less than stellar.

                                                                                              Solaris 11 systems have a newer packaging technology, IPS (aka pkg), which provides much richer tooling, strong management of dependencies between packages and version locking, network accessible repositories, etc.

                                                                                              I would just build an IPS package from the Lua files and then you can install it over and over again on new systems.

                                                                                              1. 2

                                                                                                The program started out on Solaris 5.10 and we have it working. I don’t think we’ll change, even if we’ve upgraded to Solaris 5.11 (we have a department that does the deployments and I’m not sure how they set things up).

                                                                                            2. 1

                                                                                              I’m not looking at the very narrow question of “use DLLs/.so files or not?” I am trying to ask about static vs. dynamic in the very broadest meaning possible. The essential part of static vs. dynamic is not “does this use one file or multiple files?” As the words “static” and “dynamic” mean, the question is “can part of this be swapped out or not?” Swapping parts out is an operational risk. If you distribute systems as a collection of files but don’t swap them out, they’re effectively static, you’re just relying on multiple files for whatever reason.

                                                                                          1. 7

                                                                                            “One of my mates realize me that a lot of people grown up even without ever using the INS/DEL HOME/END PGUP/PGDN layout.”

                                                                                            To take a different perspective, I see this as a tradeoff, not a loss. The tradeoff is better touch-based manipulation (indirectly via trackpads and directly via touchscreens) facilitating more natural ways of viewing content. There’s also been more content-derived methods of paging through content (e.g. the headings sidebar in google docs, cmd+t, cmd+p shortcuts in coding editors, etc.). It’s arguably more modal (i.e. less obvious) than dedicated buttons but more precise than a fixed-length approach of ‘pagination’ that could be argued as more of a skeumorphism back to physical paper.

                                                                                            1. 3

                                                                                              better touch-based manipulation (indirectly via trackpads

                                                                                              I had Macbooks for years, and the more I had to hold my fingers in the unnatural two-finger scrolling claw pose the more strained my hands felt. I was glad to eventually give up multi-touch trackpads completely. On the other hand, I never felt like the Page Up/Down buttons were giving me repetitive strain injuries.

                                                                                              1. 1

                                                                                                Were you using your thumb and pointer to scroll? I use my pointer and ring fingers, and they just fall in the right position to scroll. Easy peasy, and certainly no stress. (Macbook Pro Retina 15” 2018.)

                                                                                            1. 19

                                                                                              Out-of-tree and will never be mainlined

                                                                                              Slow performance of encryption

                                                                                              I think these are problems with linux, not about zfs.

                                                                                              Phoronix benchmarks

                                                                                              Are not rigorous, and I would not trust them, broadly speaking.

                                                                                              If you use ZFS’s volume management, you can’t have it manage your other drives using ext4, xfs, UFS, ntfs filesystems

                                                                                              That’s not true. You can run any FS you like on a zvol.

                                                                                              No disk checking tool (fsck)

                                                                                              Scrub?


                                                                                              I also find it rather strange to complain about bugs and them recommend btrfs, but maybe that’s just me…

                                                                                              1. 3

                                                                                                Out-of-tree and will never be mainlined

                                                                                                Slow performance of encryption

                                                                                                I think these are problems with linux, not about zfs.

                                                                                                I don’t know anything about ZFS encryption speed, but the license is gratuitously not GPL-compatible. Fortunately one may conjoin ZFS and Linux on one’s own machine, but it would be nice were it supportable out of the box. The steps to install with a ZFS root are more complex than is desirable.

                                                                                                1. 2

                                                                                                  People often parrot that it’s (gratuitously, really?) incompatible a lot, but it seems like Canonical, at least, don’t agree. As far as I can tell they’re shipping ZFS binaries in their distribution – and more power to them!

                                                                                                  I personally find it hard to imagine how one could conceive of a file system, constrained as it is to a loadable kernel module and developed originally for a wholly unrelated operating system, to be a derived work of Linux. Until someone takes Canonical to court we just don’t know if the two licences are actually incompatible for this particular case.

                                                                                              1. 61

                                                                                                Please don’t pay attention to this article, it’s almost completely wrong. I think the author doesn’t know what he’s talking about. I will go point by point:

                                                                                                • Out-of-tree and will never be mainlined: Please don’t use zfs-fuse, it’s long been abandoned and OpenZFS is better in every respect. The rest of the points I guess are true (or might be, eventually, sure).
                                                                                                • Slow performance of encryption: This seems to be completely wrong. I believe OpenZFS re-enabled vector instructions with their own implementation of the kernel code that can no longer be used. For an example, see https://github.com/openzfs/zfs/pull/9749 which was merged many months after the Linux kernel disabled vector instructions.
                                                                                                • Rigid: This was done deliberately so people like the author don’t shoot themselves in the foot. It would actually have been easier to make the vdev hierarchy more flexible, but ZFS is more strict on purpose, so users don’t end up with bad pool configurations.
                                                                                                • Can’t add/remove disks to RAID: I guess this is still true? I’m not entirely sure because I’m not following OpenZFS development closely nor do I use RAID-Z.
                                                                                                • RAID-Z is slow: As far as I know this is correct (in terms of IOPS), so RAID-Z pools are more appropriate for sequential I/O rather than random I/O.
                                                                                                • File-based RAID is slow: OpenZFS can now do scrubs and resilvers (mostly) sequentially, so this point is wrong now.
                                                                                                • Real-world performance is slow: I wouldn’t call it slow, but ZFS can be slower than ext4, sure (but it’s also doing a lot more than ext4, on purpose, such as checksumming, copy-on-write, etc).
                                                                                                • Performance degrades faster with low free space: The free-space bitmap comment is just weird/wrong, because ZFS actually has more scalable data structures for this than most other filesystems (such as ext4). It might be true that ZFS fragments more around 80% utilization than ext4, but this is probably just a side-effect of copy-on-write. Either way, no filesystem will handle mostly full disks very well in terms of fragmentation, so this is not something specific to ZFS, it’s just how they (have to) work.
                                                                                                • Layering violation of volume management: This is completely wrong. You can use other filesystems on top of a ZFS pool (using ZVols) and you can use ZFS on top of another volume manager if you want (but I wouldn’t recommend it), or even mix it with other filesystems on the same disk (each on their own partition). Also, you can set a ZFS dataset/filesystem mountpoint property to legacy and then use normal mount/umount commands if you don’t like ZFS’s automounting functionality.
                                                                                                • Doesn’t support reflink: This is correct.
                                                                                                • High memory requirements for dedupe: The deduplication table is actually not kept in memory (except that a DDT block is cached whenever it’s read from disk, as any other metadata). So as an example, if you have some data that is read-only (or mostly read-only) you can store it deduped and (apart from the initial copy) it will not be any slower than reading any other data (although modification or removal of this data will be slower if ZFS has to keep reading DDT blocks from disk due to them being evicted from cache).
                                                                                                • Dedupe is synchronous: Sure it’s synchronous, but IOPS amplification will mostly be observed only if the DDT can’t be cached effectively.
                                                                                                • High memory requirements for ARC: I don’t even know where to begin. First of all, the high memory requirements for the ARC have been debunked numerous times. Second, it’s normal for the ARC to use 17 GiB of memory if the memory is available otherwise – this is what caches (such as the ARC) are for! The ARC will shrink whenever memory is otherwise needed by applications or the rest of the kernel, if needed. Third, I use OpenZFS on all my machines, none of them are exclusively ZFS hosts, and there is exactly zero infighting in all of them. Fourth, again, please just ignore zfs-fuse, there is no reason to even consider using it in 2022.
                                                                                                • Buggy: All filesystems have bugs, that’s just a consequence of how complicated they are. That said, knowing what I know about the ZFS design, code and testing procedures (which is a lot, although my knowledge is surely a bit outdated), I would trust ZFS with my data above any other filesystem, bar none.
                                                                                                • No disk checking tool: This is actually a design decision. Once filesystems get too large, fsck doesn’t scale anymore (and it implies downtime, almost always), so the decision was made to gracefully handle minor corruption while the machine is running and being used normally. Note that a badly corrupted filesystem will of course panic, as it likely wouldn’t even be possible to recover it anymore, so it’s better to just restore from backups. But you can also mount the ZFS pool read-only to recover any still-accessible data, even going back in time if necessary!

                                                                                                In conclusion, IMHO this article is mostly just FUD.

                                                                                                1. 21

                                                                                                  This is actually a design decision.

                                                                                                  A question on my mind while reading this was whether or not the author knows ZFS enough to be making some of these criticisms honestly. They seem like they should, or could. I am not attacking their intelligence, however I would prefer to see a steelman argument that acknowledges the actual reasons for ZFS design choices. Several of the criticisms are valid, but on the topic of fsck, ARC and layering the complaints appear misguided.

                                                                                                  I spent 7 years using the solution they recommend (LUKS+btrfs+LVM) and have been moving to ZFS on all new machines. I’ll make that a separate top-level comment, but I wanted to chime in agreeing with you on the tone of the article.

                                                                                                  1. 7

                                                                                                    I’m not sure the check tool is really not needed. It’s not something I want to run on mount / periodically. I want a “recovery of last resort” offline tool instead and it doesn’t have to scale because it’s only used when things are down anyway. If there’s enough use case to charge for this (https://www.klennet.com/zfs-recovery/default.aspx) there’s enough to provide it by default.

                                                                                                    1. 4

                                                                                                      In general we try to build consistency checking and repair into the main file system code when we can; i.e., when doing so isn’t likely to make things worse under some conditions.

                                                                                                      It sounds like what you’re after is a last ditch data recovery tool, and that somewhat exists in zdb. It requires experience and understanding to hold it correctly but it does let you lift individual bits of data out of the pool. This is laborious, and complicated, and likely not possible to fully automate – which is why I would imagine many folks would prefer to pay someone to try to recover data after a catastrophe.

                                                                                                    2. 5

                                                                                                      Dedup does generally have high memory requirements if you want decent performance on writes and deletes; this is a famous dedup limitation that makes it not feasible in many situations. If the DDT can’t all be in memory, you’re doing additional random IO on every write and delete in order to pull in and check the relevant section of the DDT, and there’s no locality in these checks because you’re looking at randomly distributed hash values. This limitation isn’t unique to ZFS, it’s intrinsic in any similar dedup scheme.

                                                                                                      A certain amount of ZFS’s nominal performance issues are because ZFS does more random IOs (and from more drives) than other filesystems do. A lot of the stories about these performance issues date from the days when hard drives were dominant, with their very low IOPS figures. I don’t think anyone has done real performance studies in these days of SSDs and especially NVMe drives, but naively I would expect the relative ZFS performance to be much better these days since random IO no longer hurts so much.

                                                                                                      (At work, we have run multiple generations of ZFS fileservers, first with Solaris and Illumos on mostly hard drives and now with ZoL on Linux on SATA SSDs. A number of the performance characteristics that we care about have definitely changed with the move to SSDs, so that some things that weren’t feasible on HDs are now perfectly okay.)

                                                                                                    1. 20

                                                                                                      Thermonuclear take: Why use a “TUI”? You’re crudely imitating a real GUI with the crippling limitations of a vt220, when you’re in an environment that can almost certainly handle a real GUI.

                                                                                                      1. 17

                                                                                                        The biggest reasons for me:

                                                                                                        • Low resource usage
                                                                                                        • I can run it on a different machine and SSH to it (e.g. IRC bouncer)

                                                                                                        (And for a combination of those two: I can run it on a low-powered machine like my home raspberry pi server…)

                                                                                                        1. 7

                                                                                                          I’ve found that the richest TUIs are often very CPU heavy. Redrawing TUIs seems much more expensive than updating GUIs. It’s not very surprising since they’re not really meant for selective updates unlike current graphical technologies.

                                                                                                        2. 5

                                                                                                          Terminals are an excellent example of a mixed bag. There’s a lot about terminals that is not great, perhaps most importantly the use of inband signalling for control sequences. That said, they’re also a testament to what we can achieve when we avoid constantly reinventing everything all the time.

                                                                                                          There are absolutely limitations in the medium, but the limitations aren’t crippling or nobody would be getting anything done with terminal-based software. This is clearly just not true; people use a lot of terminal-based software to great effect all the time. Unlike most GUI frameworks, one even has a reasonable chance of building a piece of software that works the same way on lots of different platforms and over low-bandwidth or high-latency remote links.

                                                                                                          1. 9

                                                                                                            How are modern terminals not a case of reinventing? They’ve taken the old-school VT100 with its escape sequences and bolted on colors (several times), bold/italic/wide characters, mouse support, and so on. All of this in parallel with the development of GUIs, and mostly while running on top of an actual GUI.

                                                                                                            I’m not denying there’s a benefit to having richer I/O in a CLI process where you’re in a terminal anyway, but a lot of the fad for TUI apps (Spotify? Really?) seems to me like hairshirt computing and retro fetishization.

                                                                                                            If you’d asked 1986 me, sitting at my VT220 on a serial line to a VAX, whether I’d rather have a color terminal with a mouse or a Mac/Linux/Windows GUI desktop, I’d have thought you were crazy for even offering the first one.

                                                                                                            1. 5

                                                                                                              How are modern terminals not a case of reinventing? They’ve taken the old-school VT100 with its escape sequences and bolted on colors (several times), bold/italic/wide characters, mouse support, and so on. All of this in parallel with the development of GUIs, and mostly while running on top of an actual GUI.

                                                                                                              I would not consider it reinventing because in many cases, at least when done well, you can still use these modern applications on an actual VT220. Obviously that hardware doesn’t provide mouse input, and is a monochrome-only device; but the control sequences for each successive new wave of colour support have generally been crafted to be correctly ignored by earlier or less capable terminals and emulators. Again, it’s not perfect, but it’s nonetheless an impressive display of stable designs: backwards compatibility and long-term incremental improvement with tangible results for users.

                                                                                                              I’m not denying there’s a benefit to having richer I/O in a CLI process where you’re in a terminal anyway, but a lot of the fad for TUI apps (Spotify? Really?) seems to me like hairshirt computing and retro fetishization.

                                                                                                              I’m not sure what you mean by “hairshirt” but it certainly sounds like intentionally loaded, pejorative language. I have been using the desktop Spotify application for a while, and it uses a lot of resources to be impressively sluggish and unreliable. I expect a terminal-based client would probably feel snappy and meet my needs. Certainly Weechat does a lot better for me than the graphical Slack or Element clients do.

                                                                                                              I’m not going to make you use any of this software, but I would suggest that even if it is only a “fad”, who cares? If it makes people happy, and it hurts nobody, then people should probably just do it. Both graphical bitmap displays and character-cell terminals have been around for a long time; they both have pros and cons, and I don’t expect one size will ever fit all users or applications.

                                                                                                          2. 4

                                                                                                            That’s a very good question, honestly.

                                                                                                            However, I haven’t seen any kind of graphical application (like, using the whole set of features gotten from full access to visual display) yet still being competely usable from keyboard only. Except Emacs, which is a very nice example, but I intentionally wanted to avoid any kind of text editors in this discussion.

                                                                                                            After all, if I even stumble upon some sort of UI framework including full keyboard operation in REPL-style maner + shortcuts, showing various graphical data types (interactive tables, charts, data frames, scrollable windows, etc.) I’ll definitely test it thoroughly as long as it’s able to be shipped onto customers’ desktops (so yeah, Arcan is a suggestion, but not really fitting in current model of application deployment).

                                                                                                            1. 6

                                                                                                              Most GUI toolkits can be operated by keyboard? Windows was designed to be usable without a mouse, for instance.

                                                                                                              I do note that GUI vs. CLI (and other things like mouse/keyboard dependency) isn’t a dichotomy. See: CLIM.

                                                                                                              1. 1

                                                                                                                A couple examples of the top of my head (though I’m not trying to make the case that all GUI apps can be driven this way, and there are tons of terrible GUI apps out there) that do offer full keyboard operation:

                                                                                                                • IntelliJ IDEs
                                                                                                                • Pan newsreader
                                                                                                                • Nautilus file browser
                                                                                                                • Evince PDF reader
                                                                                                                • KeePassX

                                                                                                                Those are just some apps I regularly use with no mouse usage at all.

                                                                                                                1. 1

                                                                                                                  Most well-implemented Mac apps can be used keyboard-only, thanks to Apple’s accessibility features and lesser-known keyboard shortcuts like those for activating the menu bar.

                                                                                                                2. 3

                                                                                                                  I think it’s for the same reason people write web GUI even if native GUI is generally superior.

                                                                                                                  1. 2

                                                                                                                    If I really mess up my Linux computer and I can’t get my window manager / X11 / Wayland to run, I can still get stuff done in TUIs while I attempt to fix it.

                                                                                                                    Also, while others point out low resource usage, I’ll specifically mention lack of GPU acceleration as a situation where I’d rather use a TUI. For example, software support for the GPU on my MNT Reform is spotty, which means some GUIs are painfully slow (e.g. Firefox crawls because WebRender doesn’t support the GPU), but there’s no noticeable difference in my terminal emulator.

                                                                                                                    1. 1

                                                                                                                      I currently do all my work sshed into my desktop in the office (combination of wfh and work’s security policies which mean I can’t access the code on my laptop). TUIs are great for that.

                                                                                                                      1. 1

                                                                                                                        Because TUI might be esoteric enough to avoid the attentions that might lead it into the same CADT that X11 got?

                                                                                                                        1. 4

                                                                                                                          Unix retrofetishists love TUI stuff, so no.

                                                                                                                          Besides, ncurses and the VT isn’t much better than X API-wise, anyways.

                                                                                                                          1. 1

                                                                                                                            Guess that’s true. Still want to shake my stick at them until they get off the grass.

                                                                                                                      1. 3

                                                                                                                        Some observations from buying a stupidly large rackmount case and using it to house:

                                                                                                                        • several Raspberry Pis (OpenVPN, Home Assistant, Pi-Hole)
                                                                                                                        • an X200 games ‘server’
                                                                                                                        • a switch
                                                                                                                        • my NBN modem
                                                                                                                        • a consumer class router
                                                                                                                        • a TP-Link mesh network base
                                                                                                                        • a Pi Zero with LED status lights for each server + external Internet
                                                                                                                        • a refurbished DL320 G6 ProLiant as a 4TiB NAS

                                                                                                                        It looks fantastic :) Especially since I “connected it to the cloud” ;)

                                                                                                                        https://i.postimg.cc/C5N8GG5k/the-cloud.jpg

                                                                                                                        https://vimeo.com/manage/videos/577062093

                                                                                                                        Having everything in the one enclosure is great during power outages (not uncommon where we are). Makes it easy to hook up to a generator. Also great for cleaning and tidying.

                                                                                                                        It’s neat to play around with actual server hardware. First time I’ve ever done it and the Linux-based monitoring software is neat too. Plus fast hardware RAID is nice for its purpose as a NAS - I have 2 x 500GiB HDDs for the OS, and 2 x 4TiB HDDs for the NAS itself.

                                                                                                                        It was cheap. Only cost around AUD250 plus shipping! Although our power bill has increased 27% year on year. I’m not sure how much of that was due to WFH during lockdown, though … we’ll see over the course of the coming year.

                                                                                                                        I’ve never used a KVM either, and it’s neat having a fold-out terminal built into the server cabinet (and yes that’s a Apple MacPro keyboard, circa 2007. I was missing a few cables for the KVM at the time …).

                                                                                                                        https://vimeo.com/579389530

                                                                                                                        Finally, @1amzave is dead right about 1U servers being noisy (just listen to that video above!). That ProLiant sounds like a 747 on takeoff during POST and boot. Fortunately my bedroom is downstairs, and the kids are used to the house sounding like a data centre anyhow.

                                                                                                                        1. 2

                                                                                                                          Are the LEDs on a small mezzanine board on top of the Pi Zero? What board is that? That’s pretty cool!

                                                                                                                          1. 2

                                                                                                                            Yeah - it’s a Blinkt board, originally purchased as a build light. They’re run by a super low-fi Python script that keeps an eye on things:

                                                                                                                            #!/usr/bin/env python3
                                                                                                                            
                                                                                                                            from blinkt import set_pixel, set_brightness, show
                                                                                                                            import os
                                                                                                                            import time
                                                                                                                            
                                                                                                                            servers = [
                                                                                                                                # These are sorted to allow easy identification.
                                                                                                                                "blinkenlights.local",
                                                                                                                                "gaming.local",
                                                                                                                                "homados.local",
                                                                                                                                "pihole.local",
                                                                                                                                "vpn.local",
                                                                                                                            
                                                                                                                                # A hacky public Internet access indicator comes last.
                                                                                                                                "tpg.com.au"
                                                                                                                            ]
                                                                                                                            
                                                                                                                            while True:
                                                                                                                                for i in range(8):
                                                                                                                                    if (i >= len(servers)):
                                                                                                                                        set_pixel(i, 0, 0, 0)
                                                                                                                                    else:
                                                                                                                                        set_pixel(i, 0, 0, 3)
                                                                                                                                        show()
                                                                                                                                        result = os.system("ping -c 1 " + servers[i])
                                                                                                                                        if result == 0:
                                                                                                                                            set_pixel(i, 0, 1, 0)
                                                                                                                                        else:
                                                                                                                                            set_pixel(i, 10, 0, 0)
                                                                                                                                show()
                                                                                                                                time.sleep(60)
                                                                                                                            

                                                                                                                            The kids learned, during lockdown, to go and take a look at the cabinet lights if their Zoom or multiplayer game dropped :)

                                                                                                                            There’s a nice effect in there, which is that lights are off if unused, red if down, green if up, and blue while being pinged, so you get a nice ripple of blue across the green (or red!) every minute.

                                                                                                                            Relevant to this thread, particularly, is that my wife named the DL320 “Homados”, which is an ancient God of battle-noise. That thing is not quiet :)

                                                                                                                        1. 6

                                                                                                                          If I understand this correctly, the flow is:

                                                                                                                          1. Call sfork, now you are in the parent process but with the child’s view of kernel state (file descriptor tables and so on).
                                                                                                                          2. Open / close files, do whatever setup you need.
                                                                                                                          3. Call sfork_execve, which doesn’t execve in the parent but instead starts the binary in the child and resets the parent’s view of kernel state to the parent’s old state.

                                                                                                                          There are some really nice things here. In particular, it avoids the memory-management headaches of vfork. Anything done between sfork and sfork_execve is still visible in the parent after the sfork_execve call, which means it’s easy to free the state.

                                                                                                                          The thing that I don’t like is that it still requires copying the parent FD table (which imposes some synchronisation and memory-management overhead in the kernel, potentially a lot in a large server process). I’d prefer that the child started with an empty FD table but added an sfork_dup2 where the source was in the parent’s FD namespace and the target was in the child’s, so you could do something like:

                                                                                                                          int some_privileged_file = open(...);
                                                                                                                          sfork();
                                                                                                                          // Inherit stdin / out / errno
                                                                                                                          sfork_dup(STDIN_FILENO, STDIN_FILENO);
                                                                                                                          sfork_dup(STDOUT_FILENO, STDOUT_FILENO);
                                                                                                                          sfork_dup(STDERR_FILENO, STDERR_FILENO);
                                                                                                                          sfork_dup(PrivilegedFDNumber, some_privileged_file);
                                                                                                                          // Enter jail
                                                                                                                          jail_attach(...);
                                                                                                                          // Drop root privilege
                                                                                                                          setuid(...);
                                                                                                                          // Open some more files with reduced privileges, inside the jail
                                                                                                                          open(...);
                                                                                                                          open(...);
                                                                                                                          sfork_execve(...);
                                                                                                                          // Free any memory you allocated here.
                                                                                                                          free(...);
                                                                                                                          close(some_privileged_file);
                                                                                                                          

                                                                                                                          The sfork_execveat is slightly weird. I don’t know why you’d want that instead of an analogue of fexecve. It means that you can’t execve a binary that the child doesn’t have access to by the time that you get to the sfork_execve. Passing the file descriptor to the binary is simpler and if you open it with the close-on-exec flag then it doesn’t end up in the FD table of the child.

                                                                                                                          1. 1

                                                                                                                            The thing that I don’t like is that it still requires copying the parent FD table (which imposes some synchronisation and memory-management overhead in the kernel, potentially a lot in a large server process). I’d prefer that the child started with an empty FD table but added an sfork_dup2 where the source was in the parent’s FD namespace and the target was in the child’s, so you could do something like:

                                                                                                                            Linux’s close_range(2) removes the need for this to some degree - by closing most fds at the same time as creating the new FD table with CLOSE_RANGE_UNSHARE, you remove most of the synchronization needed. pidfd_getfd(2) also helps here.

                                                                                                                            That is, instead of copying the fd table at sfork time, you’d run sfork(CLONE_FILES) so the new process shares the parent’s fd table, then you’d unshare/copy it explicitly by calling close_range(CLOSE_RANGE_UNSHARE).

                                                                                                                            The sfork_execveat is slightly weird. I don’t know why you’d want that instead of an analogue of fexecve. It means that you can’t execve a binary that the child doesn’t have access to by the time that you get to the sfork_execve.

                                                                                                                            sfork_execveat is just exactly Linux execveat(2) wrapped so it can return. Like the unwrapped version of execveat, it supports fexec by using AT_EMPTY_PATH.

                                                                                                                            Tangentially, see “Bugs” in the Linux execveat manpage for an unfortunate interaction between fexec, CLOEXEC on the executable fd, and shebang files, which means you can’t safely set CLOEXEC. I don’t know if this also applies on FreeBSD.

                                                                                                                            1. 2

                                                                                                                              You can safely set CLOEXEC if you know that the file is not interpreted (which is the case for a lot of my uses, though not in the general case). That said, it’s usually fine to just leave an execute-only file descriptor open in the child - the kernel will keep a reference to the file on disk open for the lifetime of the program anyway so you’re using a tiny bit of extra kernel state but not really impacting anything in the child.

                                                                                                                              I’m quite tempted to do a FreeBSD implementation of a variant of sfork with the sfork_dup2 variant that I proposed above.

                                                                                                                              1. 1

                                                                                                                                I’m quite tempted to do a FreeBSD implementation of a variant of sfork with the sfork_dup2 variant that I proposed above.

                                                                                                                                If you do I’d be very interested in hearing about it. As you might perceive, I spend a lot of time thinking about better process creation and management mechanisms for Unix.

                                                                                                                                If you do make sfork_dup2, I’d recommend making it a bit more generic so it can be used outside of the sfork context. You could imagine that it works to copy file descriptors from your parent, iff you’re running in the same uid/security context as your parent. Or perhaps go all the way and make a pidfd_getfd equivalent. Either way it would then be useful for rsyscall-style approaches where you create an essentially-empty child process and then populate it from the parent by remote-calling sfork_dup2/pidfd_getfd/etc inside the child.

                                                                                                                                Note that the nature of sfork prevents you from using SCM_RIGHTS to implement sfork_dup2, otherwise that would be a tempting option.

                                                                                                                                For completeness, not that you mentioned this: The other class of alternative is a system call which takes a list of file descriptors and copies them all at once into a new fd table. In some sense this is what close_range(CLOSE_RANGE_UNSHARE) does. I’ve explored this option in rsyscall but found it doesn’t work as well, because it’s anti-modular and has many of the same downsides of posix_spawn.

                                                                                                                                1. 3

                                                                                                                                  If you do I’d be very interested in hearing about it. As you might perceive, I spend a lot of time thinking about better process creation and management mechanisms for Unix.

                                                                                                                                  There are some folks thinking about this in FreeBSD because of the colocated process work for CHERI (which allows multiple OS processes in the same address space, with CHERI providing the isolation). This is currently done with vfork + coexecve and vfork works surprisingly well but isn’t quite ideal.

                                                                                                                                  Between CHERI, jails, and Capsicum, there are a lot of things that require flexibility from process creation. I think a FreeBSD-flavoured version of sfork would mirror pdfork for the initial call, would provide an empty FD table, add an sfork_dup2 to explicitly inherit some file descriptors, and would then use an sfork_fexecve mirroring fexecve to create the process.

                                                                                                                                  A userland wrapper could implement your sfork_execveat by doing an openat of the file, reading the first two bytes, and setting the close-on-exec flag if they are not ‘#!’, then doing sfork_fexecve, but the kernel primitive could also execute files in directories that the child doesn’t have access to.

                                                                                                                                  Similarly, a userspace wrapper could implement your sfork behaviour with:

                                                                                                                                  int sfork(void)
                                                                                                                                  {
                                                                                                                                    int child_fd = pdsfork(PD_DAEMON);
                                                                                                                                    if (child_fd < 0)
                                                                                                                                    {
                                                                                                                                      return -1;
                                                                                                                                    }
                                                                                                                                    int ret = pdgetpid(child_fd);
                                                                                                                                    close(child_fd);
                                                                                                                                    return ret;
                                                                                                                                  }
                                                                                                                                  

                                                                                                                                  So the only four syscalls that you’d need are:

                                                                                                                                  • pdsfork
                                                                                                                                  • sfork_exit
                                                                                                                                  • sfork_dup2
                                                                                                                                  • sfork_fexecve

                                                                                                                                  With sfork and sfork_execveat implemented in userspace if anyone wants them (anything involving PIDs is inherently racy, so I’d much prefer people used the process descriptor versions, which aren’t).

                                                                                                                                  If you do make sfork_dup2, I’d recommend making it a bit more generic so it can be used outside of the sfork context.

                                                                                                                                  This has a lot of subtlety to get right. You definitely wouldn’t want a capability-mode process to be able to do that (and, in my ideal world, all processes end up being capability-mode processes) and it also interacts poorly with any other security feature. You’d need to check for any change in any security context, which seems fragile. In contrast, in the sfork context the only code that can run is code that is explicitly authorised by the parent process.

                                                                                                                                  You could implement something similar to pidfd_getfd by adding a permission on process descriptors that allows it but it’s an awful API because it’s inherently racy: the source process can modify its FD table concurrently with the target’s pdifd_getfd call and it doesn’t know that the target is doing this call so doesn’t have any inherent mechanism to synchronise with.

                                                                                                                                  The Windows equivalent DuplicateHandle is more symmetrical (you need the equivalent of a process descriptor for both processes and you must have the rights to modify the equivalent of the FD table in both) but it is almost always used in the push model, because the sender can guarantee that the handle is still valid.

                                                                                                                                  Restricting it to the sfork context is also nice because it avoids all of these concurrency problems (or, at least, means that they are limited to the same set as in any UNIX program): unless another thread closes and opens file descriptors in the parent, they will not be changed and the parent process knows that it should avoid concurrent modification of any of the file descriptors that are referenced in the sfork context (you can always write buggy code, but at least in this specific use case there is a mechanism for writing correct code). This may also let you avoid some locking in the kernel because you can’t create new threads in the sfork context and so you can skip any locking while modifying the child’s FD table.

                                                                                                                                  For completeness, not that you mentioned this: The other class of alternative is a system call which takes a list of file descriptors and copies them all at once into a new fd table.

                                                                                                                                  This is what I was originally considering. Actually a list of index-fd pairs, so that you can specify gaps in the FD table. This has the phase ordering problem that you mention though - it’s fine if you want to populate those bits of the child’s FD table and then open things in the child context, it doesn’t work if you need to open some things in the child and then use some things from the parent. I’m not sure that this is actually a required pattern but the sfork_dup2 mechanism is strictly more expressive since you could ask another thread to open files for you after you’ve dropped privileges in the sfork thread (again, probably not something you’d actually want to do, but it is possible).

                                                                                                                                  In some sense this is what close_range(CLOSE_RANGE_UNSHARE) does

                                                                                                                                  I don’t know what that does. FreeBSD has a close_range system call, but not a CLOSE_RANGE_UNSHARE flag, if Linux has a close_range then it’s not documented.

                                                                                                                                  1. 1

                                                                                                                                    So the only four syscalls that you’d need are:

                                                                                                                                    pdsfork sfork_exit sfork_dup2 sfork_fexecve

                                                                                                                                    Yes, that makes sense to me. I only return pids because I wrote sfork before pidfds were merged, and I only use execveat because that’s the most powerful option on Linux, but I certainly prefer a capability approach instead.

                                                                                                                                    This has a lot of subtlety to get right.

                                                                                                                                    I agree with all this. I haven’t yet thought of a clean way to approach it that is also generic to many use cases. Piggy-backing on SCM_RIGHTS is a nice approach for rsyscall (where the two processes execute concurrently), maybe there’s an SCM_RIGHTS-based approach that works for sfork?

                                                                                                                                    I don’t know what that does. FreeBSD has a close_range system call, but not a CLOSE_RANGE_UNSHARE flag, if Linux has a close_range then it’s not documented.

                                                                                                                                    I linked the manpages in my original comment (you may not have seen them because I edited it in): https://lobste.rs/s/vzavsz/sfork_synchronous_single_threaded#c_vbldck

                                                                                                                                    But, here’s the Linux close_range(2) manpage, complete with CLOSE_RANGE_UNSHARE: https://man7.org/linux/man-pages/man2/close_range.2.html

                                                                                                                                2. 1

                                                                                                                                  Oh, I should also say: Even if you use an sfork_dup2 variant you should still automatically copy all fds which are not marked CLOEXEC. That way you preserve the traditional Unix implicit-fd-inheritance feature, which is desirable for more than just stdout/in/err. You could imagine using it for a root filesystem or cwd fd, or to inherit the capability to perform some operation that is configured by an also-implicitly-inherited environment variable.

                                                                                                                                  In programming languages, this is like “implicit parameters”, which are a useful feature. In Unix, the same is true: It’s a useful feature, not a bug.

                                                                                                                                  The design bug is just that implicit-fd-inheritance is on by default, which is very annoying; it should be that CLOEXEC is the default and you opt in to implicit-fd-inheritance. But that’s no reason to remove the feature entirely.

                                                                                                                                  1. 1

                                                                                                                                    I strongly disagree on this: the first thing that I always end up doing in vfork is calling closefrom to ensure that I haven’t accidentally inherited any file descriptors. The UNIX behaviour was fine in single-threaded UNIX programs, but it’s inherently racy in multithreaded programs because any other thread may have called library functions that open files and then close them and don’t bother setting O_CLOEXEC because they are closing the file descriptor in a local scope.

                                                                                                                                    If you rely on implicitly inheriting file descriptors then you rely on every library routine that every other thread in your program calls correctly setting O_CLOEXEC on every file descriptor. You also rely on some kernel APIs that don’t exist. For example (as far as I can tell) there is no socket or socketpair analogue (even in Linux, which has done a pretty good job of adding these) that provides a CLOEXEC flag, so the other thread needs to call socket and then fcntl(, F_SETFD) and hope that you didn’t do the [s,pd,v]fork call in between the socket call and the fcntl call.

                                                                                                                                    It also violates the principle of intentionality. You should never exercise rights without explicitly choosing to. Inheriting a file descriptor without explicitly choosing to is passing a set of rights to the child process without meaning to, which is a non-intentional exercise of privilege. There have been a lot of security vulnerabilities caused by this mechanism, which is what closefrom exists and why calling it before execve is something that you’ll find in any secure programming guide for *NIX.

                                                                                                                                    1. 2

                                                                                                                                      but it’s inherently racy in multithreaded programs because any other thread may have called library functions that open files and then close them and don’t bother setting O_CLOEXEC because they are closing the file descriptor in a local scope.

                                                                                                                                      If you rely on implicitly inheriting file descriptors then you rely on every library routine that every other thread in your program calls correctly setting O_CLOEXEC on every file descriptor.

                                                                                                                                      Absolutely, so that’s why CLOEXEC should be the default.

                                                                                                                                      IMO, there should be a PR_DEFAULT_TO_CLOEXEC prctl that one can set, so that all new file descriptors are created with CLOEXEC by default. Then processes could opt-in to the sane behavior and not have to worry about libraries calling functions without passing CLOEXEC.

                                                                                                                                      For example (as far as I can tell) there is no socket or socketpair analogue (even in Linux, which has done a pretty good job of adding these) that provides a CLOEXEC flag,

                                                                                                                                      SOCK_CLOEXEC works for socket and socketpair, at least on Linux. I believe Linux has essentially complete coverage for CLOEXEC-everywhere.

                                                                                                                                      It also violates the principle of intentionality. You should never exercise rights without explicitly choosing to. Inheriting a file descriptor without explicitly choosing to is passing a set of rights to the child process without meaning to, which is a non-intentional exercise of privilege.

                                                                                                                                      Before I give my take, let me first establish that I’m hard-core in favor of explicit capability-security. I want to go as far as possible in removing any kind of implicit authority. I’ve worked hard on that in rsyscall, and I think rsyscall is a good demonstration of how explicit over implicit gives you really amazing powers and how it’s really worth it to be explicit.

                                                                                                                                      That being said, it’s precisely my hard-core belief in explicit-passing-of-capabilities that makes me interested in the places where implicit-passing is useful. Environment variables and exception handlers are two widespread examples of implicit-passing/dynamic-scope. Many people find those very useful.

                                                                                                                                      Note that implicit inheritance is already widespread in Unix: the root directory, CWD, lots of other things that are implicitly copied by a fork. In some sense, the kernel and host you’re running on is something you implicitly inherit, and it’s hard to get rid of that (rsyscall is my attempt). I emphatically agree that we should make these all explicit.

                                                                                                                                      But, it’s sometimes useful to have things be implicit, and why shouldn’t user programs be able to make their own implicitly-inherited capabilities, if they explicitly choose to make something implicit? They can implicitly inherit data with environment variables; why not capabilities? Why should only the OS designers be allowed to implicitly inherit capabilities?

                                                                                                                                      1. 1

                                                                                                                                        IMO, there should be a PR_DEFAULT_TO_CLOEXEC prctl that one can set, so that all new file descriptors are created with CLOEXEC by default. Then processes could opt-in to the sane behavior and not have to worry about libraries calling functions without passing CLOEXEC.

                                                                                                                                        I don’t disagree in principle, but I think it’s a very difficult thing to retrofit to the ecosystem, for a couple of reasons. First, you’d need to either add an NOCLOEXEC flag to everything that currently has a CLOEXEC flag and that space is quite constrained. Not insurmountable though, especially on Linux where they’re now favouring system calls that take a pointer to an extensible structure as the parameter and ignoring the fact that this completely breaks seccomp-bpf’s ability to do anything useful.

                                                                                                                                        Second, it affects the behaviour of all libraries, so any library that actually wants the inherited behaviour would need to be modified. My guess is that this is approximately zero libraries, so maybe it’s not a problem.

                                                                                                                                        SOCK_CLOEXEC works for socket and socketpair, at least on Linux. I believe Linux has essentially complete coverage for CLOEXEC-everywhere.

                                                                                                                                        Thanks, I missed that in the man page. It’s also there for FreeBSD.

                                                                                                                                        That being said, it’s precisely my hard-core belief in explicit-passing-of-capabilities that makes me interested in the places where implicit-passing is useful. Environment variables and exception handlers are two widespread examples of implicit-passing/dynamic-scope. Many people find those very useful.

                                                                                                                                        Environment variables aren’t implicit. They are an explicit parameter to execve and there is no default of ‘pass all of my argument variables to the child’. This is easy to do (just pass environ to that parameter) but the important thing is that, at the point of use, there is an explicit policy decision on what to pass. Shells, in general, pass everything. Things like ssh and sudo provide an allow list and pass any of the things that are set. Some things provide a block list and pass anything that isn’t on that list.

                                                                                                                                        Exceptions come in many forms but in general unchecked exceptions (i.e. ones where the type system doesn’t define the set of exceptions that can be thrown) are a massive source of bugs (and security vulnerabilities). C++ has only unchecked exceptions and every large-scale C++ project that I’ve worked on has had a policy of ‘don’t use exceptions’ (often compile with -fno-exceptions to enforce this). Java has a mix, everything is a checked exception except that subclasses of RuntimeException are unchecked (any object access can throw a NullPointerException, and there’s nothing in the type system that lets you prove to the compiler that your function can’t throw these. Similarly, any allocation can cause an OutOfMemoryException). The unchecked exceptions in Java have been a source of a large number of bugs and security vulnerabilities.

                                                                                                                                        Note that implicit inheritance is already widespread in Unix: the root directory, CWD, lots of other things that are implicitly copied by a fork. In some sense, the kernel and host you’re running on is something you implicitly inherit, and it’s hard to get rid of that (rsyscall is my attempt). I emphatically agree that we should make these all explicit.

                                                                                                                                        The root directory is implicit, but in a capability world you don’t have access to the namespace that contains it. CWD is one of my pet hates in *NIX. It should not be something that the kernel provides, there’s no need for it (especially with openat and friends), userspace APIs could implement it for programs that want it. Again, both of these have been sources of vulnerabilities. For example, chroot is restricted to root because if you could chroot and run a setuid binary then you could trick it into modifying the wrong bit of the filesystem. Similarly, there have used to be vulnerabilities from running some setuid daemons that then wrote to their current directory, when they expected to be run in a specific place.

                                                                                                                                        But, it’s sometimes useful to have things be implicit, and why shouldn’t user programs be able to make their own implicitly-inherited capabilities, if they explicitly choose to make something implicit? They can implicitly inherit data with environment variables; why not capabilities? Why should only the OS designers be allowed to implicitly inherit capabilities?

                                                                                                                                        Because someone has to reason about the security properties of the system. If I run a child process with a restricted set of rights, I can audit the code around process creation to be able to tell exactly what the child process can do. If rights can be implicitly inherited then I have to audit millions of lines of code to see what can be inherited.

                                                                                                                                        UNIX has a few bits of ambient authority, but I regard these as bugs not features.

                                                                                                                                        1. 1

                                                                                                                                          First, you’d need to either add an NOCLOEXEC flag to everything that currently has a CLOEXEC flag and that space is quite constrained

                                                                                                                                          Just on this point, I don’t think adding such flags would be necessary; unsetting CLOEXEC after the fact using the existing fcntl F_SETFD is sufficient for all correct usage.

                                                                                                                                          1. 1

                                                                                                                                            Yes, you’re right - you don’t have any atomicity requirements in this direction. Possibly the explicit F_SETFD would be sufficient - I doubt anyone is creating enough intentionally inherited file descriptors that an extra syscall would cost a measurable amount of performance.

                                                                                                                              2. 1

                                                                                                                                It isn’t really clear to me what the advantage is over posix_spawn(). As you note, you’re going to end up needing sfork-specific functions for arranging for things in the child – which is pretty much already how posix_spawn() works.

                                                                                                                                1. 3

                                                                                                                                  You don’t need sfork-specific functions for doing anything in the child that you can do with normal system calls. If you want to make posix_spawn as expressive as sfork or vfork then you need spawn operations for openning files, creating / binding sockets, along with setuid, jail_attach, cap_enter, and any other things that change credentials. With sfork you don’t need any of these. The only thing that you would need (if it follows my suggestion of starting with an empty file-descriptor table) is a mechanism for copying a file descriptor from the parent to the child.