1. 17
  1.  

  2. 8

    Even as a staunch advocate of C, I have a much simpler argument against this design from OpenBSD: languages which are not C should not depend on C. From a philisophical point of view, I think that a language design is much stronger when its compiler, runtime, stdlib, and so on - are all implemented in that language. If we constrain ourselves to language designs which can route syscalls through the C stdlib then we constrain our ability to innovate in new languages.

    1. 8

      language designs which can route syscalls through the C stdlib

      If your language design cannot call a function (with the platform’s standard ABI) from a shared library, something has gone terribly, horribly wrong. I haven’t seen any production language where FFI is impossible.

      If your language design just really really does not want to use syscall wrappers because it uses a completely alien calling convention and weird stacks and calling a function with platform ABI would incur performance overhead… well, I would argue something has gone wrong in this case too. I blame Linux monoculture — Linux is about the only significant unix-like where raw syscall numbers (called via raw CPU trap instructions) are the public API of the platform.

      So many advantages to defining syscalls as ELF symbols:

      • massively easier portability. Adding FreeBSD/aarch64 support to Go took three or four people, many many months of occasional work, tons of lines of code. Doing the same with Rust, D (LDC), OCaml, Crystal, Pony and others was trivial (in the syscall department — of course there were fun thread-local storage bugs and so on :D)
      • hooking via LD_PRELOAD, which is very useful for sandboxing existing programs (e.g. with libpreopen I can exec a gedit inside the incredibly strict Capsicum sandbox)
      • the platform can replace any syscall that actually goes through the kernel with a fast vDSO without breaking anything
      • with symbol versioning, old ABI can be kept around — sometimes as purely userspace wrappers around new ABI, and the kernel can be cleaned up
      • flexibility in what the platform actually even is at runtime…

      Sure, current unixes live with some legacy design decisions that actually tie everything closer to the C language — like the errno example — but this doesn’t mean you can’t design a good ABI with “C-ish” ELF symbols.

      1. 4

        You seem to be arguing with a strawman here - I never said I was opposed to a syscall wrapper library, though it has problems of its own. I’m arguing against blithely routing everything through libc.

        1. 1

          If your language design just really really does not want to use syscall wrappers because it uses a completely alien calling convention and weird stacks and calling a function with platform ABI would incur performance overhead…

          So resizable stacks are a bad thing?

          1. 1

            So this is a bit late as a comment. But not necessarily! The kernel doesn’t use your stack, it uses its own stack. So you really don’t very much stack space for these calls, and you could document and stabilise as part of the ABI the amount of required stack space needed to call these functions.

            1. 2

              So you really don’t very much stack space for these calls

              Yes you do.

              1. 1

                Any stack intensive functions can do a stack switch

                1. 1

                  That’s what Go does.

                  The problem is that this adds overhead, so in order for Go to remain competitive, it must avoid stack switches for hot functions like time().

                  1. 1

                    The problem is that the system calls and libc are designed around big C-style stacks, so they expect effectively infinite stack space to be available. If you designed a libc that was intended to be used from languages with unusual/little stack space available, I would imagine that it would be possible to design things you can provide an ABI guarantee of a maximum amount of stack usage per function in your API. I believe that in embedded systems programming it’s quite common to work out your maximum possible stack space, with compiler help, and then just manually allocate that much stack space for that thread at compile time.

                    On 64-bit systems with virtual memory, just overcommitting ~10MiB of stack for every thread and letting virtual memory handle growing it in 4KiB chunks seems like the best, simplest approach. That’s what I’ve done in my implementation of fibres. Virtual address space is rather plentiful.

                    1. 1

                      This, of course, means that every thread always uses at least 4 KB just for the stack. Golang’s default stack size on Linux was, at least at one point, 2KB.

                      1. 1

                        That’s a very tiny overhead for the simplicity and performance advantages of trivial C interop.

        2. 3

          For better or for worse, C is the Unix ABI. It’s not its contemporaries where calling conventions were either ill defined (mainframe stuff, PCs) or very language neutral (VMS, i).

          1. 2

            I think you’re looking at this from the wrong perspective. The SYSV ABI was standardized by SCO, and includes the ELF standard. It also happens to standardize the calling convention, and the way parameters about arguments (argc/argv) and the system (auxv) are passed to a program. It standardizes a lot more than that. Because of it being the System V Application Binary Interface, Linux and the BSDs chose to implement and follow that.

            For many reasons, the C standards committee chose the SYSV ABI as the calling convention for the C language.

            The C language ABI isn’t being forced on other languages, the C language chose the SYSV ABI as a base.

            Edit:

            everyone’s favorite case of errno, which you consult to get the error value from a failed system call (and from some failed library calls).

            Eh? Linux system calls are returned in rax, and I assume that’s the case for OpenBSD too?

            1. 4

              Linux system calls are returned in RAX IF you use the raw system call ABI. If you use libc, then you need to check errno.

              BSD system calls are returned in RAX:RFLAGS[OF], with OF acting as an error bit.

              1. 2

                The ABI is a different concern. I think that it’s reasonable to make an arbitrary programming language capable of speaking the C ABI. I am specifically talking about routing syscalls through libc.

              2. 0

                From my point of view, syscalls are another calling convention that occasionally leads to the C code in the kernel anyway. If platform (i.e. OpenBSD) developers want us to use cdecl instead of int 80h, then it’s fine as long as the ABI is specified.

                The main difference from the C calling convention here is that usually the kernel won’t mess with your stack. Although, I think it’s possible for libc to wrap all function calls with switch to some internal stack, and use some dynamic linker tricks or just defines in C headers to actually call wrappee in programs that use traditional C ABI.

                language design is much stronger when its compiler, runtime, stdlib, and so on - are all implemented in that language

                This, however, is not a barrier to implementing FFI in that language.

                1. 1

                  If platform (i.e. OpenBSD) developers want us to use cdecl instead of int 80h, then it’s fine as long as the ABI is specified.

                  I mostly agree. Is the ABI specified and stable though?

              3. 4

                …except that it can.

                #1, some languages (nim comes to mind) have automatic binding generators that parse c headers on the fly.

                #2, you can just put in the legwork to interface with all the libcs, like d.

                1. 2

                  But #1 is significantly complicated and #2 may break if headers contain implementation details that are subject to change, right?

                  Edit: Misread a bit your comment and the post.

                  1. 2

                    headers contain implementation details that are subject to change

                    Again, if macro or static/inline function foo in headers uses some internal function __bar, and suddenly someone decides remove that internal function, then the change breaks ABI compatibility and all your existing programs stop working because they actually use __bar, not the foo thingy.

                    1. 2

                      Yes. OpenBSD does not guarantee ABI compatibility.

                    2. 1

                      #1 is not that hard if you plug into libclang or whatever; #2 is a nonstarter because of ABI compatibility.

                    3. 1

                      2 strikes me as a really bad solution, you’re setting yourself up to breaking if any libc changes anything, you’re relying on internal implementation details, and you have a fixed set of libcs you support so D can’t work with systems which use any other libcs. 1 is interesting, but it’s a ton of (largely unnecessary) work.

                      I think my preferred solution would be to create a library in C which exports wrappers around open() and errno and whatever else you need, so the libc can export macros or functions or whatever else and your language’s libc bindings will just work still. However, that obviously has performance implications (every call to a libc function requires an extra function call), and doesn’t fix any of the other issues with limiting yourself to libc.

                      1. 1

                        breaking if any libc changes anything

                        They can’t do that, it’d break ABI compatibility.

                        fixed set of libcs you support so D can’t work with systems which use any other libcs

                        So what? If you want to work on a new platform, you have to port the compiler anyway. Porting the runtime to work with a new libc isn’t the hardest part of that.

                        1 is interesting, but it’s a ton of (largely unnecessary) work

                        It’s not that much work if you use libclang or something like that. It’s also generally a very useful thing to have, since you can communicate with arbitrary c libraries, not just libc.

                        1. 3

                          They can’t do that, it’d break ABI compatibility.

                          AFAIK OpenBSD isn’t that scared of breaking ABI compatibility?

                          So what? If you want to work on a new platform, you have to port the compiler anyway. Porting the runtime to work with a new libc isn’t the hardest part of that.

                          How do you define a “new platform”? Is Linux with musl a “different platform”? If I came out with a reimplemented libc for Linux tomorrow, and all code written for glibc was source compatible with my new libc, is that a “new platform”?

                          You may have a different opinion, but I think libc diversity is a good thing.

                          It’s not that much work if you use libclang or something like that. It’s also generally a very useful thing to have, since you can communicate with arbitrary c libraries, not just libc.

                          Sure, it’s a small enough amount of work that if you want painless C FFI, it might be worth it. However, some languages may not want or need painless C FFI; for example, for python, it might be enough to be able to dlopen a .so and call the symbols it exports. Other languages may not even have a way to express the stuff C functions do cleanly (i.e a pure functional language would maybe have trouble with how all C syscall wrappers set a global errno, and how would even a concept like calling C functions through an FFI make sense in prolog? I don’t doubt that it can be done, but it’s probably not exactly a language where you want to invest a lot into making FFI painless.)

                          it might in many circumstances not be that much more work, and it could possibly in some circumstances be useful for other stuff than calling libc functions, but I don’t think it’s wrong to say that it might be a lot of largely unnecessary work.

                          1. 1

                            How do you define a “new platform”? Is Linux with musl a “different platform”?

                            I’d argue that Linux distros should be considered as a distinct platforms. Most of them have common design choices, but some substantially diverge, e.g. NixOS, RancherOS, GoboLinux, Fedora Silverblue and Endless OS, etc. Another good example is Android (bionic libc).

                            If I came out with a reimplemented libc for Linux tomorrow, and all code written for glibc was source compatible with my new libc, is that a “new platform”?

                            That said, if someone creates a Linux distribution with your libc as the base, then it’s a new platform. Otherwise it’d probably be inconsistent with other components of the existing system.

                            1. 1

                              How do you define a “new platform”? Is Linux with musl a “different platform”? If I came out with a reimplemented libc for Linux tomorrow, and all code written for glibc was source compatible with my new libc, is that a “new platform”?

                              This is a very nice thought, and indeed I think it would be nice if there were multiple libcs. (I’m actually working on a libc, although it doesn’t get a lot of my dev time.) However, this is not really an issue in practice, and windows is the only place where there’s really libc competition.

                              This is even more so the case since libc is limited in how much it can do with macros. Many functions are not allowed to be macros, so you can take their address. This doesn’t prevent silly indirections like #define mmap mmap64, but there’s really not that much of that; and anyway, most of it is for backwards compatibility, meaning that new libcs are unlikely to adopt it, or if they do, it will be the same indirections as the old libcs.

                              languages may not even have a way to express the stuff C functions do cleanly

                              In that case, how would you run syscalls? If you had such a language, you would make such functions built-in/intrinsics/whatever, and now it becomes the implementation language’s problem.

                              it could possibly in some circumstances be useful for other stuff than calling libc functions, but I don’t think it’s wrong to say that it might be a lot of largely unnecessary work

                              I respectfully disagree. Pretty much all languages that interface at all with c have a lot of community-maintained bindings to popular c libraries. That’s a lot of manual work that could be completely elided if the language supported automatically calling into c libraries. Even Qt, which is written in c++ (not c!) and is awfully terrible to interface with even aside from that, has 18 binding sets, for a variety of languages.

                          2. 1

                            How is your solution different from #2? You’d either have to (pre)compile the library for each libc you’d like to support, or drop the support for cross compilation. Both options strike me as a really bad solution.

                            FWIW, if “libc changes anything”, then the ABI is broken and you’d have to recompile the world program anyway.

                            1. 1

                              With my solution, supporting a new libc would just involve recompiling the libc wrapper library. With #2, a new libc (or changes to an existing libc) would involve updating the source code to work with the new (undocumented, unstable) ABI; a recompile wouldn’t be enough.

                              In the FOSS world, recompiling the program (or the world) isn’t a huge deal (after all you’d just have to recompile the libc wrapper library in the same situations as a regular C program would need to be recompiled), but supporting an ever-expanding number of libcs and versions of those libcs kind of is a big deal.

                              1. 1

                                In the FOSS world

                                I’m glad you live in that world…

                                Take a look at what Go does for syscall packages (syscall, x/sys; e.g. types_darwin.go). TL;DR: essentially #2 but constants, type definitions, and some syscall-through-libc wrappers are generated so updating should be a matter of running mksyscall script on the target platform (assuming source compatibility). Sure it’s not bulletproof, but ABI breaking changes are not that common despite all the “no compatibility guarantees” talk.

                          3. 1

                            Via an external package, D can also parse C headers.

                          4. 1

                            I’m not sure I find this argument persuasive. It looks to me like the headers are implying that a symbol is private, but because it’s a preprocessor directive mapping to the private symbol, that private symbol will be explicitly referred to in binary code. So although the interface is declared to be private, in practice it must exist with the same behavior and function signature or existing compiled code will be unable to execute, which doesn’t sound very private at all.

                            (This has burned me in my day job too, where Windows headers, particularly for kernel code, use preprocessor directives to provide access to things. The ABI is therefore post-preprocessed, so things like structure elements accessed via macros become fixed in location for all time. In general using preprocessor directives in any kind of API is a bad idea because it’s just confusing where the ABI really is.)

                            1. 1

                              The problem is that nobody promises that there will always be a symbol int *__errno(void) (or int *__errno_location(void)), only that the word errno in C source code will evaluate to an int. Changes to the libc, or a new libc, might change the implementation details without notice.

                              It’s like how in C++, you can totally re-declare a class and make private members public, and you can then do whatever you want with those members, and stuff will still probably work, but it’s a terrible idea because the author explicitly told you not to depend on private members.

                              1. 1

                                What I’m saying is that the promise exists that those symbols will exist, or previously compiled code will not work. It’s an ABI promise. There’s also an API promise about errno, which is for compiling new programs, but that’s a different promise to the one that ensures existing binary programs run. I realize that in some open source circles there’s a view that binary compatibility should not be guaranteed, but making any such change will break existing programs.

                                (And this particular change is so fundamental that it’d be hard to upgrade to a libc that’s not ABI compatible without a complete offline reinstall, because the moment libc breaks binary compatibility no existing program will work, and no new program will work until the new binary behavior is in place.)

                                1. 3

                                  OpenBSD 5.5 broke its ABI by changing time_t to be 64 bits: https://www.openbsd.org/55.html

                                  Whether you think ABIs should be stable or not doesn’t really matter; there are platforms such as OpenBSD which don’t view ABI breakages as the end of the world, because software can just be recompiled. It’s not about whether ABI compatibility “should” be guaranteed, but whether it actually is.

                                  Here’s how you upgrade libc online:

                                  1. Install a new libc version with a new soname (this would probably be /usr/lib/x86_64-linux-gnu/libc.so.7 for ubuntu with glibc).
                                  2. Nothing bad has happened yet; existing programs will still end up being linked with the old libc.so.6.
                                  3. Upgrade all programs. The upgraded programs are linked against a libc with the soname libc.so.7, so they’ll both expect the new ABI and be linked against the new ABI.
                                  4. Once everything is upgraded, change the libc.so symlink to point to libc.so.7. This step would mean any programs which don’t use sonames properly, and which haven’t been upgraded to work with the new ABI, won’t be able to spawn, but anything else is fine.
                                  5. Optionally, remove the old libc.so.6. This will break software which does use sonames properly and haven’t been updated, because they won’t be able to find libc.so.6 anymore. If this turns out to be an issue, just skip this step.

                                  That’s not so terribly hard is it? I’m sure there would be complications in practice, but those could probably be overcome.

                                  1. 2

                                    Certainly I don’t dispute that OpenBSD are free to sacrifice ABI compatibility, it just seems to introduce a lot of headaches for people particularly when it’s at a lower layer like this. The issues in this thread seem minor compared to the general consequences. I haven’t used OpenBSD and don’t have experience with how they handle this.

                                    In terms of Linux though, which your example refers to, the reason it’s currently libc.so.6 is because in 1997 we had a very painful transition from libc 5, and ever since, glibc has tried to be upward ABI compatible. I’m fuzzy on the details (it was a long time ago and I was learning it at the time), but I remember libc is special because the loader (ld-linux.so) is really tied into the libc version. In theory it was possible to have both ld-linux.so.1 and ld-linux.so.2 loaders, but I never got it to work, and googling “libc5 glibc” now turned up https://ask.slashdot.org/story/99/04/19/2045257/libc5-libc6-and-peaceful-co-existance , which made me laugh. Once two versions are present, the next issue is trying to avoid instantiating both versions into a single process due to conflicting dependencies. I know in practice at the time it was always easier to stay in one world or the other.

                                    Once we settled on glibc being upward compatible, it’s possible to just upgrade a distribution by upgrading things like glibc early in the sequence, and continuing to later packages. At least, that’s how I’ve done my most recent few upgrades.

                                    Somewhat strangely though, before libc5 Linux used the a.out executable format, and it was straightforward to support a.out binaries going through libc4 in parallel with ELF binaries going through glibc, because there was no loader confusion, and any shared library that could be loaded would depend on the correct libc. I used that to run doom well after the glibc transition was fully complete.