I’m curious why this uses musl instead of FreeBSD libc. I’d love to see patches to make FreeBSD libc portable upstream. I’d not heard of DInit before but it looks very interesting.
unfortunately i don’t really have the time or resources to port a libc on top of doing everything else, so that would be the primary reason (besides, musl is generally quite alright)
like, it’s not only about the process of porting the libc, which is already a ton of effort, but then also dealing with userland software not expecting yet another combo and breaking in the process - things were already a ton of work as they are :)
freebsd libc also has some sketchy parts (libthr mallocing during pthread creation for instance) though obviously that’s stuff that could be addressed when you don’t have ABI compat to worry about
Thread creation intrinsically requires memory allocation, so whether it calls malloc or mmap it still has to handle failure. The only two possible implementations are to call malloc or to call a reimplementation of malloc. Why is the latter preferable?
Malloc failures should probably be reported as EAGAIN, but ENOMEM is more informative. As I recall, POSIX explicitly permits APIs to return more specific erroneous values.
it does not need a “reimplementation of malloc”, thread creation merely needs to map a fixed number of pages (possibly as low as one, depending on your system page size) for its stack and thread-specific data
at the best it makes the job of the malloc implementation a bit harder because malloc itself may have to deal with pthread state (as modern malloc impls don’t have a global lock, they need per-thread storage and they may need to store a pointer to that either in the pthread struct or in TLS)
it does not need a “reimplementation of malloc”, thread creation merely needs to map a fixed number of pages (possibly as low as one, depending on your system page size) for its stack and thread-specific data
Callers are permitted to provide their own stack, so you must at least allocate space for the pthread structure and TLS region in some cases. If you care about security, ideally you want to allocate these structures all separately so that overflows are harder to exploit and, if your allocator does randomisation, so that knowing the address of one does not allow you to predict the others. Even I f you don’t, rounding these up to a multiple of page size is wasteful, particularly on platforms with larger pages (e.g. PowerPC with 64 KiB pages).
at the best it makes the job of the malloc implementation a bit harder because malloc itself may have to deal with pthread state (as modern malloc impls don’t have a global lock, they need per-thread storage and they may need to store a pointer to that either in the pthread struct or in TLS)
This was not a problem for us when we wrote snmalloc or for any other malloc that I have tried.
considering you typically need 120k+ worth of stack, i don’t think there is much waste per se; application can provide its own stack, yes - that is rare, for one, for two, you can still put the TLS in the application stack space depending on how much stack the application gives you and how much TLS is needed, leaving the waste for a relatively unlikely subset of scenarios
allocating these structures separately does not really gain you much if anything security-wise, provided correct implementation
considering you typically need 120k+ worth of stack, i don’t think there is much waste per se;
That is very workload dependent. FreeBSD reserves a few MiBs of *address space *per stack, but it’s surprisingly common to see workloads that spawn tens of thousands of threads but use <16 KiB of committed memory for each one.
application can provide its own stack, yes - that is rare, for one,
It’s rare, but it also correlates strongly with people who processes that create a lot of threads and tune their stack size carefully. It’s far more common than code that depends on the precise errno returns from pthread_create.
for two, you can still put the TLS in the application stack space
Only if you like breaking application code written by people that carefully tuned how much stack space they need. In particular, if they’re creating stacks that run only application code (or code from their shared library) then they can tune the stack size but they can’t control the TLS size because that depends on how much other libraries (including libc) use and so suddenly upgrading a shared library can cause your code to crash in threads that don’t touch that library if the threading library puts things in your stack space.
allocating these structures separately does not really gain you much if anything security-wise, provided correct implementation
It prevents a stack buffer overflow or underflow from being able to corrupt data structures that are walked from other threads. It’s not a total game changer for an attacker, but it’s an incredibly convenient tool for converting a lack of bounds checking on a stack buffer into a compromise of another thread. Doing the same with separate allocations is harder and requires another step in your exploit chain (or, typically, going after some other low-hanging fruit - it becomes a lot more important if you fix the other things).
Interesting project! I’m pretty married to Debian (been using it for 22 years now), but I love to see efforts to rethink the status quo, especially ones that try to simplify. Good luck, hope to try Chimera Linux when it’s out of alpha/beta!
HYPE. i’m really excited about Chimera Linux - I think the combination of dinit, musl libc, and the freeBSD coreutils is my favorite usability/simplicity tradeoff i’ve ever experienced in a distro. so excited about the future of this distro.
why does every other linux distribution exist? most of them provide far less added value compared to the others, yet nobody questions their existence, perhaps because they are fundamentally uninteresting
it’s always a combination of many smaller factors
i just wanted to make a well-rounded and somewhat opinionated system without cruft or sketchy parts, but still featureful and generally usable (no suckless junk), with a low-maintenance but high-correctness package build system and infrastructure, no systemd but taking service management and everything around it seriously (there are exactly 0 other non-systemd distros that do that), easy to bootstrap, architecture-agnostic, and hardened (as much as possible without introducing visible breakage or significant performance loss; the only “linux distro” that really takes this seriously is android, and that’s not general-purpose)
none of the individual choices like userland base or libc or whatever are goals in themselves, they are a means to an end or just the most obvious thing given the circumstances
People have questioned from the start why we needed/had Slackware, Debian, Red Hat, SuSE, Arch Linux, Gentoo, … even though initially the differences were much larger than nowadays.
If your existence is being questioned, you’re being noticed, thought and talked about. That’s a good thing.
no systemd but taking service management and everything around it seriously (there are exactly 0 other non-systemd distros that do that),
I don’t think those distros would agree with you there ;).
they may not agree, but it’s precisely why systemd got adopted by nearly everybody; it does things people consider important which others don’t do
most non-systemd distros don’t even get as far as having service supervision by default, but even those that do, they generally don’t have things such as first-class support for user services, reliable management of bus-activated processes, robust handling of dependencies, and a ton of other things
some of them also seem to get the idea that service management is an easy problem and then “support multiple inits” while having half-assed, deficient integration of every single one they “support” (cough artix cough)
some others just wallow in their boomerware and pretend everything was fine back in sysvinit+a hacked together pile of shell scripts constituting the “service management” days
there is no certain answer to that, considering it depends on the outcome; if the only difference is “s6 instead of openrc”, then definitely (however, i don’t see alpine landing s6 anytime soon)
I would love to hear you talk more about what you think existing existing service management systems do right or wrong. My impression is that you think systemd is as good as it gets for now, but I’d love to know from your perspective why that is (I agree, for the record, but I’m not shipping my own linux distro).
hm, i’d say it comes down to a lot of things, some small, some significant
and some of it is not even a matter of which service manager you choose or whether it’s good or bad; integration and making the most of what you are given is extremely important - and systemd was probably the first thing that brought unified integration across distributions (so you can write your systemd unit file as an upstream, and have it working effectively everywhere)
the other big thing is how it works for users; systemd is a terrible implementation and has countless sketchy parts, but if a user needs to do something, systemd typically has a way to do it - of course, there are lots of service managers and some are less functionally deficient than others, but in general every one of them lacks in some important aspect - but most importantly none of them come with “here, take this suite of oneshots and core services, and upstreams can write against this, and users can expect these targets/dependencies to be present, these core services to be present, etc.”
systemd also brought some important concepts that nothing that normal people could use on linux actually implemented before, i’d say the most important ones are 1) if it’s a long-running process, it runs under systemd and is supervised and monitorable under systemd, it’s not ad-hoc spawned by some random thing 2) user services are a first-class concept, they work identically to regular services, users can make their own and the system can also run things as user services 3) login/session tracking is built-in and accessible via API, so third party daemons have an easy way to access this information
these things have some far-reaching consequences; for instance, consider dbus - previously the session bus was spawned ad-hoc typically by your x11 xsession scripts and had an abstract socket, which was not only fragile and limited (things on a different tty could not access the session bus of your desktop tty) but also rather opaque and cryptic; and things like dbus-activated services had no supervision whatsoever, which is far from robust; the desktop init sequence was random and hacky, you had no reliable way to manage your sound server, etc.
so yeah, despite all the bad, some parts that were brought (not invented of course, but made available) by systemd were important enough that they trump whatever bad there may be (even though there are tons of it) - and there was very little actual effort to do something about the situation, despite all the whining
chimera is trying to do something about this; of course, it’s not possible to implement everything in a short timeframe, but some core things are already present - we have a set of portable core services that can be deployed on any distribution using dinit, we have a robust session/login tracker and good support for user services out of box, dbus is handled in the same manner as with systemd, dbus activation integrated with service management is coming (likely alongside switch to dbus-broker), and so on
the use of musl is less of an aesthetic choice and more of a mandatory choice; glibc is generally nothing but trouble (the code is rather needlessly impenetrable and makes poor choices regarding ABI on several architectures, e.g. IBM long doubles on powerpc, and it also makes things like static linking rather problematic) and i don’t think any stable version even compiles with llvm yet (i’ve seen some post a couple months ago that it finally works experimentally, but probably still needs libgcc)
and outside of musl, there just isn’t anything else that is capable of working with roughly comparable amount of software as glibc is
Quite excited for this. I’ve been following development since early on, waiting for the guidance that it’s time to try daily driving it and that time has now arrived. There will not doubt be challenges and packages missing for things I use but that’s half the fun and a way to contribute.
This looks very exciting to me, very close to what I’d want if I were to invest time in building my own distro. I do however wonder if the authors know of distr1.org and what they think of its mounting-based package manager. APK is the fastest “traditional” package manager that I know of, but the idea and benefits of an image-based package manager intrigues me.
I’m curious why this uses musl instead of FreeBSD libc. I’d love to see patches to make FreeBSD libc portable upstream. I’d not heard of DInit before but it looks very interesting.
unfortunately i don’t really have the time or resources to port a libc on top of doing everything else, so that would be the primary reason (besides, musl is generally quite alright)
like, it’s not only about the process of porting the libc, which is already a ton of effort, but then also dealing with userland software not expecting yet another combo and breaking in the process - things were already a ton of work as they are :)
freebsd libc also has some sketchy parts (libthr mallocing during pthread creation for instance) though obviously that’s stuff that could be addressed when you don’t have ABI compat to worry about
I’m curious why you think that is sketch. It hasn’t caused any issues with any malloc implementations that I’ve used.
POSIX does not specify ENOMEM as a possible errno for that API, and the libc is allowing it to happen
i see it mostly as a bad historical decision that nothing can be done about at this point; freebsd is quite unique in doing this
Thread creation intrinsically requires memory allocation, so whether it calls malloc or mmap it still has to handle failure. The only two possible implementations are to call malloc or to call a reimplementation of malloc. Why is the latter preferable?
Malloc failures should probably be reported as EAGAIN, but ENOMEM is more informative. As I recall, POSIX explicitly permits APIs to return more specific erroneous values.
it does not need a “reimplementation of malloc”, thread creation merely needs to map a fixed number of pages (possibly as low as one, depending on your system page size) for its stack and thread-specific data
at the best it makes the job of the malloc implementation a bit harder because malloc itself may have to deal with pthread state (as modern malloc impls don’t have a global lock, they need per-thread storage and they may need to store a pointer to that either in the pthread struct or in TLS)
Callers are permitted to provide their own stack, so you must at least allocate space for the pthread structure and TLS region in some cases. If you care about security, ideally you want to allocate these structures all separately so that overflows are harder to exploit and, if your allocator does randomisation, so that knowing the address of one does not allow you to predict the others. Even I f you don’t, rounding these up to a multiple of page size is wasteful, particularly on platforms with larger pages (e.g. PowerPC with 64 KiB pages).
This was not a problem for us when we wrote snmalloc or for any other malloc that I have tried.
considering you typically need 120k+ worth of stack, i don’t think there is much waste per se; application can provide its own stack, yes - that is rare, for one, for two, you can still put the TLS in the application stack space depending on how much stack the application gives you and how much TLS is needed, leaving the waste for a relatively unlikely subset of scenarios
allocating these structures separately does not really gain you much if anything security-wise, provided correct implementation
That is very workload dependent. FreeBSD reserves a few MiBs of *address space *per stack, but it’s surprisingly common to see workloads that spawn tens of thousands of threads but use <16 KiB of committed memory for each one.
It’s rare, but it also correlates strongly with people who processes that create a lot of threads and tune their stack size carefully. It’s far more common than code that depends on the precise errno returns from pthread_create.
Only if you like breaking application code written by people that carefully tuned how much stack space they need. In particular, if they’re creating stacks that run only application code (or code from their shared library) then they can tune the stack size but they can’t control the TLS size because that depends on how much other libraries (including libc) use and so suddenly upgrading a shared library can cause your code to crash in threads that don’t touch that library if the threading library puts things in your stack space.
It prevents a stack buffer overflow or underflow from being able to corrupt data structures that are walked from other threads. It’s not a total game changer for an attacker, but it’s an incredibly convenient tool for converting a lack of bounds checking on a stack buffer into a compromise of another thread. Doing the same with separate allocations is harder and requires another step in your exploit chain (or, typically, going after some other low-hanging fruit - it becomes a lot more important if you fix the other things).
Interesting project! I’m pretty married to Debian (been using it for 22 years now), but I love to see efforts to rethink the status quo, especially ones that try to simplify. Good luck, hope to try Chimera Linux when it’s out of alpha/beta!
HYPE. i’m really excited about Chimera Linux - I think the combination of
dinit, musl libc, and the freeBSD coreutils is my favorite usability/simplicity tradeoff i’ve ever experienced in a distro. so excited about the future of this distro.Why does this exist? Why would you use this?
Tl;dr: this is Linux sans GNU as much as possible. It doesn’t even use GNU Coreutils.
It avoids systemd, too.
have you tried building systemd with an unconventional toolchain/libc/userland? :) it’s almost as if systemd doesn’t want me to use it
My hottest take is that musl + systemd would make for a great distro… if only systemd upstream wasn’t so hostile to it.
That’s almost a given among these micro-distros.
why does every other linux distribution exist? most of them provide far less added value compared to the others, yet nobody questions their existence, perhaps because they are fundamentally uninteresting
it’s always a combination of many smaller factors
i just wanted to make a well-rounded and somewhat opinionated system without cruft or sketchy parts, but still featureful and generally usable (no suckless junk), with a low-maintenance but high-correctness package build system and infrastructure, no systemd but taking service management and everything around it seriously (there are exactly 0 other non-systemd distros that do that), easy to bootstrap, architecture-agnostic, and hardened (as much as possible without introducing visible breakage or significant performance loss; the only “linux distro” that really takes this seriously is android, and that’s not general-purpose)
none of the individual choices like userland base or libc or whatever are goals in themselves, they are a means to an end or just the most obvious thing given the circumstances
People have questioned from the start why we needed/had Slackware, Debian, Red Hat, SuSE, Arch Linux, Gentoo, … even though initially the differences were much larger than nowadays.
If your existence is being questioned, you’re being noticed, thought and talked about. That’s a good thing.
I don’t think those distros would agree with you there ;).
they may not agree, but it’s precisely why systemd got adopted by nearly everybody; it does things people consider important which others don’t do
most non-systemd distros don’t even get as far as having service supervision by default, but even those that do, they generally don’t have things such as first-class support for user services, reliable management of bus-activated processes, robust handling of dependencies, and a ton of other things
some of them also seem to get the idea that service management is an easy problem and then “support multiple inits” while having half-assed, deficient integration of every single one they “support” (cough artix cough)
some others just wallow in their boomerware and pretend everything was fine back in sysvinit+a hacked together pile of shell scripts constituting the “service management” days
Will you still feel that way if Alpine ever lands s6 to replace openrc?
there is no certain answer to that, considering it depends on the outcome; if the only difference is “s6 instead of openrc”, then definitely (however, i don’t see alpine landing s6 anytime soon)
I would love to hear you talk more about what you think existing existing service management systems do right or wrong. My impression is that you think systemd is as good as it gets for now, but I’d love to know from your perspective why that is (I agree, for the record, but I’m not shipping my own linux distro).
hm, i’d say it comes down to a lot of things, some small, some significant
and some of it is not even a matter of which service manager you choose or whether it’s good or bad; integration and making the most of what you are given is extremely important - and systemd was probably the first thing that brought unified integration across distributions (so you can write your systemd unit file as an upstream, and have it working effectively everywhere)
the other big thing is how it works for users; systemd is a terrible implementation and has countless sketchy parts, but if a user needs to do something, systemd typically has a way to do it - of course, there are lots of service managers and some are less functionally deficient than others, but in general every one of them lacks in some important aspect - but most importantly none of them come with “here, take this suite of oneshots and core services, and upstreams can write against this, and users can expect these targets/dependencies to be present, these core services to be present, etc.”
systemd also brought some important concepts that nothing that normal people could use on linux actually implemented before, i’d say the most important ones are 1) if it’s a long-running process, it runs under systemd and is supervised and monitorable under systemd, it’s not ad-hoc spawned by some random thing 2) user services are a first-class concept, they work identically to regular services, users can make their own and the system can also run things as user services 3) login/session tracking is built-in and accessible via API, so third party daemons have an easy way to access this information
these things have some far-reaching consequences; for instance, consider dbus - previously the session bus was spawned ad-hoc typically by your x11 xsession scripts and had an abstract socket, which was not only fragile and limited (things on a different tty could not access the session bus of your desktop tty) but also rather opaque and cryptic; and things like dbus-activated services had no supervision whatsoever, which is far from robust; the desktop init sequence was random and hacky, you had no reliable way to manage your sound server, etc.
so yeah, despite all the bad, some parts that were brought (not invented of course, but made available) by systemd were important enough that they trump whatever bad there may be (even though there are tons of it) - and there was very little actual effort to do something about the situation, despite all the whining
chimera is trying to do something about this; of course, it’s not possible to implement everything in a short timeframe, but some core things are already present - we have a set of portable core services that can be deployed on any distribution using dinit, we have a robust session/login tracker and good support for user services out of box, dbus is handled in the same manner as with systemd, dbus activation integrated with service management is coming (likely alongside switch to dbus-broker), and so on
This was a way more comprehensive answer than I could have hoped for, thanks! It might be worth a blog post!
For most end users it’s about libc as an aesthetic choice, not a technical one.
the use of musl is less of an aesthetic choice and more of a mandatory choice; glibc is generally nothing but trouble (the code is rather needlessly impenetrable and makes poor choices regarding ABI on several architectures, e.g. IBM long doubles on powerpc, and it also makes things like static linking rather problematic) and i don’t think any stable version even compiles with llvm yet (i’ve seen some post a couple months ago that it finally works experimentally, but probably still needs libgcc)
and outside of musl, there just isn’t anything else that is capable of working with roughly comparable amount of software as glibc is
curious, what about musl do you find so aesthetically pleasing?
Quite excited for this. I’ve been following development since early on, waiting for the guidance that it’s time to try daily driving it and that time has now arrived. There will not doubt be challenges and packages missing for things I use but that’s half the fun and a way to contribute.
This looks very exciting to me, very close to what I’d want if I were to invest time in building my own distro. I do however wonder if the authors know of distr1.org and what they think of its mounting-based package manager. APK is the fastest “traditional” package manager that I know of, but the idea and benefits of an image-based package manager intrigues me.
This looks awesome. And by the looks of it the name is really fitting. Looking forward to trying thus out.