I never understood the motivation for moving everything into /usr. The separation exists for a good reason. /usr can be a separate filesystem. The tools required to be able to mount the /usr filesystem live outside of /usr. This often includes network config (if /usr is on a network share). Back in the day, the root FS was often on a floppy disk in machines with no hard disk and would then mount /usr from an NFS share. These days, it’s often a net-booted RAM disk.
The biggest problem with the traditional UNIX layout with respect to /usr is that there’s no /usr/etc. This means that you can’t trivially snapshot configuration for things that live outside of the root FS, along with the data. This is true even on *BSD systems that do have a /usr/local/etc for configuration files for things installed in /usr/local (third-party packages maintained separately from the base system). This isn’t solved by the merged /usr approach because /etc remains in the root FS. You can’t make /etc a separate filesystem because it contains fstab, which you need to be able to mount any non-root FS (you ought to be able to do it with ZFS, which doesn’t need fstab, except that [at least on FreeBSD] the init script that mounts non-root filesystems also lives in /etc).
BSD installations tend to be set up like that, but in these days of large hard drives it really isn’t necessary to have / and /usr be separate. It’s just an artifact of ye olden days that for some reason has apologists.
If you have one hard drive, then you can put / and /usr both on it and you only need to mount one filesystem. If you need reliability in the case of failure, then you have multiple hard drives in a RAID configuration, and you only need to mount one filesystem. If you have separate hard drives for / and /usr, then if one fails then it’s just a coin flip that it’s the root one anyway and you still have to recover from a boot disk.
Running applications from a NFS share sounds painful these days anyway. In my time in two different university labs, application updates were deployed by reimaging the lab machines every night via PXE boot, keeping all user data on network shares. I don’t know if that’s still considered ideal.
BSD installations tend to be set up like that, but in these days of large hard drives it really isn’t necessary to have / and /usr be separate. It’s just an artifact of ye olden days that for some reason has apologists.
In general, my view on change is that it should come with a benefit. I can see a benefit in adding /usr/etc, moving all configuration that isn’t required for mounting /usr out of /etc, and making the root FS something that I can use an integrity-protected read-only FS for and put into my secure-boot chain (and have it then validate signatures on files elsewhere if desired).
There’s less of a need for the separation now that reroot / pivot root exist on FreeBSD / Linux and you can boot from one in-RAM filesystem and then replace it with another, but the cost of maintaining the separation is practically zero so the benefit from removing it needs to be non-trivial to justify the considerable effort involved in squishing everything together (from the article, it’s taken 10 years for Debian to still be struggling with the merge).
If you have one hard drive, then you can put / and /usr both on it and you only need to mount one filesystem. If you need reliability in the case of failure, then you have multiple hard drives in a RAID configuration, and you only need to mount one filesystem. If you have separate hard drives for / and /usr, then if one fails then it’s just a coin flip that it’s the root one anyway and you still have to recover from a boot disk.
That’s an oversimplification and ignores the use cases. If you have a single hard drive, you still do software updates. Having an immutable core is very useful for rollback. With a separate root, you can shapshot that and if an update goes wrong then you can roll back and have enough working stuff to be able to recover. Apple does this with an immutable root fs that is upgraded atomically (write new one, point bootloader at it, delete old one after successful boot). This is also the idea behind ZFS boot environments, which contain everything needed to boot a system, allowing you to roll back to the last one (or temporarily install a different one for testing).
In both cases, GNU/Linux distros suffer from a lack of separation in the other direction. With *BSD, the base system is in / and /usr, your data is in /usr/home, third-party apps are in /usr/local. The boot environments do include /usr, but don’t include /usr/local and so keeping an old one around after an upgrade with a ZFS BE doesn’t cost much disk space. You can snapshot your home, and /usr/local (typically much bigger than the root because it includes big third-party things like X.org and KDE / GNOME on a workstation).
Android and macOS have similar separations, with different spellings of the directories involved. In both cases, it lets them ship the core OS image as a compressed, integrity-protected, filesystem snapshot. The fact that it’s immutable means that you can compress it a lot better and you can use a filesystem with replay protection (a Merkel tree over all blocks, as well as per-block hashes) without having to worry about the fact that maintaining such a thing over a mutable filesystem has a big performance overhead.
Combining root and /usr, if /usr is already implicitly combined with /usr/local (or /System / /system, depending on your preferred spelling) makes all of these things harder. Most of the push for it seems to be ‘the separation adds no benefit for my specific use case, everyone who needs it should STFU’.
Running applications from a NFS share sounds painful these days anyway
I can mount an NFS share from Azure on my home directory and have similar throughput and similar worst-case latency to using a local spinning-rust disk. For a single building, a network share backed by a decent amount of RAM and fast SSDs is a lot cheaper for the same performance than putting big, fast, SSDs in every machine.
I think we agree a lot more than we disagree, sorry for coming off as adversarial. XD
If you have a single hard drive, you still do software updates. Having an immutable core is very useful for rollback. With a separate root, you can shapshot that and if an update goes wrong then you can roll back and have enough working stuff to be able to recover.
That’s a very good point and, I hadn’t really considered that use case since I tend to, well, not use it. I have an ansible config that can reimage a bare desktop to my ideal configuration in two commands, and all user data is backed up and/or on a NAS. I have simply optimized for the “nuke and pave” approach rather than the “secure fallback” approach. That said, I work with systems that would really benefit from being able to install updates onto one partition, switch over to it, and then fall back to the original if something went wrong. In that case I’d go the other way, I’d agree that Linux’s / and /usr split doesn’t go far enough.
I can mount an NFS share from Azure on my home directory and have similar throughput and similar worst-case latency to using a local spinning-rust disk.
Spinning rust?! You masochist! :-P
For a single building, a network share backed by a decent amount of RAM and fast SSDs is a lot cheaper for the same performance than putting big, fast, SSDs in every machine.
That’s the thing, a “small” SSD is now like 120 GB, which can be had for $20-50 on Newegg. As far as I recall, even my Win10 install doesn’t actually use that much space on OS+applications.
Running applications from a NFS share sounds painful these days anyway.
If the pain is performance, things have changed a lot since the old days when NFS meant everything was dog slow: on a fast (10Gbps+) local network, talking to a fileserver has low enough latency and high enough throughput that the difference compared to loading from a local disk is often imperceptible to a human operator.
However, the pain of “the fileserver or the network is down and I’m dead in the water” remains.
I never understood the motivation for moving everything into
/usr. The separation exists for a good reason./usrcan be a separate filesystem. The tools required to be able to mount the/usrfilesystem live outside of/usr. This often includes network config (if/usris on a network share). Back in the day, the root FS was often on a floppy disk in machines with no hard disk and would then mount/usrfrom an NFS share. These days, it’s often a net-booted RAM disk.The biggest problem with the traditional UNIX layout with respect to
/usris that there’s no/usr/etc. This means that you can’t trivially snapshot configuration for things that live outside of the root FS, along with the data. This is true even on *BSD systems that do have a/usr/local/etcfor configuration files for things installed in/usr/local(third-party packages maintained separately from the base system). This isn’t solved by the merged/usrapproach because/etcremains in the root FS. You can’t make/etca separate filesystem because it containsfstab, which you need to be able to mount any non-root FS (you ought to be able to do it with ZFS, which doesn’t needfstab, except that [at least on FreeBSD] the init script that mounts non-root filesystems also lives in/etc).I’m curious if any modern Linux installations are set up like that.
These days, Linux has initrd/initramfs, which mostly fulfills the “tools needed to mount the rest of the system” role.
Edit: I guess putting network configuration on the initramfs would be fairly unusual. Seems doable, though.
BSD installations tend to be set up like that, but in these days of large hard drives it really isn’t necessary to have
/and/usrbe separate. It’s just an artifact of ye olden days that for some reason has apologists.If you have one hard drive, then you can put
/and/usrboth on it and you only need to mount one filesystem. If you need reliability in the case of failure, then you have multiple hard drives in a RAID configuration, and you only need to mount one filesystem. If you have separate hard drives for/and/usr, then if one fails then it’s just a coin flip that it’s the root one anyway and you still have to recover from a boot disk.Running applications from a NFS share sounds painful these days anyway. In my time in two different university labs, application updates were deployed by reimaging the lab machines every night via PXE boot, keeping all user data on network shares. I don’t know if that’s still considered ideal.
In general, my view on change is that it should come with a benefit. I can see a benefit in adding
/usr/etc, moving all configuration that isn’t required for mounting/usrout of/etc, and making the root FS something that I can use an integrity-protected read-only FS for and put into my secure-boot chain (and have it then validate signatures on files elsewhere if desired).There’s less of a need for the separation now that reroot / pivot root exist on FreeBSD / Linux and you can boot from one in-RAM filesystem and then replace it with another, but the cost of maintaining the separation is practically zero so the benefit from removing it needs to be non-trivial to justify the considerable effort involved in squishing everything together (from the article, it’s taken 10 years for Debian to still be struggling with the merge).
That’s an oversimplification and ignores the use cases. If you have a single hard drive, you still do software updates. Having an immutable core is very useful for rollback. With a separate root, you can shapshot that and if an update goes wrong then you can roll back and have enough working stuff to be able to recover. Apple does this with an immutable root fs that is upgraded atomically (write new one, point bootloader at it, delete old one after successful boot). This is also the idea behind ZFS boot environments, which contain everything needed to boot a system, allowing you to roll back to the last one (or temporarily install a different one for testing).
In both cases, GNU/Linux distros suffer from a lack of separation in the other direction. With *BSD, the base system is in
/and/usr, your data is in/usr/home, third-party apps are in/usr/local. The boot environments do include/usr, but don’t include/usr/localand so keeping an old one around after an upgrade with a ZFS BE doesn’t cost much disk space. You can snapshot your home, and/usr/local(typically much bigger than the root because it includes big third-party things like X.org and KDE / GNOME on a workstation).Android and macOS have similar separations, with different spellings of the directories involved. In both cases, it lets them ship the core OS image as a compressed, integrity-protected, filesystem snapshot. The fact that it’s immutable means that you can compress it a lot better and you can use a filesystem with replay protection (a Merkel tree over all blocks, as well as per-block hashes) without having to worry about the fact that maintaining such a thing over a mutable filesystem has a big performance overhead.
Combining root and
/usr, if/usris already implicitly combined with/usr/local(or/System//system, depending on your preferred spelling) makes all of these things harder. Most of the push for it seems to be ‘the separation adds no benefit for my specific use case, everyone who needs it should STFU’.I can mount an NFS share from Azure on my home directory and have similar throughput and similar worst-case latency to using a local spinning-rust disk. For a single building, a network share backed by a decent amount of RAM and fast SSDs is a lot cheaper for the same performance than putting big, fast, SSDs in every machine.
I don’t think that is true. The original promotional material lays out some real costs. I’m not sure whether those costs of maintaining the split are greater than the costs of the merge but they are well above zero.
I think we agree a lot more than we disagree, sorry for coming off as adversarial. XD
That’s a very good point and, I hadn’t really considered that use case since I tend to, well, not use it. I have an
ansibleconfig that can reimage a bare desktop to my ideal configuration in two commands, and all user data is backed up and/or on a NAS. I have simply optimized for the “nuke and pave” approach rather than the “secure fallback” approach. That said, I work with systems that would really benefit from being able to install updates onto one partition, switch over to it, and then fall back to the original if something went wrong. In that case I’d go the other way, I’d agree that Linux’s/and/usrsplit doesn’t go far enough.Spinning rust?! You masochist! :-P
That’s the thing, a “small” SSD is now like 120 GB, which can be had for $20-50 on Newegg. As far as I recall, even my Win10 install doesn’t actually use that much space on OS+applications.
If the pain is performance, things have changed a lot since the old days when NFS meant everything was dog slow: on a fast (10Gbps+) local network, talking to a fileserver has low enough latency and high enough throughput that the difference compared to loading from a local disk is often imperceptible to a human operator.
However, the pain of “the fileserver or the network is down and I’m dead in the water” remains.