I haven’t been a Mac/macOS user for many years but I’m still excited to see Linux, FreeBSD, and Windows (via BootCamp?) running on the M1 because I think all their mainstream system schedulers are too reliant on NUMA & SMP behavior and could use some overhauling to better take advantage of non-NUMA/non-SMP compute power.
There are a lot of subtle assumptions that have big impact. For example, even though AMD goes to great lengths to make ThreadRipper appear to be a NUMA architecture there were a number of Windows cpu scheduler bugs that caused a high standard deviation in latencies of many operations due to the non-uniform memory access patterns of the various chiplets/cores in the TR1/TR2 processors. Microsoft patched those in a Windows 10 point release, but I think there’s a lot more low-hanging fruit no one has bothered picking.
Arm worked a lot on their Intelligent Power Architecture in Linux, which provides big.LITTLE power-aware scheduling. The Linux kernel is very Intel-focused and so this was the biggest (non-disruptive) change that they could get upstreamed. FreeBSD has nothing equivalent (Robin, who led the IPA project, was interested in helping but no one volunteered to do or fund the work). In general, it’s a very difficult problem for a few reasons:
Even nominally SMP systems have some interesting corner cases. We found, for example, that disabling CPU affinity in the FreeBSD scheduler made some parallel workloads faster because CPU affinity would cause whatever the most-used execution units in a particular workload were to get hot and then cause thermal throttling to kick in. Without it, there was a small penalty for migration (but within a socket, remote cache snooping is now so fast that it doesn’t make much difference) but then the CPU heated more evenly and maintained a higher clock speed.
It’s very easy to make things perform differently with small tweaks to the scheduler. Making that a consistent performance improvement is incredibly hard.
I just want to say thanks for taking the time out to write that. It’s a greatly informative response, and I have a lot of topics to research/chase down.