Congrats! I believe Hare uses QBE as a backend, and there may be others. Perhaps some users could be listed on the home page?
QBE’s goals make a good bit of sense to me; it seems right for the backend to focus on emitting really good machine code, and it’s ok to have a “garbage in, garbage out” philosophy. LLVM was designed to handle almost everything, but recent languages (Swift, Rust, Julia) have their own middle IRs for language-specific optimisations. So it increasingly makes sense to expect that the backend receives relatively good IR, and doesn’t need some of the more magical simplifications. All those arch-specific peephole optimisations provide the real value.
I’m only slightly disappointed to see Phi nodes, which are a bit less elegant than block arguments IMO (which MLIR, Swift and Cranelift’s newer IRs use – rationale). But of course it’s no deal-breaker.
I was quite disappointed to see that there’s no pointer type in the IR. That means that it will never be able to target a CHERI platform (or any other architecture where pointers are not Integers), so the Morello system that I’m writing code on right now can never be a target.
It looks like it changed, but I remember at the beginning the goal of QBE was to be super simple as opposed to the “bloated LLVM”, they were planning to only target amd64 and arm64. It looks like they now also support riscv64, so they might have changed and give up on that “few architecture” goals.
Unfortunately, the GPU driver doesn’t work yet, but apparently Wayland does once the GPU driver is working. I’m hoping to start using it as my work desktop soon, for now I’m sshing into it from a Windows machine. My normal working environment is the Windows Terminal sshing into a FreeBSD/x86-64 VM, so switching it to sshing into a FreeBSD/Morello computer isn’t that big a change…
On a quick skim, over the first few search results, CHERI is an ISA, something similar to RISC, just extended with some capabilities around virtualization, memory protection? And Morello is …a CPU? SoC? As in, not exactly ARM, but something like that. Am I in the neighbourhood?
Can you try to explain to a layman, what does it do differently then arm or riscv?
CHERI is a set of extensions that add a capability model to memory. The hardware supports a capability type that is protected by a non-addressable tag bit when stored in memory. A capability grants rights to a range of an address space (e.g. read, write, and / or execute permissions). Every memory access (load, store, instruction fetch) must be authorised by a capability, which must be presented for the operation to succeed. For new load and store instructions, the base operand is a capability in a general-purpose capability register. For legacy instructions, the capability is an implicit default data capability.
In CHERI C/C++, every pointer is lowered by the compiler to be represented by a capability. This means that you cannot access any memory except via a pointer that was created by something holding a more powerful capability. For example, the compiler will derive bounded capabilities to from the stack capability register for stack allocations. The OS will return a capability in response to mmap system calls, which the memory allocator will then hold and subdivide to hand out object-bounded capabilities in response to malloc. This means that you cannot forge a pointer and you cannot ever access out of bounds of an object (guaranteed by the hardware). With our temporal safety work, you also cannot access an object that has been freed and reallocated (guaranteed by software, accelerated by some hardware features). To be able to compiler for this kind of target, the compiler must maintain the integer/pointer distinction all of the way through to the final machine-code lowering (arithmetic on pointers uses different instructions to arithmetic on integers, for example).
The CHERI extensions are designed to be architecture neutral. We originally prototyped on MIPS and are now dong RISC-V prototyping and are in the early stages of an official CHERI RISC-V extension. Morello is a localisation to AArch64 and Arm has produced a few thousand test chips / dev systems based on the Neoverse N1 for software development and experimentation. This is what I have under my desk: a modified quad-core Neoverse N1 running at 2.5GHz with 16 GiB of RAM in a desktop chassis with a 250 GiB SSD. We also have a load of them in a rack for CI (snmalloc CI on Morello and benchmarking.
If all goes well, I expect to see CHERI extensions on mainstream architectures on the next few years and so developing a compiler toolchain based on an abstraction that can’t possibly support them without significant rearchitecting seems like an unfortunate decision, especially when maintaining a separate pointer type is fairly simple if you design it in from scratch. The fact that LLVM had separate integer and pointer types in IR made the original CHERI work feasible, the fact that it loses that distinction by the time it reaches SelectionDAG and the back end (one of the first questions the target-agnostic code generator asks the back end is ‘so, which integer type do you want to use for pointers?’) made it harder.
CHERI is (I think) both a security model and set of hypothetical and/or experimental extensions for multiple ISA including MIPS, ARM, RISC-V, and x86 the latter of which is currently just a “sketch” [1]
Morello is the realization of actual silicon implementing the extensions [2]
As for an actual description I’d rather point you towards the Microsoft article (I honestly really liked it). That and the discussions were what painted most of my picture of the project(s). There’s also the technical report An Introduction to CHERI which helped fill in other details but there were things or referenced concepst I wasn’t clear on.
To date, our published research has been based on the 64-bit MIPS ISA; MIPS was the dominant RISC ISA in use in 2010 when the project began.
…
However, since that time we have performed significant investigation into CHERI as portable architectural security model suitable for use in multiple ISAs.
We have also developed an “architectural sketch” of a CHERI-x86-64 that extends the 64-bit x86 ISA with CHERI support.
The Morello CPU is a quad-core, 2.5GHz modified Arm Neoverse N1, a contemporary superscalar server core. Prior to this, the most advanced CHERI implementation was the CHERI version of Toooba, which can run in an FPGA at 50MHz in a dual-core configuration and is roughly equivalent in microarchitecture to a mid-‘90s CPU.
The panfrost bit. There are some PCIe slots, but I’ve not tried plugging anything into them yet. We’re hoping to set some of them up with some RDMA-capable smartNICs and see if we can do something interesting eventually.
Right there in the menu as well! Thank you for pointing this out to me. I think I had assumed this would be something like a community page (eg “user mailing list”).
At that level of integration, maybe a JIT library like GNU Lightning would make more sense. I like QBE’s commitment to the text API, because it means that I don’t have to write my compilers in languages with strong C/C++ FFI.
Congrats! I believe Hare uses QBE as a backend, and there may be others. Perhaps some users could be listed on the home page?
QBE’s goals make a good bit of sense to me; it seems right for the backend to focus on emitting really good machine code, and it’s ok to have a “garbage in, garbage out” philosophy. LLVM was designed to handle almost everything, but recent languages (Swift, Rust, Julia) have their own middle IRs for language-specific optimisations. So it increasingly makes sense to expect that the backend receives relatively good IR, and doesn’t need some of the more magical simplifications. All those arch-specific peephole optimisations provide the real value.
I’m only slightly disappointed to see Phi nodes, which are a bit less elegant than block arguments IMO (which MLIR, Swift and Cranelift’s newer IRs use – rationale). But of course it’s no deal-breaker.
I was quite disappointed to see that there’s no pointer type in the IR. That means that it will never be able to target a CHERI platform (or any other architecture where pointers are not Integers), so the Morello system that I’m writing code on right now can never be a target.
It looks like it changed, but I remember at the beginning the goal of QBE was to be super simple as opposed to the “bloated LLVM”, they were planning to only target
amd64andarm64. It looks like they now also supportriscv64, so they might have changed and give up on that “few architecture” goals.Exciting! If I had a desktop-capable CHERI machine on my desk, I would also think first of coming online to tell the world :-)
Unfortunately, the GPU driver doesn’t work yet, but apparently Wayland does once the GPU driver is working. I’m hoping to start using it as my work desktop soon, for now I’m sshing into it from a Windows machine. My normal working environment is the Windows Terminal sshing into a FreeBSD/x86-64 VM, so switching it to sshing into a FreeBSD/Morello computer isn’t that big a change…
On a quick skim, over the first few search results, CHERI is an ISA, something similar to RISC, just extended with some capabilities around virtualization, memory protection? And Morello is …a CPU? SoC? As in, not exactly ARM, but something like that. Am I in the neighbourhood?
Can you try to explain to a layman, what does it do differently then arm or riscv?
CHERI is a set of extensions that add a capability model to memory. The hardware supports a capability type that is protected by a non-addressable tag bit when stored in memory. A capability grants rights to a range of an address space (e.g. read, write, and / or execute permissions). Every memory access (load, store, instruction fetch) must be authorised by a capability, which must be presented for the operation to succeed. For new load and store instructions, the base operand is a capability in a general-purpose capability register. For legacy instructions, the capability is an implicit default data capability.
In CHERI C/C++, every pointer is lowered by the compiler to be represented by a capability. This means that you cannot access any memory except via a pointer that was created by something holding a more powerful capability. For example, the compiler will derive bounded capabilities to from the stack capability register for stack allocations. The OS will return a capability in response to
mmapsystem calls, which the memory allocator will then hold and subdivide to hand out object-bounded capabilities in response tomalloc. This means that you cannot forge a pointer and you cannot ever access out of bounds of an object (guaranteed by the hardware). With our temporal safety work, you also cannot access an object that has been freed and reallocated (guaranteed by software, accelerated by some hardware features). To be able to compiler for this kind of target, the compiler must maintain the integer/pointer distinction all of the way through to the final machine-code lowering (arithmetic on pointers uses different instructions to arithmetic on integers, for example).The CHERI extensions are designed to be architecture neutral. We originally prototyped on MIPS and are now dong RISC-V prototyping and are in the early stages of an official CHERI RISC-V extension. Morello is a localisation to AArch64 and Arm has produced a few thousand test chips / dev systems based on the Neoverse N1 for software development and experimentation. This is what I have under my desk: a modified quad-core Neoverse N1 running at 2.5GHz with 16 GiB of RAM in a desktop chassis with a 250 GiB SSD. We also have a load of them in a rack for CI (snmalloc CI on Morello and benchmarking.
If all goes well, I expect to see CHERI extensions on mainstream architectures on the next few years and so developing a compiler toolchain based on an abstraction that can’t possibly support them without significant rearchitecting seems like an unfortunate decision, especially when maintaining a separate pointer type is fairly simple if you design it in from scratch. The fact that LLVM had separate integer and pointer types in IR made the original CHERI work feasible, the fact that it loses that distinction by the time it reaches SelectionDAG and the back end (one of the first questions the target-agnostic code generator asks the back end is ‘so, which integer type do you want to use for pointers?’) made it harder.
Thanks for the summary, very cool stuff. Kudos for pushing this long.
Discussions from around the Morello announcement: https://lobste.rs/s/w32bav/morello_arm_cheri_prototype_hits_major <- I recommend the Microsoft article https://lobste.rs/s/wqts1n/capability_hardware_enhanced_risc
CHERI is (I think) both a security model and set of hypothetical and/or experimental extensions for multiple ISA including MIPS, ARM, RISC-V, and x86 the latter of which is currently just a “sketch” [1]
Morello is the realization of actual silicon implementing the extensions [2]
As for an actual description I’d rather point you towards the Microsoft article (I honestly really liked it). That and the discussions were what painted most of my picture of the project(s). There’s also the technical report An Introduction to CHERI which helped fill in other details but there were things or referenced concepst I wasn’t clear on.
[1] https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-faq.html:
[2] https://msrc-blog.microsoft.com/2022/01/20/an_armful_of_cheris/
Thank you. I’ll put the MS article on a reading list, sounds very interesting.
By GPU do you mean the Panfrost port br@ is working on, or an amdgpu in a PCIe slot? How’s the PCIe situation on Morello?
The panfrost bit. There are some PCIe slots, but I’ve not tried plugging anything into them yet. We’re hoping to set some of them up with some RDMA-capable smartNICs and see if we can do something interesting eventually.
Would be very interesting to try amdgpu :)
There’s a “Users” tab at the top that lists cproc, hare and others: https://c9x.me/compile/users.html
Right there in the menu as well! Thank you for pointing this out to me. I think I had assumed this would be something like a community page (eg “user mailing list”).
So can you link against it yet?
Not sure what you mean? QBE is a standalone program that chokes down some SSA and spits out a binary. There’s nothing to link against.
Yes, which makes it unpalatable as a LLVM replacement. People have been asking for a way to link against QBE and feed it SSA directly for a while.
At that level of integration, maybe a JIT library like GNU Lightning would make more sense. I like QBE’s commitment to the text API, because it means that I don’t have to write my compilers in languages with strong C/C++ FFI.
I think LLVM demonstrates that it’s possible to have a commitment to both text and C APIs at once.