I went through this exact process recently. I’ve built a run from RAM image for a Raspberry Pi Zero W with buildroot. I worked to slim it down on the weekend from 35Mb (including wpa_supplicant and my custom application) to 12Mb. There’s probably more bits of the kernel I could turn off but I spent enough time on reducing the size already.
I’ve recently been doing some work with Poudriere’s image building mode. Until recently I was completely unaware that this existed. Poudriere is normally used to build FreeBSD package sets but it can also build the base system, including using a custom build configuration, and can build various forms of disk image (or image-like things, including ZFS snapshots or tarballs). I’ve been extending it to be able to build ZFS disk images that are bootable with both BIOS and UEFI, that I can deploy into Azure.
It provides a single tool that can build a custom configuration of the base system, packages with custom options, and then generate an image that contains both plus anything else you want to add via an overlay, so you can use it for building VM images, bootable USB images, or things like Raspberry Pi images.
I’d love to see it extended to be able to build OCI container images in the same way.
I’m doing extremely similar stuff with FreeBSD, but not using Poudriere. I wrote my own “image compiler” that takes its configuration from files specifying, among other things, the custom kernel conf to use, the apps to include, 3rd party (as git submodules) projects to additionally deploy via make/cmake/whatever (if they’re not in the FreeBSD ports repo), Python modules to install out-of-band from ports/pkg, and scripts to run at startup. It uses ZFS to accelerate the build process by creating temporary ZFS snapshots to use a CoW “savepoints” so changes to one layer don’t require a rebuild of the previous layers. On one hand, I generate ISO images from the pipeline that get distributed to customers to boot into a live environment, and on the other hand I generate images that can be run as FreeBSD-based alternatives to AWS Lambda in response to scripted hooks.
This would all be so much easier if FreeBSD actually had a working overlay fs; all the current options are decrepit garbage and either extremely unstable or barely worth using (because they can’t poke holes to the lower layer).
To be the stereotypical Lobste.rs Nix enthusiast, you can also do this sort of thing with Nix, though it might require extra work to slim down the closure size.
Maybe I’ll try seeing how small of a NixOS image I can make as a weekend project.
EDIT: Hm, this depends on whether you can actually build a system on musl, systemd won’t work so it would have to use a different init system which complicates things. It looks like nixwrt is trying this sort of thing - it’s not actually NixOS, but that doesn’t really matter since it has different goals than NixOS proper.
A basic install of Alpine Linux, which uses musl libc, is something around 8MB (IIRC, but without the kernel). It uses openrc for the init system and busybox.
edit: the alpine docker container page claims the rootfs is 5MB: https://hub.docker.com/_/alpine …
See also, possibly, https://github.com/cleverca22/not-os; I’m not sure how much that one fits the bill, but at least some parts may be steal-… uhm, I mean, reuse-worthy. I’m pretty sure I also stumbled upon some other attempt at bootstrapping an alternative to Nixpkgs for “embedded-like” environments; but I can’t seem to be able to find it in my GH stars nor bookmarks 😞 (I think it was something else yet than nixwrt; or was it? 🤔)