I’ve went through similar gymnastics with Docker, but finally Hetzner has ARM-based VPSes and I don’t need to fiddle with cross-compilation or suffer qemu speeds!
Highly recommend checking cargo-zigbuild, I’ve been using it indirectly thru maturin for making Python wheels from a Mac Silicon to x86_64 Linux and it just works. It is also way faster than QEMU or the experimental Rosetta2/vz support in colima.
(Is it silly to build a Python wheel containing a Rust binary with C++ deps cross-compiled with Zig? Yes. Is it amazing that it all works? Also yes. =P)
Thank you so much! I should try to integrate this in my workflow ;) I am particularly interested in the speed gain. Compiling the project takes several minutes!
I don’t understand what value Docker is providing to the build process here – you’re using Rust (a native cross-compiler), have no C dependencies (after replacing OpenSSL with rustls), and don’t even need any system headers. Why not build the binary directly in macOS? This should be as simple as setting the --target flag to Cargo and maybe using lld for linking (https://john-millikin.com/notes-on-cross-compiling-rust#rustup-and-cargo).
The presence(?) of QEMU in the build is also very confusing. You’re running Docker on macOS, which implies there’s already one VM involved, and then within that VM you configure Cargo to do … something … with QEMU. But you’re not installing QEMU! And why would you need to run QEMU anyway, when the Docker VM is already configured to use the amd64 platform?
Unless there’s additional complexity not mentioned here (C/C++ dependencies), I suspect the following dockerfile fragment (untested) would work just as well:
FROM rust:1.68.2-slim-buster as backend
RUN rustup target add x86_64-unknown-linux-musl
COPY backend /app/backend
RUN cd /app/backend && cargo build --target x86_64-unknown-linux-musl --release
I was also confused by the qemu bit. As I understand it, Docker on Apple’s Arm platforms can already launch x86 VMs that run x86 containers with the provided Rosetta emulator. Why do you need anything else? As long as you start with an x86-64 base layer, everything should just work.
Clang is also a native cross-compiler, so you can even build C/C++ libraries if you install the right sysroot.
It often is easier to build across architectures than across platforms, so an Arm Linux container might be simpler than cross building from Mac, but this all looks like a lot of complexity if you’re already buying into a stack that solves these problems for you.
That’s a great observation but I will eventually have to hand over the code to another team and I’d like to give them a generic build process that should work on other machines as well with minimal effort… That’s why I am trying to have a Docker-based build and not trying to build directly from Mac.
Thanks a lot for the updated snippet. I am giving it a try and if it works I’d be really happy with it.
I guess there’s one piece of detail that I left out from the story. The reason why I’d like to do a build directly from Docker (even if it’s not necessarily the most optimal solution on Mac), is that I will eventually have to hand over the code to another team and I’d like to give them a generic build process that should work on other machines as well with minimal effort…
Now I remember why I ended up playing around with those compilation settings (from Sylvain’s article).
When building without all the additional settings I was getting an error compiling the ring crate. WIth your suggested changes that error pops up again.
Here’s a slightly less reduced version of your Dockerfile that builds on my desktop. Whether you use it or keep what you currently have, I recommend writing a nice big comment so that future readers will have a head start on understanding some of this build’s hidden depths.
FROM rust:1.68.2-slim-buster as backend
RUN rustup target add x86_64-unknown-linux-musl
COPY backend /app/backend
# Dependency `ring` requires a cross-compiler for bundled C/C++
# sources, and may require Perl for some target platforms.
RUN apt update && apt install -y --no-install-recommends clang llvm perl
ENV CC_x86_64_unknown_linux_musl=clang
RUN cd /app/backend && cargo build --target x86_64-unknown-linux-musl --release
Good point, but I didn’t mention that this container has a bunch of frontend assets (probably around 8MBs) and a GeoIP DB embedded in it (about 70MBs). I think these are the ones taking the bulk of the space (outside the rust binary)…
I’ve went through similar gymnastics with Docker, but finally Hetzner has ARM-based VPSes and I don’t need to fiddle with cross-compilation or suffer qemu speeds!
Awesome, the prices seem quite reasonable too! Thanks for mentioning this. I might give it a go in the future!
Highly recommend checking cargo-zigbuild, I’ve been using it indirectly thru maturin for making Python wheels from a Mac Silicon to x86_64 Linux and it just works. It is also way faster than QEMU or the experimental Rosetta2/vz support in colima.
(Is it silly to build a Python wheel containing a Rust binary with C++ deps cross-compiled with Zig? Yes. Is it amazing that it all works? Also yes. =P)
Thank you so much! I should try to integrate this in my workflow ;) I am particularly interested in the speed gain. Compiling the project takes several minutes!
I don’t understand what value Docker is providing to the build process here – you’re using Rust (a native cross-compiler), have no C dependencies (after replacing OpenSSL with rustls), and don’t even need any system headers. Why not build the binary directly in macOS? This should be as simple as setting the
--targetflag to Cargo and maybe usinglldfor linking (https://john-millikin.com/notes-on-cross-compiling-rust#rustup-and-cargo).The presence(?) of QEMU in the build is also very confusing. You’re running Docker on macOS, which implies there’s already one VM involved, and then within that VM you configure Cargo to do … something … with QEMU. But you’re not installing QEMU! And why would you need to run QEMU anyway, when the Docker VM is already configured to use the
amd64platform?Unless there’s additional complexity not mentioned here (C/C++ dependencies), I suspect the following dockerfile fragment (untested) would work just as well:
I was also confused by the qemu bit. As I understand it, Docker on Apple’s Arm platforms can already launch x86 VMs that run x86 containers with the provided Rosetta emulator. Why do you need anything else? As long as you start with an x86-64 base layer, everything should just work.
Clang is also a native cross-compiler, so you can even build C/C++ libraries if you install the right sysroot.
It often is easier to build across architectures than across platforms, so an Arm Linux container might be simpler than cross building from Mac, but this all looks like a lot of complexity if you’re already buying into a stack that solves these problems for you.
Thanks, David.
That’s a great observation but I will eventually have to hand over the code to another team and I’d like to give them a generic build process that should work on other machines as well with minimal effort… That’s why I am trying to have a Docker-based build and not trying to build directly from Mac.
Thanks a lot for the updated snippet. I am giving it a try and if it works I’d be really happy with it.
I guess there’s one piece of detail that I left out from the story. The reason why I’d like to do a build directly from Docker (even if it’s not necessarily the most optimal solution on Mac), is that I will eventually have to hand over the code to another team and I’d like to give them a generic build process that should work on other machines as well with minimal effort…
Now I remember why I ended up playing around with those compilation settings (from Sylvain’s article).
When building without all the additional settings I was getting an error compiling the
ringcrate. WIth your suggested changes that error pops up again.here it is:
That’s an impressively long error message for what I would have expected to be a straightforward dependency. I looked into
ring/build.rsand discovered they’ve continued OpenSSL’s use of “PerlAsm”: https://github.com/briansmith/ring/blob/main/crypto/fipsmodule/modes/asm/aesni-gcm-x86_64.plHere’s a slightly less reduced version of your
Dockerfilethat builds on my desktop. Whether you use it or keep what you currently have, I recommend writing a nice big comment so that future readers will have a head start on understanding some of this build’s hidden depths.Awesome! This worked, thanks a lot, I really appreciate the help.
I’ll certainly update the blog post with this new option (which seems much cleaner, and faster to me)
100MB is still pretty large to hold a single binary. There might be another 8-10x potential size gain in there.
Good point, but I didn’t mention that this container has a bunch of frontend assets (probably around 8MBs) and a GeoIP DB embedded in it (about 70MBs). I think these are the ones taking the bulk of the space (outside the rust binary)…
Now added a section in the article. Thanks a lot for the inspiration :)