1. 9

  2. 6

    I’ve went through similar gymnastics with Docker, but finally Hetzner has ARM-based VPSes and I don’t need to fiddle with cross-compilation or suffer qemu speeds!

    1. 1

      Awesome, the prices seem quite reasonable too! Thanks for mentioning this. I might give it a go in the future!

    2. 5

      Highly recommend checking cargo-zigbuild, I’ve been using it indirectly thru maturin for making Python wheels from a Mac Silicon to x86_64 Linux and it just works. It is also way faster than QEMU or the experimental Rosetta2/vz support in colima.

      (Is it silly to build a Python wheel containing a Rust binary with C++ deps cross-compiled with Zig? Yes. Is it amazing that it all works? Also yes. =P)

      1. 2

        Thank you so much! I should try to integrate this in my workflow ;) I am particularly interested in the speed gain. Compiling the project takes several minutes!

      2. 3

        I don’t understand what value Docker is providing to the build process here – you’re using Rust (a native cross-compiler), have no C dependencies (after replacing OpenSSL with rustls), and don’t even need any system headers. Why not build the binary directly in macOS? This should be as simple as setting the --target flag to Cargo and maybe using lld for linking (https://john-millikin.com/notes-on-cross-compiling-rust#rustup-and-cargo).

        The presence(?) of QEMU in the build is also very confusing. You’re running Docker on macOS, which implies there’s already one VM involved, and then within that VM you configure Cargo to do … something … with QEMU. But you’re not installing QEMU! And why would you need to run QEMU anyway, when the Docker VM is already configured to use the amd64 platform?

        Unless there’s additional complexity not mentioned here (C/C++ dependencies), I suspect the following dockerfile fragment (untested) would work just as well:

        FROM rust:1.68.2-slim-buster as backend
        RUN rustup target add x86_64-unknown-linux-musl
        COPY backend /app/backend
        RUN cd /app/backend && cargo build --target x86_64-unknown-linux-musl --release
        1. 2

          I was also confused by the qemu bit. As I understand it, Docker on Apple’s Arm platforms can already launch x86 VMs that run x86 containers with the provided Rosetta emulator. Why do you need anything else? As long as you start with an x86-64 base layer, everything should just work.

          Clang is also a native cross-compiler, so you can even build C/C++ libraries if you install the right sysroot.

          It often is easier to build across architectures than across platforms, so an Arm Linux container might be simpler than cross building from Mac, but this all looks like a lot of complexity if you’re already buying into a stack that solves these problems for you.

          1. 1

            Thanks, David.

            That’s a great observation but I will eventually have to hand over the code to another team and I’d like to give them a generic build process that should work on other machines as well with minimal effort… That’s why I am trying to have a Docker-based build and not trying to build directly from Mac.

          2. 1

            Thanks a lot for the updated snippet. I am giving it a try and if it works I’d be really happy with it.

            I guess there’s one piece of detail that I left out from the story. The reason why I’d like to do a build directly from Docker (even if it’s not necessarily the most optimal solution on Mac), is that I will eventually have to hand over the code to another team and I’d like to give them a generic build process that should work on other machines as well with minimal effort…

            1. 1

              Now I remember why I ended up playing around with those compilation settings (from Sylvain’s article).

              When building without all the additional settings I was getting an error compiling the ring crate. WIth your suggested changes that error pops up again.

              here it is:

              error: failed to run custom build command for `ring v0.16.20`
               Caused by:
                 process didn't exit successfully: `/app/backend/target/release/build/ring-82edbdc9e3773afb/build-script-build` (exit status: 101)
                 --- stdout
                 OPT_LEVEL = Some("3")
                 TARGET = Some("x86_64-unknown-linux-musl")
                 HOST = Some("x86_64-unknown-linux-gnu")
                 CC_x86_64-unknown-linux-musl = None
                 CC_x86_64_unknown_linux_musl = None
                 TARGET_CC = None
                 CC = None
                 RUSTC_LINKER = None
                 CROSS_COMPILE = None
                 CFLAGS_x86_64-unknown-linux-musl = None
                 CFLAGS_x86_64_unknown_linux_musl = None
                 TARGET_CFLAGS = None
                 CFLAGS = None
                 CRATE_CC_NO_DEFAULTS = None
                 DEBUG = Some("false")
                 CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
                 --- stderr
                 running "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/app/backend/target/x86_64-unknown-linux-musl/release/build/ring-a71171d6e7e315a3/out/aesni-x86_64-elf.o" "/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"
                 thread 'main' panicked at 'execution failed', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
                 stack backtrace:
                    0:       0x400009f46a - std::backtrace_rs::backtrace::libunwind::trace::ha271a8a7e1f3d4ef
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
                    1:       0x400009f46a - std::backtrace_rs::backtrace::trace_unsynchronized::h85739da0352c791a
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
                    2:       0x400009f46a - std::sys_common::backtrace::_print_fmt::hbc6ebcfb2910b329
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:65:5
                    3:       0x400009f46a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he1c117e52d53614f
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:44:22
                    4:       0x40000bfefe - core::fmt::write::h25eb51b9526b8e0c
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/fmt/mod.rs:1213:17
                    5:       0x400009bd05 - std::io::Write::write_fmt::ha9edec5fb1621933
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/io/mod.rs:1682:15
                    6:       0x400009f235 - std::sys_common::backtrace::_print::hf8657cd429fc3452
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:47:5
                    7:       0x400009f235 - std::sys_common::backtrace::print::h41b9b18ed86f86bd
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:34:9
                    8:       0x40000a0c3f - std::panicking::default_hook::{{closure}}::h22a91871f4454152
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:267:22
                    9:       0x40000a097b - std::panicking::default_hook::h21ddc36de0cd4ae7
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:286:9
                   10:       0x40000a1349 - std::panicking::rust_panic_with_hook::h5059419d6d59b3d0
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:688:13
                   11:       0x400006c068 - std::panicking::begin_panic::{{closure}}::hbdda51bf5702d725
                   12:       0x4000076066 - std::sys_common::backtrace::__rust_end_short_backtrace::he40d767cc972d93c
                   13:       0x400006bfb3 - std::panicking::begin_panic::h54e9da36f8578628
                   14:       0x400001f73c - build_script_build::run_command::hb0174b05994d124c
                   15:       0x400001e239 - build_script_build::compile::h0b91dbbc6aba3d9d
                   16:       0x400001dd3e - build_script_build::build_library::{{closure}}::h3a0c97b7c18c7b6b
                   17:       0x400001b1db - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once::he7807d09707bf493
                   18:       0x4000019607 - core::option::Option<T>::map::h4e7418d1cdb58b81
                   19:       0x40000250d4 - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::hb8a3136a9ebf4e25
                   20:       0x4000014108 - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter::h586aa38910b7932e
                   21:       0x4000016b9a - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter::hfb00823396d26eae
                   22:       0x40000167ce - <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter::h26d56c8c746760c1
                   23:       0x400002562a - core::iter::traits::iterator::Iterator::collect::h0eea12240674d1b3
                   24:       0x400001d4b5 - build_script_build::build_library::hf907256320d461f7
                   25:       0x400001d2de - build_script_build::build_c_code::{{closure}}::hb344d68bc4ed763e
                   26:       0x400001807d - <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::for_each::hba0d7b51fcea2959
                   27:       0x400001cede - build_script_build::build_c_code::h6b092e7c2024544e
                   28:       0x400001bb30 - build_script_build::ring_build_rs_main::h2e93932be9c027f0
                   29:       0x400001b38d - build_script_build::main::h83a23c78895f021c
                   30:       0x4000026da3 - core::ops::function::FnOnce::call_once::hf7cf10f12fa9e5c9
                   31:       0x4000018ab9 - std::sys_common::backtrace::__rust_begin_short_backtrace::hb794a24425c90e55
                   32:       0x400001a959 - std::rt::lang_start::{{closure}}::h6740db5e99f0b233
                   33:       0x400009817c - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h203afb3af230319a
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/function.rs:287:13
                   34:       0x400009817c - std::panicking::try::do_call::hf68e87013b70f3c5
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:483:40
                   35:       0x400009817c - std::panicking::try::h040ea8f298390ba2
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:447:19
                   36:       0x400009817c - std::panic::catch_unwind::h1e17b198887a05fa
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panic.rs:140:14
                   37:       0x400009817c - std::rt::lang_start_internal::{{closure}}::hfb902d8927e51b86
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/rt.rs:148:48
                   38:       0x400009817c - std::panicking::try::do_call::h354e6eb41f2e7d42
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:483:40
                   39:       0x400009817c - std::panicking::try::h4a39749cd018228c
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:447:19
                   40:       0x400009817c - std::panic::catch_unwind::h30bce83b8de61cca
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panic.rs:140:14
                   41:       0x400009817c - std::rt::lang_start_internal::h8f7e70b1a2558118
                                                at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/rt.rs:148:20
                   42:       0x400001a937 - std::rt::lang_start::he1cca5ad024e176b
                   43:       0x40000215f5 - main
                   44:       0x40019a409b - __libc_start_main
                   45:       0x4000013bea - _start
                   46:                0x0 - <unknown>
              1. 2

                That’s an impressively long error message for what I would have expected to be a straightforward dependency. I looked into ring/build.rs and discovered they’ve continued OpenSSL’s use of “PerlAsm”: https://github.com/briansmith/ring/blob/main/crypto/fipsmodule/modes/asm/aesni-gcm-x86_64.pl

                Here’s a slightly less reduced version of your Dockerfile that builds on my desktop. Whether you use it or keep what you currently have, I recommend writing a nice big comment so that future readers will have a head start on understanding some of this build’s hidden depths.

                FROM rust:1.68.2-slim-buster as backend
                RUN rustup target add x86_64-unknown-linux-musl
                COPY backend /app/backend
                # Dependency `ring` requires a cross-compiler for bundled C/C++
                # sources, and may require Perl for some target platforms.
                RUN apt update && apt install -y --no-install-recommends clang llvm perl
                ENV CC_x86_64_unknown_linux_musl=clang
                RUN cd /app/backend && cargo build --target x86_64-unknown-linux-musl --release
                1. 1

                  Awesome! This worked, thanks a lot, I really appreciate the help.

                  I’ll certainly update the blog post with this new option (which seems much cleaner, and faster to me)

          3. 2

            100MB is still pretty large to hold a single binary. There might be another 8-10x potential size gain in there.

            1. 2

              Good point, but I didn’t mention that this container has a bunch of frontend assets (probably around 8MBs) and a GeoIP DB embedded in it (about 70MBs). I think these are the ones taking the bulk of the space (outside the rust binary)…

              1. 1

                Now added a section in the article. Thanks a lot for the inspiration :)