Zig’s cross-compilation story is the best I’ve ever seen. It’s so good I didn’t even think it would be possible. Even if Zig-the-language never gains any traction (which would be a tragedy), Zig-the-toolchain is already fantastic and will be around for a long time.
Go’s is good, don’t get me wrong, but Zig solves a much harder problem and does it so amazingly seamlessly.
To be honest, the difficulty of cross compilation is something I have never really understood. A compiler takes source code written in some human readable formalism, and produces binary code in some machine readable formalism. That is it. It’s frankly baffling, and a testament to a decades long failure of our whole industry, that “cross compilation” is even a word: it is after all just like compilation: source code in, machine code out. We just happen to produce machine code for other systems than the one that happens to host the compiler.
I see only two ways “cross” compilation can ever be a problem: limited access to target specific source code, and limited access to the target platform’s specifications. In both cases, it looks to me like a case of botched dependency management: we implicitly depend on stuff that vary from platform to platform, and our tools are too primitive or too poorly designed to make those dependencies explicit so we can change them (like, depending on the target platform’s headers and ABI instead of the compiler’s platform).
I would very much like to know what went wrong there. Why is it so hard to statically link the C standard library? Why do Windows programs need VCRedists? Can’t a program just depend on it’s OS’s kernel? (Note: I know the security and bloat arguments in favour of dynamic linking. I just think solving dependency hell is more important.)
Good grief, glibc is insane. What it does under the hood is supposed to be an implementation detail, and really should not be affected by linking strategy. Now, this business about locales may be rather tricky; maybe the standard painted them into a corner: from the look of it, a C standard library may have to depend on more than the kernel¹ to fully implement itself. And if one of those dependencies does not have a stable interface, we’re kinda screwed.
When I write a program, I want a stable foundation to ship on. It’s okay if I have to rewrite the entire software stack to do it, as long as I have stable and reliable ways to make the pixels blinks and the speaker bleep. Just don’t force me to rely on flaky dependencies.
[1]: The kernel’s userspace interface (system calls) is very stable. The stackoverflow page you link to suggests otherwise, but I believe they’re talking about the kernel interface, which was never considered stable (resulting in drivers having to be revised every time there’s a change).
It’s worth noting (since your question was about generic cross-platform cross-compilation, and you mentioned e.g. Windows) that this comment:
The kernel’s userspace interface (system calls) is very stable.
is only really true for Linux among the mainstream operating systems. In Solaris, Windows, macOS, and historically the BSDs (although that may’ve changed), the official, and only stable, interface to the kernel, is through C library calls. System calls are explicitly not guaranteed to be stable, and (at least on Windows and Solaris, with which I’m most familiar) absolutely are not: a Win32k or Solaris call that’s a full-on user-space library function in one release may be a syscall in the next, and two separate syscalls in the release after that. This was a major, major issue with how Go wanted to do compilation early on, because it wanted The Linux Way to be the way everywhere, when in fact, Linux is mostly the odd one out. Nowadays, Go yields to the core C libraries as appropriate.
As long as I have some stable interface, I’m good. It doesn’t really matter where the boundary is exactly.
Though if I’m being honest, it kinda does: for instance, we want interfaces to be small, so they’re easier to stabilise and easier to learn. So we want to find the natural boundaries between applications and the OS, and put the stable interface there. It doesn’t have to be the kernel to be honest.
I agree it is over complicated, but it isn’t as simple as you are saying.
One answer is because many tools want to run code during the build process, so they need both compilers and a way to distinguish between the build machine and target machine. this does not need to be complicated, but immediately breaks your idealized world view.
Another answer is our dependency management tools are so poor it is not easy to setup the required libraries to link the program for the target.
many tools want to run code during the build process
Like, code we just compiled? I see two approaches to this. We could reject the concept altogether, and cleanly separate the build process itself, that happens exclusively on the source platform from, tests, that happen exclusively on the target platform. Or, we could have a portable bytecode compiler and interpreter, same as Jonathan Blow does with his language. I personally like going the bytecode route, because it make it easier to have a reference implementation you can compare to various backends.
a way to distinguish between the build machine and target machine.
As far as I understand, we only need a way to identify the target machine. The build machine is only relevant insofar as it must run the compiler and associated tools. Now I understand how that alone might be a problem: Microsoft is not exactly interested in running MSVC on Apple machines… Still, you get the idea.
Another answer is our dependency management tools are so poor it is not easy to setup the required libraries to link the program for the target.
There are two common cases of this. The first is really a bug in the build system: try to compile something, run it, examine its behaviour, and use that to configure the build. This breaks even the really common cross-compilation use case of trying to build something that will run on a slightly older version of the current system. Generally, these should be rewritten as either try-compile tests or run-time configurable behaviour.
The more difficult case is when you have some build tools that are built as part of the compilation. The one that I’m most familiar with is LLVM’s TableGen tool. To make LLVM support cross compilation, they needed to first build this for the host, then use it to generate the files that are compiled for the target, then build it again for the target (because downstream consumers also use it). LLVM is far from the only project that generates a tool like this, but it’s one of the few that properly manages cross compilation.
Oh, so that what you meant by distinguishing the build platform from the target platform. You meant distinguishing what will be build for the host platform (because we need it to further the build process) from the final artefacts. Makes sense.
Another example would be something like build.rs in rust projects, though that seems less likely to cause problems. The linux kernel build also compiles some small C utilities that it then uses during the build so they have HOSTCC as well as CC.
On its surface this sounds preposterous. Can you elaborate? I know of maybe a dozen self-hosted languages since Pascal so I think I must be misunderstanding.
Edit: I’m guessing you mean that not only is the compiler self-hosted, but every last dependency of the compiler and runtime (outside the kernel I guess?) is also written in the language? That is a much more limited set of languages (still more than zero) but it’s not the commonly accepted meaning of self-hosted.
The original Project Oberon kernel was written in assembly, but the newer version is written almost entirely in Oberon.
Some of the early Smalltalks were written almost entirely in Smalltalk, with a weird syntactic subset that had limited semantics but compatible syntax that could be compiled to machine code.
And of course LISP machines, where “garbage collection” means “memory management.”
It’s an interesting distinction even if the terminology isn’t what I’d use. There’s a trend right now among languages to hop on an existing runtime because rebuilding an entire ecosystem from first principles is exhausting, especially if you want to target more than one OS/architecture combo. Sometimes it’s a simple as just “compile to C and benefit from the existing compilers and tools for that language”. But it seems fitting that we should have a way to describe those systems which take the harder route; I just don’t know what the word would be.
limited access to the target platform’s specifications. […] it looks to me like a case of botched dependency management
This is exactly what’s going on. You need to install the target platform’s specifications in an imperative format (C headers), and it’s the only format they provide.
And it makes extreme assumptions about file system layout, which are all necessarily incorrect because you’re not running on that platform.
Zig compiles Zig and C from almost any platform to almost any platform pretty seamlessly.
As I understand it, Zig doesn’t do much more than clang does out of the box. With clang + lld, you can just provide a directory containing the headers and libraries for your target with --sysroot= and specify the target with -target. Clang will then happily cross-compile anything that you throw at it. Zig just ships a few sysroots pre-populated with system headers and libraries. It’s still not clear to me that this is legal for the macOS ones, because the EULA for most of them explicitly prohibits cross compiling, though it may be fine if everything is built from the open source versions.
This is not the difficult bit. It’s easy if your only dependency is the C standard library but most non-trivial programs have other dependencies. There are two difficult bits:
Installing the dependencies into your sysroot.
Working around build systems that don’t support cross compilation and so try to compile and run things and dependencies that are compiler-like.
The first is pretty easy to handle if you are targeting an OS that distributes packages as something roughly equivalent to tarballs. On FreeBSD, for example, every package is just a txz with some metadata in it. You can just extract these directly into your sysroot. RPMs are just cpio archives. I’ve no idea what .deb files are, but probably something similar. Unfortunately, you are still responsible for manually resolving dependencies. It would be great if these tools supported installing into a sysroot directly.
The second is really hard. For example, LLVM builds a tablegen tool that generates C++ files from a DSL. LLVM’s build system supports cross compilation and so will first build a native tablegen and then use that during the build. If you’re embedding LLVM’s cmake, you have access to this. If you have just installed LLVM in a sysroot and want to cross-build targeting it then you also need to find the host tablegen from somewhere else. The same is true of things like the Qt preprocessor and a load of other bits of tooling. This is on top of build systems that detect features by trying to compile and run something at build time - this is annoying, but at least doesn’t tend to leak into downstream dependencies. NetBSD had some quite neat infrastructure for dealing with these by running those things in QEMU user mode while still using host-native cross-compile tools for everything else.
As I understand it, Zig doesn’t do much more than clang does out of the box. With clang + lld, you can just provide a directory containing the headers and libraries for your target with –sysroot= and specify the target with -target. Clang will then happily cross-compile anything that you throw at it. Zig just ships a few sysroots pre-populated with system headers and libraries.
That’s what it does but to say that it “isn’t much more than what clang does out of the box” is a little disingenuous. It’s like saying a Linux distro just “packaged up software that’s already there.” Of course that’s ultimately what it is, but there’s a reason why people use Debian and Fedora and not just Linux From Scratch everywhere. That “isn’t much more” is the first time I’ve seen it done so well.
It solves the trivial bit of the problem: providing a sysroot that contains libc, the CSU bits, and the core headers. It doesn’t solve the difficult bit: extending the sysroot with the other headers and libraries that non-trivial programs depend on. The macOS version is a case in point. It sounds as if it is only distributing the headers from the open source Apple releases, but that means that you hit a wall as soon as you want to link against any of the proprietary libraries / frameworks that macOS ships with. At that point, the cross-compile story suddenly stops working and now you have to redo all of your build infrastructure to always do native compilation for macOS.
The usual problem encountered when cross-compiling from a non-macOS system to macOS is you need the macOS headers and it’s against the licence agreement to redistribute them or even use them on non-Apple hardware:
You may not alter the Apple Software or Services in any way in such copy, e.g., You are expressly prohibited from separately using the Apple SDKs or attempting to run any part of the Apple Software on non-Apple-branded hardware.
How does Zig handle this?
Edit: having said that, this repo has existed for a long time and hasn’t been taken down yet…
That’s the one. If I recall correctly, Google originally lost, then appealed, and the ruling was basically reversed to “interfaces are not subject to copyright”.
Now that was American law. I have no idea about the rest of the world. I do believe many legislations have explicit exceptions for interoperability, though.
That’s the one. If I recall correctly, Google originally lost, then appealed, and the ruling was basically reversed to “interfaces are not subject to copyright”.
The Supreme Court judgement said ‘assume interfaces are copyrightable, in this case Oracle still loses’ it did not make a ruling on whether interfaces are copyrightable.
and the ruling was basically reversed to “interfaces are not subject to copyright”
Not exactly, the ruling didn’t want to touch the “interfaces are not subject to copyright” matter since that would open a big can of worms. What it did say, however, was that Google’s specific usage of those interfaces fell into the fair use category.
Ah, so in the case of Zig, it would also be fair use, but since fair use is judged on a case by case basis, there’s still some uncertainty. Not ideal, though it looks like it should work.
There’s no useful precedent. Google’s fair use was from an independent implementation of an interface for compatibility. Zig is copying header files directly and so must comply with the licenses for them. The exact licenses that apply depend on whether you got the headers from the open source code dump or by agreeing to the XCode EULA. A lot of the system headers for macOS / iOS are only available if you agree to the XCode EULA, which prohibits compilation on anything other than an Apple-branded system.
Java doesn’t have any analogue of .h files, they wrote new .java files that implemented the same methods. There is a difference between creating a new .h file that contains equivalent definitions and copying a .h file that someone else wrote. If interfaces are not copyrightable, then the specific serialisation in a text file may still be because it may contain comments and other things that are not part of the interface.
I’d be super nervous about using this in production. This is using code under the Apple Public Source License, which explicitly prohibits using it to circumvent EULAs of Apple products. The XCode EULA under which the SDKs are prohibited explicitly prohibits cross-compiling from a non-Apple machine. I have no idea what a judge would decide, but I am 100% sure that Apple can afford to hire better lawyers than I can.
Zig has nothing to do with xcode. Zig does not depend on xcode or use xcode in any way. The macos headers have to do with interfacing with the Darwin kernel.
Zig’s cross-compilation story is the best I’ve ever seen. It’s so good I didn’t even think it would be possible. Even if Zig-the-language never gains any traction (which would be a tragedy), Zig-the-toolchain is already fantastic and will be around for a long time.
Go’s is good, don’t get me wrong, but Zig solves a much harder problem and does it so amazingly seamlessly.
To be honest, the difficulty of cross compilation is something I have never really understood. A compiler takes source code written in some human readable formalism, and produces binary code in some machine readable formalism. That is it. It’s frankly baffling, and a testament to a decades long failure of our whole industry, that “cross compilation” is even a word: it is after all just like compilation: source code in, machine code out. We just happen to produce machine code for other systems than the one that happens to host the compiler.
I see only two ways “cross” compilation can ever be a problem: limited access to target specific source code, and limited access to the target platform’s specifications. In both cases, it looks to me like a case of botched dependency management: we implicitly depend on stuff that vary from platform to platform, and our tools are too primitive or too poorly designed to make those dependencies explicit so we can change them (like, depending on the target platform’s headers and ABI instead of the compiler’s platform).
I would very much like to know what went wrong there. Why is it so hard to statically link the C standard library? Why do Windows programs need VCRedists? Can’t a program just depend on it’s OS’s kernel? (Note: I know the security and bloat arguments in favour of dynamic linking. I just think solving dependency hell is more important.)
Well, because glibc… Maybe musl will save us 😅
If you really want to go down that rabbit hole: https://stackoverflow.com/a/57478728
Good grief, glibc is insane. What it does under the hood is supposed to be an implementation detail, and really should not be affected by linking strategy. Now, this business about locales may be rather tricky; maybe the standard painted them into a corner: from the look of it, a C standard library may have to depend on more than the kernel¹ to fully implement itself. And if one of those dependencies does not have a stable interface, we’re kinda screwed.
When I write a program, I want a stable foundation to ship on. It’s okay if I have to rewrite the entire software stack to do it, as long as I have stable and reliable ways to make the pixels blinks and the speaker bleep. Just don’t force me to rely on flaky dependencies.
[1]: The kernel’s userspace interface (system calls) is very stable. The stackoverflow page you link to suggests otherwise, but I believe they’re talking about the kernel interface, which was never considered stable (resulting in drivers having to be revised every time there’s a change).
It’s worth noting (since your question was about generic cross-platform cross-compilation, and you mentioned e.g. Windows) that this comment:
is only really true for Linux among the mainstream operating systems. In Solaris, Windows, macOS, and historically the BSDs (although that may’ve changed), the official, and only stable, interface to the kernel, is through C library calls. System calls are explicitly not guaranteed to be stable, and (at least on Windows and Solaris, with which I’m most familiar) absolutely are not: a Win32k or Solaris call that’s a full-on user-space library function in one release may be a syscall in the next, and two separate syscalls in the release after that. This was a major, major issue with how Go wanted to do compilation early on, because it wanted The Linux Way to be the way everywhere, when in fact, Linux is mostly the odd one out. Nowadays, Go yields to the core C libraries as appropriate.
As long as I have some stable interface, I’m good. It doesn’t really matter where the boundary is exactly.
Though if I’m being honest, it kinda does: for instance, we want interfaces to be small, so they’re easier to stabilise and easier to learn. So we want to find the natural boundaries between applications and the OS, and put the stable interface there. It doesn’t have to be the kernel to be honest.
I agree it is over complicated, but it isn’t as simple as you are saying.
One answer is because many tools want to run code during the build process, so they need both compilers and a way to distinguish between the build machine and target machine. this does not need to be complicated, but immediately breaks your idealized world view.
Another answer is our dependency management tools are so poor it is not easy to setup the required libraries to link the program for the target.
Like, code we just compiled? I see two approaches to this. We could reject the concept altogether, and cleanly separate the build process itself, that happens exclusively on the source platform from, tests, that happen exclusively on the target platform. Or, we could have a portable bytecode compiler and interpreter, same as Jonathan Blow does with his language. I personally like going the bytecode route, because it make it easier to have a reference implementation you can compare to various backends.
As far as I understand, we only need a way to identify the target machine. The build machine is only relevant insofar as it must run the compiler and associated tools. Now I understand how that alone might be a problem: Microsoft is not exactly interested in running MSVC on Apple machines… Still, you get the idea.
Definitely.
There are two common cases of this. The first is really a bug in the build system: try to compile something, run it, examine its behaviour, and use that to configure the build. This breaks even the really common cross-compilation use case of trying to build something that will run on a slightly older version of the current system. Generally, these should be rewritten as either try-compile tests or run-time configurable behaviour.
The more difficult case is when you have some build tools that are built as part of the compilation. The one that I’m most familiar with is LLVM’s TableGen tool. To make LLVM support cross compilation, they needed to first build this for the host, then use it to generate the files that are compiled for the target, then build it again for the target (because downstream consumers also use it). LLVM is far from the only project that generates a tool like this, but it’s one of the few that properly manages cross compilation.
Oh, so that what you meant by distinguishing the build platform from the target platform. You meant distinguishing what will be build for the host platform (because we need it to further the build process) from the final artefacts. Makes sense.
Another example would be something like build.rs in rust projects, though that seems less likely to cause problems. The linux kernel build also compiles some small C utilities that it then uses during the build so they have HOSTCC as well as CC.
The concept of self hosted language is fading away. The last truly self hosted language might have been Pascal.
On its surface this sounds preposterous. Can you elaborate? I know of maybe a dozen self-hosted languages since Pascal so I think I must be misunderstanding.
Edit: I’m guessing you mean that not only is the compiler self-hosted, but every last dependency of the compiler and runtime (outside the kernel I guess?) is also written in the language? That is a much more limited set of languages (still more than zero) but it’s not the commonly accepted meaning of self-hosted.
The original Project Oberon kernel was written in assembly, but the newer version is written almost entirely in Oberon.
Some of the early Smalltalks were written almost entirely in Smalltalk, with a weird syntactic subset that had limited semantics but compatible syntax that could be compiled to machine code.
And of course LISP machines, where “garbage collection” means “memory management.”
Yes the latter. Sorry, maybe the terminology I used is off.
It’s an interesting distinction even if the terminology isn’t what I’d use. There’s a trend right now among languages to hop on an existing runtime because rebuilding an entire ecosystem from first principles is exhausting, especially if you want to target more than one OS/architecture combo. Sometimes it’s a simple as just “compile to C and benefit from the existing compilers and tools for that language”. But it seems fitting that we should have a way to describe those systems which take the harder route; I just don’t know what the word would be.
This is exactly what’s going on. You need to install the target platform’s specifications in an imperative format (C headers), and it’s the only format they provide.
And it makes extreme assumptions about file system layout, which are all necessarily incorrect because you’re not running on that platform.
Could you elaborate on what “the harder problem” is?
Go can cross-compile Go programs but cgo requires an external toolchain even natively; cross compiling cgo is a pain.
Zig compiles Zig and C from almost any platform to almost any platform pretty seamlessly.
As I understand it, Zig doesn’t do much more than clang does out of the box. With clang + lld, you can just provide a directory containing the headers and libraries for your target with
--sysroot=and specify the target with-target. Clang will then happily cross-compile anything that you throw at it. Zig just ships a few sysroots pre-populated with system headers and libraries. It’s still not clear to me that this is legal for the macOS ones, because the EULA for most of them explicitly prohibits cross compiling, though it may be fine if everything is built from the open source versions.This is not the difficult bit. It’s easy if your only dependency is the C standard library but most non-trivial programs have other dependencies. There are two difficult bits:
The first is pretty easy to handle if you are targeting an OS that distributes packages as something roughly equivalent to tarballs. On FreeBSD, for example, every package is just a txz with some metadata in it. You can just extract these directly into your sysroot. RPMs are just cpio archives. I’ve no idea what .deb files are, but probably something similar. Unfortunately, you are still responsible for manually resolving dependencies. It would be great if these tools supported installing into a sysroot directly.
The second is really hard. For example, LLVM builds a tablegen tool that generates C++ files from a DSL. LLVM’s build system supports cross compilation and so will first build a native tablegen and then use that during the build. If you’re embedding LLVM’s cmake, you have access to this. If you have just installed LLVM in a sysroot and want to cross-build targeting it then you also need to find the host tablegen from somewhere else. The same is true of things like the Qt preprocessor and a load of other bits of tooling. This is on top of build systems that detect features by trying to compile and run something at build time - this is annoying, but at least doesn’t tend to leak into downstream dependencies. NetBSD had some quite neat infrastructure for dealing with these by running those things in QEMU user mode while still using host-native cross-compile tools for everything else.
That’s what it does but to say that it “isn’t much more than what clang does out of the box” is a little disingenuous. It’s like saying a Linux distro just “packaged up software that’s already there.” Of course that’s ultimately what it is, but there’s a reason why people use Debian and Fedora and not just Linux From Scratch everywhere. That “isn’t much more” is the first time I’ve seen it done so well.
It solves the trivial bit of the problem: providing a sysroot that contains libc, the CSU bits, and the core headers. It doesn’t solve the difficult bit: extending the sysroot with the other headers and libraries that non-trivial programs depend on. The macOS version is a case in point. It sounds as if it is only distributing the headers from the open source Apple releases, but that means that you hit a wall as soon as you want to link against any of the proprietary libraries / frameworks that macOS ships with. At that point, the cross-compile story suddenly stops working and now you have to redo all of your build infrastructure to always do native compilation for macOS.
The usual problem encountered when cross-compiling from a non-macOS system to macOS is you need the macOS headers and it’s against the licence agreement to redistribute them or even use them on non-Apple hardware:
How does Zig handle this?
Edit: having said that, this repo has existed for a long time and hasn’t been taken down yet…
it’s not against the license agreement. the header files are under the APSL https://spdx.org/licenses/APSL-1.1.html
Even if it was, it’s probably not enforceable. Didn’t we have a ruling a while back stating that interfaces were not eligible for copyright?
That was Oracle v Google, right?
That’s the one. If I recall correctly, Google originally lost, then appealed, and the ruling was basically reversed to “interfaces are not subject to copyright”.
Now that was American law. I have no idea about the rest of the world. I do believe many legislations have explicit exceptions for interoperability, though.
The Supreme Court judgement said ‘assume interfaces are copyrightable, in this case Oracle still loses’ it did not make a ruling on whether interfaces are copyrightable.
Not exactly, the ruling didn’t want to touch the “interfaces are not subject to copyright” matter since that would open a big can of worms. What it did say, however, was that Google’s specific usage of those interfaces fell into the fair use category.
Ah, so in the case of Zig, it would also be fair use, but since fair use is judged on a case by case basis, there’s still some uncertainty. Not ideal, though it looks like it should work.
There’s no useful precedent. Google’s fair use was from an independent implementation of an interface for compatibility. Zig is copying header files directly and so must comply with the licenses for them. The exact licenses that apply depend on whether you got the headers from the open source code dump or by agreeing to the XCode EULA. A lot of the system headers for macOS / iOS are only available if you agree to the XCode EULA, which prohibits compilation on anything other than an Apple-branded system.
I recall that Google did copy interface files (or code) directly, same as Zig?
Java doesn’t have any analogue of .h files, they wrote new .java files that implemented the same methods. There is a difference between creating a new .h file that contains equivalent definitions and copying a .h file that someone else wrote. If interfaces are not copyrightable, then the specific serialisation in a text file may still be because it may contain comments and other things that are not part of the interface.
Interesting. Ok so does Zig just include the headers from the most SDK then?
The way zig collects macos headers is still experimental. We probably need to migrate to using an SDK at some point. For now it is this project.
I’d be super nervous about using this in production. This is using code under the Apple Public Source License, which explicitly prohibits using it to circumvent EULAs of Apple products. The XCode EULA under which the SDKs are prohibited explicitly prohibits cross-compiling from a non-Apple machine. I have no idea what a judge would decide, but I am 100% sure that Apple can afford to hire better lawyers than I can.
Zig has nothing to do with xcode. Zig does not depend on xcode or use xcode in any way. The macos headers have to do with interfacing with the Darwin kernel.
Apple generally doesn’t bother with small-scale infringement. They care about preventing cross compilation only insofar as it might hurt Mac sales.