So, this is no long and insightful write-up, but I still think it’s worth writing a few words about and submitting here.
I recently needed to compile Chromium for work. Chromium has a total of 76924 C and C++ files. A single complete compile, with 32-bit ARM as the target architecture, on a build server with an 8 core/16 thread Xeon E5-2660 and a ton of memory and no other CPU intensive task running, takes almost 6 and a half hours. That means it’s compiling at the speed of ~3.2 files per second.
Compiling the entire Linux kernel for ARM though, from http://github.com/raspberrypi/linux, takes around 5 and a half minutes.
Compiling Firefox on 8 core ryzen targetting the host arch takes between 10 and 15 minutes.
Wow that is fast, takes ~2h on a build server that takes 7 hours for chromium. All the recent rust stuff really slowed it down.
Oof, yeah, I bet. Rust is notoriously slow to compile. Luckily, incremental compilation is in the nightly compiler right now. I’ve been using it wherever I can and it really does make a difference. But I suppose it wouldn’t do much for an initial compilation of a project. :(
In this case a large chunk of this is just bindgen; we need to generate bindings so we throw a libclang-based-bindings generator at all of the header files. Twice (complicated reasons for this).
It’s also pretty single threaded (codegen units will help with this, but currently isn’t, and I have to investigate why).
Incremental compilation and working codegen units and cleaning up the bindgen situation will help a lot. Going to take a while, though.
Also not going to help packagers who use only temporary compilation environments which are discarded after a package is built.
Package managers (and regular builds also) should not be starting from scratch every time. Even if we insist on doing all source editing in ASCII, we need to be delivering modules as fully-typed, parsed ASTs.
This insistence on going back to plain source code every chance we get and starting over is easily wasting more energy than the bitcoin bubble.
They should if you want reproducible builds.
These are completely orthogonal. There’s nothing stopping reproduceable builds where you run the entire pipeline if you insist on comparing the hash of the source and the hash of the output. And you would still get the same benefit by comparing source<->ast and ast<->binary
https://bazel.build/faq.html
Yeah, ideally a compiler would take a difference in source to a difference in output. Compiler differentiation?
I believe you can do a decent amount of caching in this space? Like MSFT and co. have compile serverse that will store incrementally compiled stuff for a lot of projects so you’re only compiling changes
My non-beefy lenovo x series laptop takes ~45 minutes for a complete recompile, ~12min for average changes w/ ccache etc. and ~min for JS-only changes (which you can do as artifact builds, so they’re always 3min unless you need to build C++/Rust components
I’ve worked on a team with twenty minute builds (also a C++ application), and found that all sorts of practices are rejected because “build times”. Test-first? I’m not waiting twenty minutes to see my failed test, so I’ll get it working then I’ll add tests. Actually, it looks like this is hard to test, I’ll skip it just this once. Continuous integration? I’m not twiddling my thumbs waiting for Jenkins, I’ll merge then fix any problems later. Pay down some technical debt? Why would I spend half a day moving the build from 20 mins to 19.5 mins? No thanks.
On the other hand, who’s doing clean builds? If you change one file then your incremental build is, based on your estimate, around a third of a second. That’s acceptable. And Google presumably have some computers to throw at the problem to use distributed compilation and artefact caches. It’s possible that Google does not have compilation time problems, but does have data centre management and CI scale problems.
There’s also the issue that some changes require recompilation of almost everything. Want to disable debug mode such that debug assertions don’t crash Chromium? Well, time to find something else to do for the day and let the build server compile chromium overnight. Let’s hope no issues come up during the compilation, because then maybe only half the project was compiled, and most of tomorrow goes to waiting for it to compile too.
You’re probably right that Google has a ton of compute power to throw at the problem though.
Our build system keeps two build trees, one for debug and one for release.
It does, and is how Chromium works around the problem – super beefy development machines.
Test First in C++ can be beneficial. It can bias you toward making new abstractions with fewer dependencies rather than hacking things onto existing structures - because the latter gives you those 20 minute compile times. Better to have something that can just compile as cpp/h pair that stands alone with only a few includes.
we have even longer build cycles, but we’ve learned the hard way enough times that a bug caught in CI is worth 10 caught in production, easily.
Being patient and making sure that test tooling is always improving is also super important. Tests shouldn’t be written in a throwaway manner, because that makes the experience even that more miserable.
Historically, C++ has had this problem due to its compilation model. Every class has to include the header of classes it uses (transitively) unless you use some gimmicks or bias toward C++‘s equivalent of interfaces. When dependencies aren’t managed well, this can lead to N^2 build-time.
Java fixed this by making the compiler aware of the .class files (making it a linker in a way), so no there’s need to be aware of transitive dependences. A big impetus for Go was to make build time rational with its interface scheme.
People have tried everything in C++ including taking all files into memory at once to avoid disk I/O. But the language still pushes toward compile time work with compute intensive template meta-programming, etc. It’s a long story. I’ve spent years breaking dependencies in C++ to aid testing. In my experience C is far more tractable this way.
Isn’t that part of why modules are being introduced?
https://clang.llvm.org/docs/Modules.html
Aka it allows the compiler the ability to treat things a lot more granular and not have to recompile the world due to textual changes to unrelated header files.
Yes. I wish them luck. It’s sad that build-times spin out of control in C++ unless you apply diligence. This isn’t true for many other languages.
Did you follow any of the instructions in https://chromium.googlesource.com/chromium/src/+/lkcr/docs/linux_build_instructions.md#faster-builds?
I think that topic is slightly misleading. It sounds like Google itself has a problem, but in fact you’re referring only to their browser. (Won’t deny that these compilation times are a bit too much in case of Chromium.)
You’re right. A better tilte might’ve been “Chromium has a compilation time problem” or something.
Brings me back to my LFS & Gentoo years. According to the BLFS handbook, Chromium takes whole 94 SBU (with 4 threads) to build. LibreOffice takes 41 (with 8 threads), which is about the same. At the time, OpenOffice was the king of the long build times. The only package Gentoo users were actually downloading precompiled instead of building it themselves.
Oh my, I still remember how happy I was when my OpenOffice build finished successfully after 13 hours. You never knew what could go wrong. It was pretty cool memtest, too. :-)
Maybe 10-12 years ago I ran Slackware on all my home machines, and liked to build all of the software I used on a daily basis from source. I remember Firefox being painful (lots of dependencies), but after a day or two of installing dependencies (also from source), I was up and running.
OpenOffice was the only one I ever gave up on, and a decade later I still dislike it because of that. I remember it being a nightmare of extensive dependencies (often requiring specific versions) and requiring multiple languages (at least C++ and Java, I think also Perl and others?). And it required a TON of disk space and memory to build. After struggling for a while, I decided it just wasn’t worth it.
Well, it depends. If they cache object files that do not need recompilation and thus only compile anything changed, then times should drop down considerably.
That doesn’t help with first builds, and doesn’t help if you’re changing headers which are included by most files (such as changing whether debug mode is on or off, or changing log levels, or simply changing a core abstraction).
First builds are a lost cause anyway. But after them, with proper binary caching I guess you can have faster loops. As for changing core abstractions, how often would that happen? How often will that header file change?
Also, do not forget that this is an organization with too much computer power and unlimited disk space. So keep those object files there for some time, have them tagged properly with regards to the changes that you mention and eventually the build will be fast.
Is such a system a lot of work? Definitely? But (a) they have lots of people to work on such a problem and (b) they do devote actual time on it, instead of being like us, thinking about it for 5 minutes on a work break without that much information about their internals.
First builds are definitely not a lost cause. As I said, crass compiling Linux, a rather big project, takes just over 5 minutes.
Header files need to change annoyingly often; remember, in C++, even adding a private member requires changing the header file.
Linux is an equivalently-complex project, but in a significantly simpler language (C). This might be more of a damning indictment of C++ than it is of Chrome.
Languages such as Modula-3 and Go were designed to compile fast on the first time. Far as metaprogramming, the industrial LISP’s compile faster than C++. The D language compiles way faster. The C++ language was just poorly designed. Sacrificing reasonable compilation speed, either first or later build, better be done to obtain a different, worthwhile benefit. That’s not the case with C++.
Edit: An organization with piles of computer power and disk space could use the time wasted on compile overhead to do stuff such as static analysis or property-based testing with that time.
Titus Winters was asked about build times at this years CppCon:
https://youtu.be/tISy7EJQPzI?t=1h23m10s
He never considers the opportunity cost of using that massive build farm for C++. Think of all the bitcoin that could be mined.
Opportunity cost? More like opportunity; without C++’ compile times, no such build farm would’ve been built. Because the build farm is built, and presumably doesn’t have to compile code 24/7, there’s now an opportunity to automatically mine bitcoin while the CPUs aren’t busy compiling!
Bitcoin mining with CPUs hasn’t been profitable for years.
If compilation speeds are a general Google problem it would explain the heavy focus on compilation speed in Go.
Yep - this has been stated explicitly by the team 😀
@mort the link goes to this entry not the article…
This is just a text post, not a link to a blog post :p
Google pushes the envelope with compilation and linker features on their browser. I’d wonder whether this is a bug or a feature. Was LTO or ThinLTO enabled for the build? That alone could be responsible for a large portion of the time.
Also any significant compilation work should consider the I/O to the files as a potential bottleneck. Was the filesystem cache cold or hot? Was
ccacheenabled?Notably this is a project which is entirely C + assembly and no C++.
Michael Zolotukhin gave a presentation called “LLVM Compile-Time: Challenges. Improvements. Outlook. “ at the 2017 US LLVM Developers’ Meeting. Much of this talk is regarding performance over commits but IIRC there was a decomposition of a single compile and where the time is spent. I recall the C++ frontend being a significant contributor to compile time.
I built a brand new machine with NVMe, 32G of ram, and a ryzen CPU. Then I decided to contribute to chrome and I was glad I had a brand new machine. IIRC it took 110 minutes to compile chromium.
That is astonishingly slow for a modern machine.
Yeah, basically the only way it can happen is if you have have an
O(n^2)bug, which C++ encourages through its use of header files :-(/I believe basically all C++ projects are structured to have
O(n^2)bugs in compilation time. Putting more and more code in headers basically guarantees that.I’ve seen C projects that compile very fast, but not any significant C++ projects. You need to use something like PIMPL [1], which I’ve heard of often, but never really seen used. It comes with a slight runtime performance cost, which I suppose people are not willing to accept. That, and it involves a lot of boilerplate.
[1] http://oliora.github.io/2015/12/29/pimpl-and-rule-of-zero.html
It also depends on what is in your headers. Unfortunately defines for debugging flags, and template code may be required, changes to those can require recompiling most/all files. If class B has a function that takes a reference to A as a parameter, add “class A;” to to b.h and put the implementation in the cpp. Then the 20 files that include b.h don’t have to include a.h and related headers. I haven’t seen much of the Chromium code, but possibly they are adding too much code to the headers, I know they use some templates, so possibly there is an over reliance there, possibly they are just inclining too many non-template functions and rely on includes too much.
Why don’t you have a fast build server? 8 cores seems small for a “server”.
It’s a startup without an infinite supply of money.
Last I looked, chromium ships with 720 megabytes of third party source, including ffmpeg, libc++, and a bunch of other things. Of course it takes forever to compile.