seeing Shake’s influence here

How so?

I think the core issue is that Bazel (and I assume Buck) works on a monorepo with a single global version number.

That doesn’t totally gibe with the way open source works, where you usually have separate version numbers for each package, and then a dependency solver for version constraints.

In fact I remember Jane St. (who also employed one of the authors) said that they sponsored an open source OCaml package manager with a dependency solver, but they don’t actually use it. Instead they use a monorepo, because it simplifies life if you can get away with it.

It would basically be like if you had not just all Debian packages in a monorepo, but also all Python packages, all R packages, Java packages, etc. It’s possible, but it’s a lot of work.

I guess Nix does this by hard-coding versions in dependencies? I’m a little fuzzy on that, and am interested in how it works. I think it’s closer to a monorepo, and it’s open source with a very large number of packages.

Also, in some ways you have to “pick your poison” with regard to portability: do you want to portable across languages, or across OSes? Most language package managers work on Windows to some extent. But a lot of the global build systems work across languages, but are worse on Windows.

There is a polyglot vs. monoglot tension here too… Most language package managers optimize for the happy path of everything in that language.

e.g. a Python package manager fundamentally has no idea what versions of C/C++ libraries are installed on your system, and this is the ultimate source of a lot of build failures and lack of reproducibility. (The Conda package manager was written separately to address this problem)

Also I believe node.js and Rust have problems with C/C++ dependencies too. It’s an inherently messy problem. So some hypothetically nicely-buildable language would basically have to outlaw those – sort of like Java and Go heavily discourage foreign native code, but they still have them anyway because that’s life :-)

In plainer language, I’d say the observation/motivation is that not only do compiling and linking benefit from incrementality/caching/parallelism, but so does the build system itself. That is, the parsing of the build config, and the transformation of the high level target graph to the low level action graph.

IMO, a parallel build system (i.e., one that performs its own housekeeping in parallel, not only runs external tools in parallel like, say, make) is a no brainer in this day and age. build2 was a multi-threaded build system from the early days and parsing things (like buildfiles, -M output, etc) is only half of the story. You also often need to store intermediate build state (like partially preprocessed C/C++ translation units) which can be quite large and need to be compressed (e.g., with something cheap like lz4). Being able to compress in parallel helps a lot, naturally. But the biggest benefit in having a multi-threaded build system, IMO, is that you can now implement certain rules in-process. For example, in build2 we have a rule for processing the venerable .in files and it is implemented essentially as a function which the build system calls whenever it needs to update an .in file.

So maybe they can allow execution of the preprocessor to feed back into the shape of the target graph or action graph. Although I wonder effect that has on performance.

The trick to good performance here is to not throw away the result of preprocessing. In fact, it can even be faster compared to the traditional model of extracting header dependencies as byproduct of compilation proper. Details are here: https://build2.org/article/preprocess-compile-performance.xhtml

Build systems like Buck and Bazel are huge graphs of tiny objects, and that really stresses the GC more than most programs.

In the paper linked in the blog post you’ll see that the main things the program does are parsing, deserializing, and then lowering the target graph to the action graph. They are considering millions of files in a monorepo at once, which will easily lead to tens of millions of objects.

Although I would agree that you’re not dropping frames, and garbage collection is very useful for graph-based workloads. If the GC is efficient, and the JVM’s is, then it should be a win?

However another problem is that the JVM lacks value types on the stack, and so it creates more garbage. Go for example has value types.

I googled “Bazel Garbage Collection”, thinking it would be more of a problem, and ended up with this link instead! It does seem like the JVM isn’t ideal, but not because of its GC!

https://github.com/bazelbuild/bazel/issues/6514#issuecomment-675157826

There was a widespread desire for a more integrated implementation, so various interested people had a meeting some time in early 2006 to discuss what the next system should look like. There was no question that it had to be a typed language, which at that point meant C++ or Java. (Rob Pike was in the room. Sadly Go wasn’t invented for four more years, as it would have been the ideal tool for the job. Google uses very little Rust, Scala, and Haskell.) If memory serves, Java won primarily because it had a garbage collector—and half the job of a build tool is concurrent operations on often-cyclic graphs. And I’m sure that at least in part it was because Java was the language Johannes Henkel and I, who started the project, were using at the time.

The other half of a job of a build tool is interacting with the operating system: reading files and directories, writing files, communicating over a network, and controlling other processes. In hindsight, the JVM was a poor choice for this work, and many of Blaze’s problems stem from it. Most system calls are inaccessible. Strings use UTF-16, requiring twice as much space and expensive conversions at I/O boundaries. Its objects afford the user little control over memory layout, making it hard to design efficient core data structures, and no means of escape for performance-critical code. Also, compiling for the JVM is slow—surprisingly, slower than C++ or Go even though the compiler does less—yet the resulting code is also slow to start and slow to warm up, pushing CPU costs that should be borne by the developer onto the user. (Google runs Blaze’s JVM on an 18-bit number of cores.) The JVM is opaque, noisy, and unpredictable, making it hard to get accurate CPU profiles, which are crucial to finding and optimizing the slow parts. The only thing I really like about the JVM is that it can run in a mode in which Object pointers occupy 32 bits but can address 32GB, which is a significant space saving in a pointer-heavy program.

This is after doing a project that I’m guessing has 100K to 500K lines of Java code written from scratch, maybe more. FWIW Alan and Johannes were my team-mates at the time! I worked on Blaze a tiny bit, but it wasn’t my main project.

FWIW I wrote a comment on “Rust is the future of JS infrastructure” here, explaining what I learned about similar workloads:

https://news.ycombinator.com/item?id=35045520

However this is more about static typing than garbage collection. Java is better than say node.js or Python for sure.

GC pauses get significant when you need either consistently low latency, or prolonged periods of high performance with a bunch of memory operations. For web servers, there’s a bunch of empty time between requests, where the GC can run “for free”, and the request usually doesn’t take a lot of memory allocations or time. But for build systems, when you run it, it’s gonna be a couple seconds of high intensity planning work, with a lot of memory allocations (creating the dependency/task graphs, etc.), which will trigger GC several times during the work (no empty time to fit the pause in unlike in web servers) and that will directly impact the performance the user feels for interactive compilation.

TBH, that whole paragraph looks rather incoherent. I think the goal there was “say something, fit it into two lines, don’t start Java vs Rust vs Go vs C++ flamewar”, rather than “summarize the actual reasoning”. GC pauses are certainly irrelevant for a build system.

Here’s what I’d say instead (I do believe that today Rust is by far the best choice for build systems):

• Rust would use significantly less memory, as its object graph has far less indirection. Given that the build system runs compilers who tend to devour RAM, setting less ran aside for build system is important.
• Rust would run somewhat faster. Less memory usage and less pointer chairing improves cache efficiency. Runtime speed is a bottleneck for no-op build, and there are some rather cpu-heavy operations (dependency resolution & hashing)
• Rust starts up significantly faster. No-op build doesn’t have time to warm the JIT up.
• Build systems are nastily concurrent, and Rust is great at that, as it tracks thread-safety in the type system, and is somewhat-adequate for evented programming.
• Build system can build anything except itself, so it needs to solve “bootstrap problem” — getting build system itself to the user’s machine. That’s much easier with a statically linked binary.
• Build systems are IO-heavy and sometimes do systemsy-schenanigans, so having direct access to OS facilities is a plus.