I highly recommend anyone interested in this article also read the paper “Build Systems a la Carte”. You can skip/ignore the Haskell bits, and focus on the description of build systems as a combination of a scheduler (which decides what tasks to run and when) and a rebuilder (which decides when to build/rebuild artifacts).
The paper specifically discusses Bazel as one example, where it uses a “restarting” scheduler (if a task is stuck because a dependency isn’t ready, kill the task and restart it from scratch later) and a “constructive trace” rebuilder (storing the results of prior steps, not just using a hash or setting a dirty bit).
I’m actually a pretty big fan of Bazel; the approach really resonates with me: distributed build, do it once, make it reproducible.
But getting into bazel is extremely terse, maybe it’s because I never worked at google.
The terse-ness means I can’t get adoption in any companies I work for, since it’s so much more complex and non-approachable than a simple build from scratch.
Thats pretty common I would say. Bazel is a specialized tool.
Imo its best used in a big organization where you have thousands of technical users and multiple languages, services and you need to staff a DevEx team to help enforcing standards across org. Bazel then act as one of the key piece glueing things together so that the centralized team could build a DevEx platform on top.
In smaller orgs where a lot of those requirements do not matter, you would be able to move much faster by leveraging the open source ecosystem toolings instead of crafting your own.
I can agree that not being able to leverage existing build definitions (without wrapping) is a big disadvantage, but you can go a long way with a pretty limited subset of Bazel and it’s the most mainstream of the choices which offer similar advantages, so I prefer it, even for extremely small projects. I caught several cases where the original project (CMake) specified dependencies incorrectly and outright missed header files (which would result in not recompiling when those files changed) thanks to its sandboxing.
I think to help people work in the areas where Bazel is simplest to work with, it just takes a little OSS investment in providing remappings, which will pay for itself at soon as you need to track down sandboxing, accidental system dependencies (essential for multiplatform), build farms, or multi-language workflows.
I have been talking and working with some folks on solutions to help reduce the pain points of adopting Bazel. I really hope the entry barrier is lowered overtime, which is also what motivates me to write these blog posts.
But after working as a Bazel consultant for a few customers, I can definitely tell that there are a lot of pain still and more works to be done so that we can confidently say this is strictly a better choice than other alternatives.
Interesting. On a particular setup (Linux, with mostly C++ and Python), it seems Bazel setup is pretty simple, it can leverage system libraries for C++ code, and can use PIP for python libraries.
Is it a particular language painful? A particular OS (Windows? Or macOS?)? Or just a lot of 3rd party libraries without Bazel BUILD? Or Bazel issues harder to debug even if you can have setup started pretty easy?
Is it a particular language painful?
A particular OS (Windows? Or macOS?)?
Or just a lot of 3rd party libraries without Bazel BUILD?
Or Bazel issues harder to debug even if you can have setup started pretty easy?
Yes to all the above.
First of all, not all languages work with Bazel out of the box. There need to be rules set to tell the build tools on how to build and cache your language artifacts. This exists for the established languages such as C/C++, Java, Go… and NodeJS, Python, Docker with lower maturity… But if you are trying to use an exotic lanugage like Julia, or Matlab then you are out of luck.
OS support is also a big issue. Google does not use Windows so the experience over there has been quite subpar. Many orgs have been moving to Cloud Workspace solutions as a work around for Windows users.
3rd party libraries come with their own challanges as well. I.e. some third party lib would assume a certain thing about the environment they are built in (some system dependency of the host) and you would need to work around that. Nix Package Manager has recently been coupled with Bazel to solve this particular class of problem.
Finally, Debugging and IDE support are very very poor atm. Again, this is because Google uses an internal non-opensource setup and thus, pay little effort in supporting the opensource use case. As of yesterday, Jetbrain has stepped up in supporting the Bazel IntelliJ plugin as a co-maintainer with Google… but there is still no official support for VSCode or Neovim or Emac etc…
So there are plenty of pain points, plenty of footguns… But despite all that, I would say it’s still very much worth it for the clients I have worked with recently to adopt Bazel, and things are getting better everyday.
I used bazel for a mobile app with a C++ library, and one pain point was that IDE (Xcode, Android Studio) were not fully aware of the toolchain details and debugging was often not working properly and had to be tweaked (source maps for instance).
It was not much faster than the “native” way of doing things (on mobile at least), and when something went wrong you had to debug the build system.
This aligns with my experience. We had an application that supported both iOS and Android, and used several C++ libraries. Enough of that used bazel that we decided to take a flyer, and I came to the same conclusion that you did. Though I’d add that we never found a cross-platform mobile build system that we didn’t need to debug in a poorly supported way when something went wrong. And we never found a nice way to make the native stuff use foreign build systems that didn’t have the same problem.
So while that observation did mean that I was never inspired to use bazel heavily for any new work, I never shied away from adopting it if it was in place for a substantial chunk of my app’s dependencies for that reason.
Thanks for this post, it was quite informative. Earlier this week I was thinking how we don’t see many articles about build and CI systems, and to my delight I was proved wrong :)
I highly recommend anyone interested in this article also read the paper “Build Systems a la Carte”. You can skip/ignore the Haskell bits, and focus on the description of build systems as a combination of a scheduler (which decides what tasks to run and when) and a rebuilder (which decides when to build/rebuild artifacts).
The paper specifically discusses Bazel as one example, where it uses a “restarting” scheduler (if a task is stuck because a dependency isn’t ready, kill the task and restart it from scratch later) and a “constructive trace” rebuilder (storing the results of prior steps, not just using a hash or setting a dirty bit).
Extended version of the paper: https://www.cambridge.org/core/journals/journal-of-functional-programming/article/build-systems-a-la-carte-theory-and-practice/097CE52C750E69BD16B78C318754C7A4
Ah, I didn’t realize there was an extended version! Thanks for sharing :D
I’m actually a pretty big fan of Bazel; the approach really resonates with me: distributed build, do it once, make it reproducible.
But getting into bazel is extremely terse, maybe it’s because I never worked at google.
The terse-ness means I can’t get adoption in any companies I work for, since it’s so much more complex and non-approachable than a simple build from scratch.
Bazel has good ideas, but it’s googleware. Large, complex, poorly documented, and full of edge cases and quirks. It just feels bad to use.
Thats pretty common I would say. Bazel is a specialized tool.
Imo its best used in a big organization where you have thousands of technical users and multiple languages, services and you need to staff a DevEx team to help enforcing standards across org. Bazel then act as one of the key piece glueing things together so that the centralized team could build a DevEx platform on top.
In smaller orgs where a lot of those requirements do not matter, you would be able to move much faster by leveraging the open source ecosystem toolings instead of crafting your own.
I can agree that not being able to leverage existing build definitions (without wrapping) is a big disadvantage, but you can go a long way with a pretty limited subset of Bazel and it’s the most mainstream of the choices which offer similar advantages, so I prefer it, even for extremely small projects. I caught several cases where the original project (CMake) specified dependencies incorrectly and outright missed header files (which would result in not recompiling when those files changed) thanks to its sandboxing.
I think to help people work in the areas where Bazel is simplest to work with, it just takes a little OSS investment in providing remappings, which will pay for itself at soon as you need to track down sandboxing, accidental system dependencies (essential for multiplatform), build farms, or multi-language workflows.
Me too man.
I have been talking and working with some folks on solutions to help reduce the pain points of adopting Bazel. I really hope the entry barrier is lowered overtime, which is also what motivates me to write these blog posts.
But after working as a Bazel consultant for a few customers, I can definitely tell that there are a lot of pain still and more works to be done so that we can confidently say this is strictly a better choice than other alternatives.
Interesting. On a particular setup (Linux, with mostly C++ and Python), it seems Bazel setup is pretty simple, it can leverage system libraries for C++ code, and can use PIP for python libraries.
Is it a particular language painful? A particular OS (Windows? Or macOS?)? Or just a lot of 3rd party libraries without Bazel BUILD? Or Bazel issues harder to debug even if you can have setup started pretty easy?
Yes to all the above.
First of all, not all languages work with Bazel out of the box. There need to be
rulesset to tell the build tools on how to build and cache your language artifacts. This exists for the established languages such as C/C++, Java, Go… and NodeJS, Python, Docker with lower maturity… But if you are trying to use an exotic lanugage like Julia, or Matlab then you are out of luck.OS support is also a big issue. Google does not use Windows so the experience over there has been quite subpar. Many orgs have been moving to Cloud Workspace solutions as a work around for Windows users.
3rd party libraries come with their own challanges as well. I.e. some third party lib would assume a certain thing about the environment they are built in (some system dependency of the host) and you would need to work around that. Nix Package Manager has recently been coupled with Bazel to solve this particular class of problem.
Finally, Debugging and IDE support are very very poor atm. Again, this is because Google uses an internal non-opensource setup and thus, pay little effort in supporting the opensource use case. As of yesterday, Jetbrain has stepped up in supporting the Bazel IntelliJ plugin as a co-maintainer with Google… but there is still no official support for VSCode or Neovim or Emac etc…
So there are plenty of pain points, plenty of footguns… But despite all that, I would say it’s still very much worth it for the clients I have worked with recently to adopt Bazel, and things are getting better everyday.
I used bazel for a mobile app with a C++ library, and one pain point was that IDE (Xcode, Android Studio) were not fully aware of the toolchain details and debugging was often not working properly and had to be tweaked (source maps for instance).
It was not much faster than the “native” way of doing things (on mobile at least), and when something went wrong you had to debug the build system.
This aligns with my experience. We had an application that supported both iOS and Android, and used several C++ libraries. Enough of that used bazel that we decided to take a flyer, and I came to the same conclusion that you did. Though I’d add that we never found a cross-platform mobile build system that we didn’t need to debug in a poorly supported way when something went wrong. And we never found a nice way to make the native stuff use foreign build systems that didn’t have the same problem.
So while that observation did mean that I was never inspired to use bazel heavily for any new work, I never shied away from adopting it if it was in place for a substantial chunk of my app’s dependencies for that reason.
Thanks for this post, it was quite informative. Earlier this week I was thinking how we don’t see many articles about build and CI systems, and to my delight I was proved wrong :)