Docker Bake is a very flexible & powerful build orchestration tool. It’s been marked as experimental for quite a few years, so it’s nice to see it become GA.
It can run ad-hoc build tasks (like unit tests, linting) as well as creating build outputs as regular files and container images.
It’s a more general version of language-specific build matrix tools, like Python’s Tox, and it can run build jobs in the same way in CI and a dev machine, to avoid the fun of iteratively debugging CI Yaml configs.
A more advanced but really nice feature is that you can speed up CI jobs a lot if you can provide a long-running buildkit daemon service to your CI jobs. You configure docker bake to run builds on the buildkit daemon, which allows the buildkit service to re-use its build cache across jobs, so you get the same speed in CI as you do on a dev machine with a hot cache (without the delay of a CI job pulling a cache from storage each time).
Because build tasks are defined using Dockerfiles, you can depend on existing images, or create custom images to run build steps in, which avoids problems with missing or inconsistent dependencies in local dev environments.
Here’s an example where I’ve used it in a project of mine (a Python package that needs to be tested to work in all the current Python versions):
This PR runs several tasks defined in a bake file (click “View Details” on the “23 checks passed” section to see them)
The docker-bake.hcl config file defines the build tasks, some of which use a matrix generate multiple steps from a list of options (Python version numbers here)
You do need to be careful to not invalidate the build cache to get good performance on things like lint/unit test jobs. E.g. the build steps here install and cache the Python dependencies first, and only-refetch deps if the two files that define dependencies change. Other steps use --mount=type=cache to re-use cache files created by build steps
This is a wonderful reply! I’m always on the look-out to speed up development processes and CI seems like the perfect target in most cases. I’ll definitely pair the OP with this to get a better understanding of everything.
Thanks, glad you liked it. If you want to focus on speed I’d definitely recommend looking into running a shared buildkitd to reuse between CI jobs. As I mentioned, you can reuse the build cache efficiently, but also scale to multiple builder instances and run builders for specific architectures to avoid emulating. You can also give dev environents access to a remote builder offload build tasks from a dev machine. I believe this is what Docker’s own build cloud does, and there are a few companies offering this as a service, but it’s not too hard to run your own buildkitd. The buildx docs aren’t great for using network builders, but this is a starting point: https://docs.docker.com/build/builders/drivers/remote/
(It could be worth segregating builders used for release artifacts from others though, to reduce the risk of a buildkitd sandbox escape from one build compromising others.)
Docker Bake is a very flexible & powerful build orchestration tool. It’s been marked as experimental for quite a few years, so it’s nice to see it become GA.
It can run ad-hoc build tasks (like unit tests, linting) as well as creating build outputs as regular files and container images.
It’s a more general version of language-specific build matrix tools, like Python’s Tox, and it can run build jobs in the same way in CI and a dev machine, to avoid the fun of iteratively debugging CI Yaml configs.
A more advanced but really nice feature is that you can speed up CI jobs a lot if you can provide a long-running buildkit daemon service to your CI jobs. You configure docker bake to run builds on the buildkit daemon, which allows the buildkit service to re-use its build cache across jobs, so you get the same speed in CI as you do on a dev machine with a hot cache (without the delay of a CI job pulling a cache from storage each time).
Because build tasks are defined using Dockerfiles, you can depend on existing images, or create custom images to run build steps in, which avoids problems with missing or inconsistent dependencies in local dev environments.
Here’s an example where I’ve used it in a project of mine (a Python package that needs to be tested to work in all the current Python versions):
--mount=type=cacheto re-use cache files created by build stepsThis is a wonderful reply! I’m always on the look-out to speed up development processes and CI seems like the perfect target in most cases. I’ll definitely pair the OP with this to get a better understanding of everything.
Thanks, glad you liked it. If you want to focus on speed I’d definitely recommend looking into running a shared buildkitd to reuse between CI jobs. As I mentioned, you can reuse the build cache efficiently, but also scale to multiple builder instances and run builders for specific architectures to avoid emulating. You can also give dev environents access to a remote builder offload build tasks from a dev machine. I believe this is what Docker’s own build cloud does, and there are a few companies offering this as a service, but it’s not too hard to run your own buildkitd. The buildx docs aren’t great for using network builders, but this is a starting point: https://docs.docker.com/build/builders/drivers/remote/
(It could be worth segregating builders used for release artifacts from others though, to reduce the risk of a buildkitd sandbox escape from one build compromising others.)