Our repository is so large/active that our CI is painful. You could also say our build/test process is so slow that our CI is painful. I’m looking for tooling to reduce the load on our CI.
For example, if only documentation has been changed, there is no need to run the tests. If no documentation is changed there is no need to rebuild the Doxygen website. If only a one component is changed, the unit tests for the others need to be run. If only tests have been changed there is no need to deploy the software. If only build config is changed, there is no need to run the unit tests.
I’m not sure what to google for. Is there a name for such mechanisms? Buildpipe seems roughly the right idea but is not really applicable without Buildkite.
The configuration for such a tool is so complex that a Python library might make more sense than an executable? We are certainly not the first ones. Does everybody write this from scratch? It makes sense for monorepos but the awesome list does not contain anything.
Can you share more details about the pain points you see please? Is it the wall-clock time that’s painful, or the CPU time? For us it was the former and we had some success with doing more things in parallel on CircleCI: from over 13 mins to under 4 minutes.
If you’re willing to make changes to how you build your project you might consider something like the Buck build tool. It was developed at Facebook but it’s used by other organisations as well, and here’s a case study of sorts from Uber which mentions it. I’ve personally never used it, but it looks interesting.
The wall-clock time is the primary focus. We already tried to sacrifice CPU time for it: Bigger build slaves and more jobs in parallel. The low hanging fruit is harvested there.
Since we use a heavily customized CMake build system switching to something like Buck is a big investment. Nevertheless, we have some people investigating that route already.
bazel, like Buck/pants/please handles the build/testing part of your requirements (and possibly the documentation part), but doesn’t address the deploying.
If you run across anything that handles deploy dependencies, I’d love to hear it.
If your project is not too big you might want to split the build between pull request and master builds. This way engineers get a fast feedback but extensive tests still run on master. We use this approach for Marathon. It does require human intervention but you could configure bors-ng to ensure master is always green.
At my job we switched from Jenkins to TeamCity. According to a colleague TeamCity is better. But it’s not that great either. The UI is so heavy (with JavaScript?) that my old notebook stopped to render it properly after half an hour or so. We also had problems to connect it to the company’s DC/LDAP.
I’m not familiar with other off-the-shelf solutions, but if I’d need one I’d probably just use a cronjob (or git-hook) that
makes it