1. 11

I’m looking into setting up some CI both for work and hobby projects. My default has always been Jenkins, but I see all the new options available. What are you using? Is everyone hopping onto Github Actions? Is Jenkins still the best option if you don’t mind doing some admin setup? Or are the cloud CIs good enough that you don’t need to be able to configure them so much?


  2. 20

    I recommend sr.ht because it has first class support for BSDs!

    1. 3

      Happy sr.ht customer checking in.

      I’ve used builds.sr.ht for a couple of different things. It’s very easy to interface with using jq + curl. I was able to build a CI bot for Agar with a simple shell script that polls the GitHub every few minutes and submits a builds.sr.ht manifest for each new commit, posting to the IRC channel when the build completes with a status and links to the logs.

      This is great, since it becomes totally trivial to test on different platforms (currently it builds on debian, ubuntu, and openbsd, but it would be straightforward to add more if I felt like it).

      The only potential downside (for some types of projects) is that it doesn’t directly support Docker, although you could always configure it to install Docker in the VM once it’s running. I haven’t found this to be a blocker for any project I’ve used builds.sr.ht for, but it might for some people.

    2. 6

      I’ve thoroughly tested a lot of CI/CD services about a year back. I finally settled on GitLab CI as best choice. It requires you to host your repository on GitLab though.

      1. 3

        Interesting! I know a lot of people like GitLab because it’s not part of the Github/Microsoft monoculture. Was that a driving reason, or were there others?

        1. 1

          I’ve been using GitLab for almost 4 years now, largely for reasons stated in https://www.jvt.me/posts/2017/03/25/why-you-should-use-gitlab/ but also for all the extra stuff they’ve built on top over the years. Originally it was just because of private repos, but evolved into a love of the platform being built in a way that folks can contribute, unlike GitHub.


        2. 2

          Why was that your choice? What usecase?

          I don’t love gitlab CI for the reason it isn’t accommodating of usecase it’s not designed for, and I’m stuck with both. This is obviously very different from the cases where you are in the designed usecases.

          And actually one peeve is that it encourages the use of “fat” docker containers.

          1. 4

            My experience was the opposite. We went “all-in” on gitlab and love the ci process. Initially we used 2 bare runners, windows and linux, and installed whatever software we wanted on those. CI just called what it needed in those runners. Worked great.

            All the online examples used docker and we found that adding a docker runner allowed us to do more. Now we build our own docker images to use so they aren’t “fat”, they are minimal to what that step needs.

            Ultimately, we found that it was completely customizable to what we wanted.

        3. 5

          For work, BuildKite is my personal favourite.

          • Pricing is per user rather than per agent
          • You run the agents on your own hardware (fast!)
          • BK don’t have access to your source code
          • All pipeline configuration is in the repo, with the code
          1. 2

            Does everything run on your own servers? I don’t fully understand their licensing, yet. They talk about an open source agent that runs on your servers, but not about the component that triggers the agent. And are the non-free parts also in a FOSS client or is that distributed as a compiled binary only (e.g. Single-Sign-On)?

            I’m curious, because I like it that there is still paid software with local installations (for small companies).

            1. 1

              The build agent runs on your own servers, and is open source. Buildkite runs the “director/manager” service, web ui, etc themselves.

              We use it at work too, and really like it.

              1. 1

                The closed source part handles distribution of jobs to agents, artefact/log storage, presentation. They host that for you.

                The agent is fully open source and easy to build yourself (pure golang).

                This way, you can be confident that your sources remain on your own servers. The agent can only receive “run file x from git commit y” instructions, so they can’t inject malicious code that way.

            2. 4

              Heroku CI when it makes sense (it doesn’t always!). Otherwise, I’m favoring CircleCI right now over the other one we’re also using, Travis.

              1. 4

                Gitlab CI (for hobby work):

                • builds took 5+ minutes for a relatively small, free project using custom built docker images
                • limited number of “build server minutes” for free (sorry, I’m cheap)
                • yaml config

                Jenkins (pro/hobby):

                • companies set up A LOT of custom stuff
                • testing custom Groovy work is an absolute pain, especially if your CI is slow
                • super easy to set up locally
                • Jenkinsfiles can be used to store CI config

                GoCD (hobby)

                • fast
                • stored config from web interface as one huge XML (or best as I could tell)
                • UI kinda wonky, often required loading a new screen for simple things which shouldn’t require loading a new screen.

                I converted all my home stuff from Gitlab CI to GoCD (on home server) to Jenkins (on home server). I think having your own hardware locally is a big boon for hobby work, especially if you host your own git server, code search or other tools. I didn’t think GoCD was bad, Jenkins just seems easier to configure. It’s not fancy or pretty, but it gets the job done.

                1. 4

                  The Pony project is primarily using:

                  We still have a few older TravisCI, Appveyor, and CircleCI tasks around, but those are all being deprecated.

                  GitHub actions are very nice for us as it is much easier to automate portions of our workflow than it was with “external” CI services.

                  CirrusCI is awesome because… we can test Linux, Windows, macOS, and FreeBSD. But most importantly, we have CI jobs that build LLVM from scratch and CirrusCI allows use to get 8 CPU machines for the job. The only CI services we tried that have been able to handle the jobs that build LLVM are CircleCI and CirrusCI.

                  1. 1

                    I switched a project from a self-hosted Buildbot to CirrusCI and I’ve also found it to be great. It’s so useful that they can spin up builders on demand and do so on your own GCP account.

                  2. 3

                    For rigorous/correct CI/CD at $work, Jenkins (which does need management - we have a team of dedicated Jenkins maintainers).

                    For quick and dirty CI/CD (also at $work), I’ve been using Nomad batch jobs on our Nomad cluster. I have language-specific builder containers that build and upload artifacts (package Python wheels and upload them to our private pip repo with twine, build Go binaries and upload them to a ceph bucket, etc.).

                    I started this movement to scratch my own itch (and not rely on the Jenkins team for when I needed some very specific new build containers, e.g. an Alpine Go container with static librdkafka to statically compile Go binaries with the confluent-kafka-go library), but it proliferated.

                    For most usecases it’s sufficient (but with 0 sophistication around branches, snapshots, master builds, etc.). It even fits OK with Jenkins 2.0 - in my Jenkinsfile we can submit the Nomad job and periodically poll for its logs and exit status to display blue/red success/fail. I have a Python script to do that (nomad_submit_and_poll.py) which I can then invoke with an sh block in a Jenkins 2.0 file.

                    1. 3

                      Having used Travis, GitHub Actions, Jenkins, GitLab CI, Circle CI, Drone, GoCD, and a few others here and there, I’ll take GitLab CI over the others just about any day. GitHub Actions is growing on me but its lack of cache yet makes builds with a lot of cacheable dependencies take forever.

                      I use Drone at work right now and its shortcomings are exacerbated by our implementation being behind in releases, customized a lot, and having insufficient number of workers for the amount of work we put on it. That last one is getting better but it’s frustrating having to mentally adjust settings from examples whenever I spend some time investigating new plugins.

                      1. 3

                        At $dayjob we use a hosted service called BitRise. It’s good: on an average day it just works and I do not think about it at all.

                        The worst thing I can say about it is that the web frontend is noticeably slow (*).

                        The configuration isn’t hard (for the not-complicated projects we build). It’s not an overcomplicated mess. Our builds work and we didn’t have to employ staff full time to babysit CI (*2). When the build does break, the CI platform itself isn’t making debugging that any harder for us. It has first class support for building on MacOSX VMs which is super important for us because our needs include a need to ship iPhone apps.

                        One slightly weird thing is their ops people seem to be bizarrely fastidious about updating the status page: they mark it as status yellow at times when, as far as I can tell, everything is working perfectly. I’ve never seen it fall over without the status display being red (and not for long then).

                        (* it’s got AJAX performance disease: absolutely every button on screen will cause a loading spinner to appear and a bunch of network requests. Automated time to first paint metrics probably look amazing but time to first useful paint is really bad for every single screen. It feels like maybe someone got too enthusiastic with the code-splitting and now whatever you try to do, the code for that feature isn’t yet loaded until 1500ms after you click the button.)

                        (*2 I knew of a company that tried to use buildbot in production and it sucked so bad they hired a core buildbot maintainer to work on it full time and they still did not have acceptably working CI so… that sucked for them.)

                        1. 2

                          I’m using Hercules CI, and I’m happy with it. We have a Hercules agent running on a 64GB Hetzner machine, connected to Cachix. This now means that compiling ~160,000 lines of Haskell and a few thousand lines of Elm, plus running all the integrated tests, all takes about 8 minutes. When I deploy the app now, it just pulls the pre-built binary from Cachix and sends that to AWS together with any new configuration and systemd units, so deploys now take about a minute.

                          1. 2

                            travis, gitlab for hobby work.

                            jenkins for day job.

                            if I was looking for a long term day job thing, I’d be aimed at tekton.

                            1. 2

                              Used Jenkins for years (and Hudson) and CruiseControl and Bamboo, but haven’t really kept in touch. Drone was the best thing I’ve ever used, but Gitlab CI seems to work ok as well (current job).

                              1. 2

                                Gitlab-ci is flexible and free but personally I have been eye-ing Jenkin-X with the model of knative’s new build: Tekton pipeline.

                                Alternatively, Bazel/Buck support for monorepo seems quite appealing to me but I never get enough time to learn them. They are build tools but with remote build and remote caching supported with build rules for maven/docker/node/go etc… Check out https://github.com/bazelbuild/bazel-buildfarm

                                1. 2

                                  Gitlab CI and Jenkins

                                  1. 2

                                    Gitlab CI when I get to choose, but usually I’ve used Drone or Jenkins

                                    1. 2

                                      I tried out several of the recommendations here. My favorite so far is Travis–it was the fastest to get working out of the lot, slick, minimal. (Github Actions took me a bit of trial and error around getting my billing settings right, for just a demo repo.)

                                      But I think I’m still recommending Jenkins for work. We do a lot of ARM builds, and Travis is the only online option that has prebuilt arm targets. But it’s new and still a bit buggy looking, so old reliable it is. Exciting to see so much competition though, and Travis might be worth revisiting again even in a few months.

                                      1. 5

                                        Travis has been bought by a firm known for buying and operating SAAS without making further modifications, so I wouldn’t hold my breath for improvement.

                                      2. 2

                                        Jenkins, TravisCI, AWS CodePipeline/CodeBuild, plain old git hooks + bash.

                                        1. 2

                                          At work we use CircleCI. It’s alright. The UI is not very user friendly in my opinion (no search, no obvious way to just get a list of all your projects - the only such list lives under the Add Project tab, which is something you just have to remember). There was a period about 6 months ago where outages were pretty frequent, but that seems to have gotten better. The product doesn’t really seem to be evolving or improving at all, so I don’t know what they’re working on, but presumably the company is doing fine.

                                          1. 2

                                            I use a mix of Travis CI and Cirrus CI. Cirrus is super fast and has the benefit of supporting FreeBSD as well as macOS and Linux.

                                            1. 1

                                              We use Concourse and GoCD. I am not a fan of Concourse personally. GoCD is nice though.

                                              1. 1

                                                Curious what issues you have with Concourse? I’ve always been interested in using it, but that may be mostly UI focused.

                                                1. 2

                                                  The main issue I have is that it runs in a container, so if you are trying to run smoke tests in an integrated containerized environment, the performance of the nested containerization is extremely poor. My nit would be that the UI is beyond awful and still somehow manages to make my laptop fans start spinning the moment I open the page.

                                                  1. 1

                                                    Ah that’s sad to hear. Thanks for the info.

                                              2. 1

                                                My own one called Cinderella now :) (it’s extremely new, so much stuff is not polished)

                                                I did not find any CI engine that was lightweight enough for my use cases (single user, bare git repos, no docker).

                                                I have used Gitlab and Jenkins and found them both too big for me. I also used Travis, but it’s hosted and only free for FOSS repositories. I also found other solutions, but none of them worked well with my ecosystem.

                                                Thus, I decided to develop a small CI engine that runs on the host itself (yes, no isolation) and integrates well into bare git repositories. You set a post-update hook that will execute cinderella and done.

                                                A small overview over my reasons and more CI engines I considered not appropriate can be found here: https://github.com/aufziehvogel/Cinderella/blob/master/ProjectOutline.md

                                                1. 1

                                                  Ohh this looks very interesting. Seems like an ideal candidate for good cross platform testing since you aren’t relying on Docker.