1. 30
  1. 6

    It seems to me that if one is going to go that far off the beaten path (i.e. not just running “docker build”), then it would also be worth looking into Buildah, a flexible image build tool from the same group as Podman. Have you looked into Buildah yet? I haven’t yet used it in anger, but it looks interesting.

    1. 6

      +1000 for Buildah.

      No more dind crap in your CI.

      Lets you export your image in OCI format for, among other useful purposes, security scanning before pushing, etc.

      Overall much better than Docker’s build. Highly recommend you try it.

      1. 3

        Added looking into it to my todo list, thanks for the suggestion @mwcampbell and @ricardbejarano.

        1. 2

          Im intrigued, what do you use for security scanning the image?

          1. 4

            My (GitLab) CI for building container images is as follows:

            • Stage 1: lint Dockerfile with Hadolint.
            • Stage 2: perform static Dockerfile analysis with Trivy (in config mode) and TerraScan.
            • Stage 3: build with Buildah, export to a directory in the OCI format (buildah push myimage oci:./build, last time I checked, you can’t do this with the Docker CLI), pass that as an artifact for the following stages.
            • Stage 4a: look for known vulns within the contents of the image using Trivy (this time in image mode) and Grype.
            • Stage 4b: I also use Syft to generate the list of software in the image, along with their version numbers. This has been useful more times than I can remember, for filing bug reports, comparing a working and a broken image, etc.
            • Stage 5: if all the above passed, grab the image back into Buildah (buildah pull oci:./build, can’t do this with Docker’s CLI either) and push it to a couple of registries.

            The tools in stage 2 pick up most of the “security bad practices”. The tools in stage 4 give me the of known vulnerabilities in the image’s contents, along with their CVE, severity and whether there’s a fix in a newer release or not.

            Having two tools in both stages is useful because it increases coverage, as some tools pick up vulns that others don’t.

            Scanning before pushing lets me decide whether I want the new, surely vulnerable image over the old (which may or may not be vulnerable as well). I only perform this manual intervention on severities high and critical, though.

            1. 1

              Thanks for the response. What are your thoughts on https://github.com/quay/clair which seem to replace both Gripe and Trivy?

              1. 1

                I haven’t used it, can’t judge.

                Thanks for showing it to me.

          2. 1

            I’ve never used dind, but have only used Jenkins and GitHub Actions. Is that a common thing?

            1. 1

              IIRC GitHub Actions already has a Docker daemon accessible from within the CI container. So you’re already using Docker in Whatever on your builds.

              There are many problems with running the Docker daemon within the build container, and IMO it’s not “correct”.

              A container image is just a filesystem bundle. There’s no reason you need a daemon for building one.

          3. 4

            I have not looked at it, but my understanding is that Podman’s podman build is a wrapper around Buildah. So as a first pass I assume podman build has similar features. It does actually have at least one feature that docker build doesn’t, namely volume mounts during builds.

            1. 2

              If I remember correctly, the Buildah documents specify that while yes - podman build is basically a wrapper around Buildah - it doesn’t expose the full functionality of Buildah, trying to be more of a simple wrapper for people coming from Docker. I can’t recall what specific functionality was hidden from the user, but it was listed in the docs.

          4. 1

            Unfortunately my current pipelines are still stuck with kaniko in GitLab-CI, which is quite slow - even two minutes “build” time for a bigger rails image and all layers cached, the moment you have nodejs it becomes even slower - this is mostly due to biggest penalty being file io. But it does its job. Passing secrets works via --build-args since they aren’t stored in the image manifest (in contrary to docker which will persist the args). Since it needs some a bit of scripting I wrote for the other devs a template that they can reuse, bonus is I can get to introduce additional changes to all future image builds in case I want to add improvements later.

            1. 2

              I worked on a lot of optimizations to gitlab and gitlab CI over the years. To bring jobs under a minute, you need to do A LOT of local caching. That involves maintaining a data vol locally and mount it onto the job container