1. 12

  2. 13

    I’ve said this before, and probably will say it again in the future: kubernetes reminds me, and not in a good way, of the old days of sendmail. I did not expect “the tool’s configuration is so difficult even for experts that the only option is to use a tool that takes input in some other format and outputs a configuration” to come roaring back.

    1. 1

      Back in the day when I wore my sysadmin hat, I had the Bat Book, the thousand page tome (literally) that just covered the Sendmail configuration file, which I had to use to modify sendmail.cf to do what we wanted to do. Hideous thing. Glad I no longer have to do worry about that anymore.

      1. 1

        I thinking exactly the same thing yesterday, and writing a little story in my head:

        I used to work at a service provider. At that company, we ran a certain piece of software, which I will henceforth refer to as the Software. The Software was considered to be one of the most highly valued and critical pieces of infrastructure at the company; in many circles, it was simply unfathomable that a “real” tech company would NOT run it - why even have servers, if not to run the Software?

        Configuring the Software was extremely complex, to the point you didn’t just have subject matter experts in configuring it, but different subject matter experts for configuring it in different ways. The proponents of the Software argued that this complexity was absolutely necessary, so that the Software could handle all of the possible use-cases they had anticipated. The configuration of a large deployment was a holy document, which was the result of many engineer hours. Each of these configurations was unique, and familiarity with one configuration of the Software did not necessarily translate into understanding of other configurations.

        An entire industry developed around the Software. There were books, there were classes, there were conference tracks. There were consultants who would configure the Software for you. There were companies that would run the Software for you, as a service. None of this was cheap, but it was all considered reasonable, and again, absolutely necessary.

        This software had a few competitors, but they were largely dismissed. Alternatives to the Software were not as configurable. There was a great amount of fear in the industry that alternatives might not be compatible with the Software. All the Big Companies ran the Software, and they obviously knew what they were doing.

        What’s Kubernetes? I was talking about sendmail.

        1. 1

          I think it’s fine that Kubernetes is very configurable, but it’s not fine that these config files are presented as the human interface. Nor are Helm or Kustomize or etc adequate. Unfortunately Kubernetes is missing the concept of “delete missing resources” (there’s some sketchy —purge flag for doing this, but I don’t trust it), otherwise a well-written config generator would be fine (sounds like OP has been burned with badly written config generators).

        2. 5

          TL;DR—use the tools I use to manage YAML files.

          1. 3

            Yup, and also, strange that Helm isn’t mentioned, given that’s pretty much Industry Best Practice for handling this exact problem.

            1. 2

              If you’re using something like the Helm–k3s integration, then Helm can be relatively lightweight; otherwise, it’s a hassle. The mandatory boilerplate is heavy and confusing. Just like with Docker, a registry is required. Composition of charts is messy.

              1. 1

                If you’re willing to use them, git submodules can replace a registry.

            2. 2

              Kustomize might have a silly name, but it is built into the standard Kubernetes tools at this point, and even has documentation. I read this article more as a recommendation to not write a brand-new tool just to replace Kustomize.

              1. 1

                Yeah, this does look opinionated, but it’s better than a tool we built at the last place that conflated business logic, templates, and data. All in one giant erb. So you’d need to know Ruby, ERB, k8s YAML/theory, terraform, AWS, and whatever language your service was written in. Plus, if you had to add a section, you’d have a bunch of duplication, logic, loops, and shell outs all in the ERB.

              2. 3

                Uhhhhhhh….. Too late?


                1. 2

                  In $PREVIOUS_JOB, cdk8s hit the right spot for us. It allowed us to build useful abstractions on top of kubernetes in a programming language we were used to, catch typos and inconsistencies at build time that could have bubbled into deployment errors, via static analysis. On top of that, we were able to never have to write the common Helm {{ $.Values.path.to.value | indent 14 | trim }} atrocity, which mangles both logic and formatting.

                  It has support for CRDs as well, which does not hurt.

                  To each their own obviously, but it really solved the issue of managing both configuration complexity and YAML format correctness at the same time.

                  1. 1

                    Oh wow, cdk8s has matured nicely! I have a bunch of mind-bending Helm charts that just yesterday were the cause of a hard-to-spot issue. I might start looking into converting some of them to cdk8s.

                    1. 1

                      Wow, cdk8s looks really nice. At work we’re currently using Jsonnet to define our manifests but Jsonnet is ugly. I might try to push for this.