1. 22

Crustaceans, how do you do parameterized configuration?

Problem setting:

  • You have config files where some parts are parameterized - they depend on the particular instance being configured (e.g. IDs, API keys, rules)
  • These config files need to be tracked (VCS) and automatically applied (CI) to multiple instances of the thing being configured.

My current solution:

  • envsubst, bash, and [the thing that applies a config]. (i.e. a bash script that takes CLI args, sets env variables using those args, uses envsubst to replace $VARIABLES by values in the config file, and applies the resulting config)

Why the current solution sucks:

  • The config and the parameters are both structured data (some combination of maps, lists and primitive types), not arbitrary text. Generating / preserving that structure is tedious and error-prone when using pure bash to set the parameters.
  • Dependency/aggregation is cumbersome. If you have a (parameterized) configuration for component A stored in repo git://A, and configuration for component B stored in repo git://B, and you need to apply both configurations with some shared values, you need to explicitly check out each repo at a given branch/tag/commit and then run the respective shell script, providing the parameters as CLI arguments. The CLI syntax could either vary or is duplicated between A and B.
  • There is no standard way of tracking which specific config was applied - the bash script applying a given config can do whatever it wants. No record of the actual values of the environment variables used by envsubst is kept.

My attempt at an improvement:

  • Use go-templates to parameterize the config files and JSON (or YAML or TOML) to define the parameter values. (https://github.com/sgreben/render)
  • Record the provided parameter values

Why the improvement still sucks:

  • No notion of dependencies/packages/interfaces, this still needs to be done “manually”
  • Too flexible: there are too many ways of achieving the same thing using the template tool.

I’m not satisfied. Maybe someone else has figured this out already? Hence, the question - how do you deal with this?

  1.  

  2. 16

    I’m not a devops guy, but I use Nix for this sort of thing in my personal projects.

    A less invasive approach might be Dhall, which I’ve not used but AFAIK you can write your config with functions, etc. and run a command to “compile” it into e.g. a JSON file.

    1. 7

      I can vote for Nix as well, it can do exactly what OP needs. Nix is a lazy, functional language pretty much made for writing configuration. Specifically Nix can solve their problem because:

      • It can build output paths, which can for example contain a from Nix generated config, along with the parameters used.
      • Fetch Nix expressions from a repository and evaluate them, this let’s you parameterize your repos.
      • Nix has lists and attrsets (records), which represent structured data.
      • Makes it very easy to deal with environment variables

      Edit: As an interesting example, check out my ssh keys NixOS configuration which sets up an nginx virtual host for all my public keys (declared in another Nix file) including an automatically renewing Let’s Encrypt certificate. The function to convert an attrset to a folder structure I wrote is here (should probably make a PR to add it to nixpkgs).

      Nix has so much more to it though, there’s NixOS built on Nix with an insanely powerful module system, there’s NixOps for deploying NixOS machines. And most importantly nixpkgs and the Nix package manager which builds packages in a reproducible fashion and enables a lot of stuff you wouldn’t even have thought of. For the interested I can recommend dropping in #nixos on Freenode.

      1. 4

        thumbs up, dhall looks esoteric, but actually seems like an amazing solution for making statically typed config generators.

        1. 2

          Both nix and deals look neat, but for ops the last thing we want is more esoteric things. Life is hard enough using normal tools!

          1. 3

            I don’t think you understand the point of it, sometimes you need to step outside of the normal things to make life simple again.

            1. 3

              I’d recommend considering that the reason life is so hard with normal tools is because they’re all trying to solve the same problem in the same fundamental way. Trying new tools that work the same way is going to ultimately end up in the same painful position all the other ones have lead us to. This is why I recommend trying Nix.

              1. 1

                I totally agree…

                I recently shifted my desktop to nix, it is fundamentally different, and solves all these problems better. I was experimenting with broken configs, and could revert the whole OS atomically via grub. That is power I haven’t seen anywhere else, and yet I didn’t even need to know how it works, it was just an option presented to me when I booted.

                My next set of servers is going to be nixops, or something directly inspired by it.

              2. 1

                I think it’s relative: I think if you’re using Chef/Puppet/Ansible/Docker/etc. then Nix is just another alternative, which IMHO is cleaner. It’s got a decent community, commercial users, etc. so if you’re thinking about adopting one of these technologies, Nix is definitely worth a shot and I wouldn’t consider it esoteric at all.

                The problem with Nix for a situation like this is that it’s quite heavyweight; just like switching a server over to Puppet or something is also a heavyweight solution to the problem of “put different values in these config files”. I would say Dhall is certainly esoteric, but I think it’s worth a look for solving this problem since it’s so lightweight: it’s just a standalone command which will spit out JSON. In that sense, I think it’s a more reliable alternative to hand-rolled scripts, which wouldn’t require any extensive changes to implement (call it from bash, like anything else; use the JSON, like anything else).

          2. 7

            We use Ansible, which has some support for this already. I think that that might be close to what you need.

            1. 3

              I’ll second the Ansible suggestion. You can set variables for each host and use them.

              Apart from using variables in any playbook tasks as linked above, you may be interested in the template task specifically: you give it a template for a config file, and Ansible fills in the template’s variables and deploys it to the target location.

              1. 2

                To me ansible seems like a poor scripting language. I don’t see the point of it to be honest.

              2. 4

                Sounds like you might want to look into using a configuration management tool - ansible is a good place to start and is an easy step to make from using bash scripts to configure environments.

                For the specific problem of parameterizing configuration files, I use Chef/Ansible/etc and their templating functions to write a file with variables inserted. In the case of Chef, I did some work to write a custom resource that accepted a Ruby map structure containing my configuration and create a JSON config file, which avoided the possibility of generating a syntactically invalid configuration file.

                These days I work more on the software development side of things, and am starting to find that designing software to have the simplest configuration options possible makes my life much easier when it comes to deploying that software and managing it’s configuration.

                1. 3

                  We simply keep our configuration in Python, so that any kind of parameterization, validation or logic becomes trivial to add. We tend to store the actual output as JSON since that’s what most things want and is language independent.

                  1. 3

                    Mild heuristic evidence for Python being the new Lisp, as this has long been a common way to configure Lisp programs (.emacs files are like this, among other examples).

                    1. 1

                      Never thought about it that way but it makes sense :)

                      1. 1

                        Strong heuristic evidence for “Mild heuristic evidence for X being the new Y” meaning “That aspect of X reminds me of Y.” ;-) Having joked that, do you feel there are other ways in which Python seems like the/a new Lisp? Because that sounds like an interesting conversation.

                        On the code as config subject, I think most of the interpreted language communities have some number of projects that do this – I know for a fact it is also done in Ruby and Lua, and of course JSON was explicity based on this principle. People do it in Bash, too, I suspect?

                        1. 3

                          do you feel there are other ways in which Python seems like the/a new Lisp?

                          I think I first ran across the idea from Peter Norvig.

                          1. 2

                            Neat article, thanks for sharing the link!

                    2. 3

                      Kubernetes uses ConfigMaps: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/

                      They are basically YAML/JSON properties which can be sent to containers in various ways:

                      env:
                      - name: LOG_LEVEL
                          valueFrom:
                          configMapKeyRef:
                              name: env-config
                              key: log_level
                      

                      Kubernetes handles the rollout of changes, and since a lot of infrastructure tasks are pre-defined (like routing from one service to another ala istio) there’s a lot less one-off config changes that you need to do. They support literals, files and directories. You can also do secrets: https://kubernetes.io/docs/concepts/configuration/secret/

                      1. 2

                        We adapted golang’s text/template as a cli:

                        https://github.com/blendlabs/template

                        1. 2

                          If I understand the problem correctly, I’m surprised nobody has mentioned Consul + consul-template.

                          You’ll first want to set up a consul cluster of 3 or 5 hosts (for redundancy) then configure consul as an agent as well as consul-template templates on the hosts that need to get their configuration dynamically updated. The configuration that you want can be pulled from the Consul key-value store. This key-value store can also be used for “any number of purposes, including dynamic configuration, feature flagging, coordination, leader election, and more.”

                          1. 1

                            I like envsubst format and bash scripts because you don’t need to learn new lang/format. If I was to adopt a new system, I’d use something very general purpose, like the same language used for the project, or Nix or Dhall. I’m still trying to find a use for lambdatext, but it seems to be too specific for most applications.

                            1. 1

                              Nthing the other recommendations to use an automated configuration management system like chef, ansible, puppet, or salt. They all support doing this pretty smoothly. For a real world example, here’s one where I conditionally template some ports (that vary by host) into an nftables config.

                              1. 1

                                We use SaltStack for this. It has a concept of Pillar which is specifically meant for storing structures of data and distributing them to nodes in a secure manner.

                                Pillars also support what’s called a “renderer”. And if you use the GPG renderer you can put your sensitive data in version control, all encrypted.

                                1. 1

                                  I put “config” in code - indeed I reject the whole distinction. The things a config system needs to be good at are the same things a programming language needs to be good at. An inheritance hierarchy is often a good model for the way things need to be different on different hosts. I’ll pass one parameter to the program to tell it which environment it’s in, and then it will have some very simple logic to turn that into an object instance that I can read parameters as fields from; since this is a plain old value I can also easily e.g. expose it as JSON so that I can see what config a given instance is running with. I make changes the same way I’d make code changes: submit a PR, go through the process, deploy it.

                                  1. 1

                                    Recently I discovered yasha for similar uses. But, your render library appears to have more features than yasha!

                                    1. 1

                                      Maybe http://jsonnet.org/ could be of some benefit?

                                      “This case study illustrates that Jsonnet can be used to centralize, unify, and manage configuration for all parts of a cloud hosted multi-tier web application (a Mandelbrot viewer). Jsonnet centralizes configuration files for the various application software, database schemas and initial data sets, system configuration files, package manifests, software build configurations, image configurations (Packer) and cloud resources / connectivity (Terraform). Although the running example is deployed on Google Cloud Platform and uses specific application software, Jsonnet can be used to generate configuration for any application and (via Packer & Terraform) a wide range of cloud providers.”

                                      1. 1

                                        In the lesser known Cuisine (I’m the author) you have functions to ensure that lines are present in a configuration file (text_ensure_line) or replace an existing line with another (text_replace_{line|regex}). These simple functions actually cover quite a lot along with text_template (a wrapper around Python’s string.Template).

                                        Nix is definitely more elegant and guarantees to produce the same result, Cuisine on the other hand is about scripting common sysadmin tasks on running systems.