1. 18
  1.  

  2. 14

    The (almost) same idea is also known under the name of “Configuration complexity clock” (by Mike Hadlow): http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html

    1. 4

      I thought the same thing! Thanks for reposting this article.

      Both these articles have a flavor of satire/this is insanity. One thing I mused on recently in my last two jobs is maybe we should lean in to this pattern, because a lot of the choices are sensible, and set out to establish formal or informal best practices as to when to move on from one stage to the next, and how best to manage, document, and plan the transitions.

      1. 2

        I don’t think this is insanity, although its comical that even principal engineers get stuck in this cycle without realizing it.

        To your point, I do think that we should absolutely lean into this and carve out some best practices. I might write a follow-up soon!

        1. 1

          Thanks! I wasn’t sure how to read your tongue in cheekness!

      2. 1

        Yep! Mike’s post was top of mind when I wrote this. I thought I’d update it a decade later with how its changed (albeit only slightly) today.

      3. 4

        Lots of software practices follow a similar pattern, just over a much longer timescale. The pendulum swings eternal.

        1. 1

          Hardware has the wheel of reincarnation which is a similar concept. I submitted the original paper earlier this week.

          1. 1

            Nicely identified. I feel like abstraction inversion is another example (or perhaps a rewording?) of the swinging pendulum. X isn’t perfectly suited, so we build Y on top, and then Y isn’t perfectly suited, so we build X’ on top which rebuilds some of the features of X in a worse way…

            On the sort-of-plus side I think being aware of this and being willing to use old things/old design practices is kind of a superpower for software design… it can make things a lot simpler if you just follow the path that’s already been trod http://catern.com/reuse.html

          2. 2

            One should short circuit the process by just sticking to hard coding, IMO. http://catern.com/config.html

            1. 1

              It’s a useful separation to have the same binary running two different configurations (prod vs staging vs. dev, etc.). I’d go as far as to say that if you don’t use that flexiblity for a given config variable, then you can gradually migrate it into the code. In other words, configuration has a place (IME more often than not), but if you have a small/simple system, it’s true you might not need it at all.

              1. 3

                It’s a useful separation to have the same binary running two different configurations (prod vs staging vs. dev, etc.)

                Why is that? You can have the exact same libraries with a different main function - what does it matter if the main function is different? “A different main function” is the same as “a different configuration”.

                Any property you could enforce by constraining what the configuration can do, you could enforce by constraining the interface of the libraries the main function is calling (I discuss this in http://catern.com/config.html as linked above)

                1. 1

                  So are you saying that if you need have a web server that can listen on either port 80 or port 8888, you would build 2 different binaries with 2 different main functions for that?

                  If so, that seems silly for a whole bunch of reasons :)

                  One reason is that often there are admins/operators who run binaries and want to reconfigure them without rebuilding from source. Another reason is that deploying binaries to large clusters can be surprisingly expensive (e.g. to the point where people use BitTorrent to optimize it). Deploying configuration is cheap by comparison.

                  Those are 2 reasons, but there are many more, like being able to do a gradual (non-atomic) upgrade of clusters via config changes (something I asserted your last blog posted ignored). Fast rollbacks are another reason, etc.

                  1. 1

                    One reason is that often there are admins/operators who run binaries and want to reconfigure them without rebuilding from source.

                    Is your concern that rebuilding from source is slow? If so, use an interpreted language or rely on fast incremental builds.

                    Is your concern that admins/operators shouldn’t be writing code? I don’t think there’s a principled reason for that… even if they don’t know how to program it should be no harder for them to compose some libraries together than it is to write configurations (the code involved should be basically the same either way) - and they’ll get much better error messages too.

                    Another reason is that deploying binaries to large clusters can be surprisingly expensive (e.g. to the point where people use BitTorrent to optimize it). Deploying configuration is cheap by comparison.

                    Do you mean purely in terms of size? But a binary should be smaller than a (human readable) configuration, because e.g. it can store {“implementation”: “very_cool_fast_version”} as a single byte enum at the right offset in a struct. So binaries should be cheaper to deploy than configuration, in terms of size.

                    Fast rollbacks are another reason, etc.

                    Why shouldn’t I be able to quickly roll back to an old binary? That’s an ability that one desperately needs anyway. Or are you implicitly implying that my configuration will be dynamic and live-updatable while the system is running? That’s a whole different question and, as I’m sure you know, can be very tricky to get right. But if you support that ability then there’s no reason why your initial configuration has to come from a separate file instead of hardcoded in your binary.

                    1. 4

                      I’d like to live in a world where all of this is feasible and can be easily budgeted for. Examples from my current work:

                      use an interpreted language or rely on fast incremental builds.

                      We have about 2.5 million lines of C++ that gets deployed to remote embedded-ish systems. Often changes need to happen in distant places over unreliable cellular internet. I’d love fast incremental builds; as it is an incremental build takes at least 1 minute on a rather beefy machine, and 10 on some of the machines we’d like to be using in the near future. We also can not put our source code on these remote machines, for IP reasons.

                      Is your concern that admins/operators shouldn’t be writing code?

                      Our operators/users are experts at operating high-tech machinery, not experts at coding. They have to worry about things like “make sure this machine doesn’t break itself or hurt anyone”; learning git is not very high on their priority list, and I’d frankly prefer to keep it that way.

                      Do you mean purely in terms of size?

                      A compressed binary deployment of our software is about 2.3 gigabytes. This is literally all code, libraries, and operational data. If we put the engineer-hours into cutting out all the stuff that changes rarely so we could deploy only changed code, setting up a new deploy pipeline to handle it, and training people to use it, we can probably get it down to 500 megs. (Hopefully we can get the time to do that sometime this year.) Again, remote updates usually have to happen via crap cellular internet. If rsync suddenly stopped existing the company would implode in two weeks.

                      Why shouldn’t I be able to quickly roll back to an old binary? That’s an ability that one desperately needs anyway.

                      And hopefully someday soon I’ll get the money to implement it!

                      Or are you implicitly implying that my configuration will be dynamic and live-updatable while the system is running?

                      We are desperately trying to make this not the case, but currently this is still necessary for field debugging and R&D. Having two packaging/deployment pipelines, one for production and one for development, would make it easier to handle this, and we are slooooooowly meandering towards that, but it takes a lot of up-front work and training.

                      It’s been very interesting working here, and after bitching about it for two years I’m now a lead dev on the team responsible for actually finding solutions to these problems. It’s gonna be fun! But one of the things I’ve learned is how much of the world assumes “computer tech” equates to “a server living in a datacenter”. Make that not the case and suddenly a ton of existing systems and solutions stop working.