1. 51
  1. 26

    Long ago had a discussion about conscription with someone who was, I believe, a reserve officer in the Swedish military (bear with me, it’s related). I had shortly before spent a lot of effort escaping conscription in Israel; he thought conscription was an excellent idea, because otherwise “only people who want to be in the military end up as soldiers.”

    The last paragraph in this article reminded me of his argument:

    Borg sold me on the idea of this kind of computing, and Kubernetes un-sold me on it. I currently believe that the very best container orchestration system is no container orchestration system, and that effort would be well spent avoiding that rabbit hole at all costs. Obviously, that idea is incompatible with building a container orchestration system :).

    There is an interesting point here, where the people who build technology X are the people who think X is a good idea; the people who think X is a bad idea (and might be better informed about what can go wrong!) are not going to write X, but might well be better at it.

    1. 9

      Agreed. I think that’s one of the eldritch secrets of software engineering.

      1. 7

        This is exactly my conclusion too. k8s is a product for people who do not want to set up load-balancing and auto-scaling, and do not know how to package up and deploy their applications. These people are willing to use a product that does all of these and million things more in a way how they see fit doing such things. In reality it is always specific to your organization, problem, team. I bet k8s is able to help where engineering culture is not that strong or where people think this is going to be a faster way of solving these problems at once. In my experience this ends up:

        • they build a black box solution and when it breaks they need external help
        • they spend more time on learning k8s than spending time on building (pretty much choosing) building blocks of this problem domain

        Obviously these things are anecdotal and there is a great resource on the net about how k8s caused an outage for some companies (https://k8s.af/). I really like to read that time to time.

        Just like many other things in engineering, this is a hype train and the need of using k8s is largely driven by FOMO instead of driven by a real need. I think for a healthy discussion it would be great if any engineering organization contemplating using k8s should start with why and what, instead of how. I have lived through many hype waves in the last decades in IT and it was always funny to see how people through out reasoning through the window in situations like this. I believe that k8s is not going to be a thing 10 years from now. It is simply too complex, fragile and error prone to be remain successful. Maybe I am wrong.

        1. 4

          there is a great resource on the net about how k8s caused an outage for some companies (https://k8s.af/).

          Even if I think this helps as an argument, I think it’s fallacious. Why isn’t there the same for MySQL? Or even Oracle? Or… C++?

          What I mean is that all those are complex too and incident with those components happened in the past too. That doesn’t mean that it’s hype or bad.

          The way I see it is that instead of being hype and that only dumb people use it, I feel it’s mostly a step forward toward a shift in operations, it’s probably bloated etc, but it’s still a step forward.

          1. 3

            Even if I think this helps as an argument, I think it’s fallacious. Why isn’t there the same for MySQL? Or even Oracle? Or… C++?

            It does happen for MySQL. Rarely for Oracle since you pay Oracle a lot to prevent it. C++ isn’t a service, so it’s not in the same vein.

            1. 1

              Well there is the general engineering principle that every moving part can cause bugs / problems / outages. So you should minimize the number of parts to the ones necessary to solve the problem.

              If you have a single service, then you don’t need Kubernetes. It would be just another thing to cause an outage.

              On the other hand, if you have dozens of different services authored by different teams, Kubernetes may benefit you. The benefit has to justify the cost.

              A lot of people liked this comment on when to use Kubernetes: https://lobste.rs/s/kx1jj4/what_has_your_experience_with_kubernetes#c_qonwag

              If you don’t need MySQL, then you shouldn’t use MySQL either. (But you also probably shouldn’t construct your own DB out of flat files or a key-value store, etc., unless that really solves the problem)


              An analogy is the recent news about doorbells going down because AWS was down. If you don’t need the features of an internet-connected doorbell, then don’t get one. If you do, then it may be worth the extra problems.

              Personally I bought a Nest, but I never connected it to the Internet for exactly this reason. I wanted to see if it worked reliably without Internet, and it dose (at least the one I bought 5+ years ago does).

              1. 1

                Totally agree. I was just pointing out that saying we shouldn’t use Kubernetes because there are so many failure modes like it’s written on this website.

                Many software components have many failure modes, and you won’t know them all. The best thing you can do is have a good approach to this fact.

          2. 3

            might well be better at it

            In my experience this counts double for ops roles: have people do that who don’t really like doing it. They’ll make sure it’s stable and simple, so no-one will ever bother them about it ever again.

          3. 4

            I am a big fan of hashicorp nomad too, it gives you a lot more control and feels like a better building block for a system.

            1. 2

              It is sad that CoreOS’ fleetctl didn’t launched as it was mostly it - services scheduler on top of systemd. Probably the main problem was that it was too focused on Docker integration, however I feel like there should be revival of that concept and I even wanted to work on something like that in Erlang. This shouldn’t be that hard to handle services launching as the DBus API for systemd is quite well documented, so you can work on top of that.

              1. 2

                An interesting take: Kubernetes API for systemd

                https://github.com/miekg/vks

              2. 1

                An interesting read, but at some points I asked myself whether I and the author live in different worlds:

                • AFAICT, he suggests putting all pods of a cluster together in a single broadcast domain and using IPv6 autodiscovery. (big broadcast domains are not a good idea)
                • There actually are some very simple options for Kubernetes networking that function like this
                • I think the Kubernetes networking model of having a network namespace with an IP for a pod is conceptually quite simple, certainly easier than running in the host network namespace and needing each service to bind on ports passed in an as environment variables or whatever. The alternative suggestion of using Docker-style port forwarding from the host network namespace I find even more confusing.

                I do want to point out that I’m a happy user of the author’s MetalLB in our Kubernetes clusters though.