1. 10
  1. 6

    The hype cycle around Kubernetes has been interesting to watch.

    What’s left after close to a decade is (in my opinion after deploying K8s at two different companies at this point) are couple very good managed providers (GKE probably being the best, EKS is fine), and some good basic tooling to get most of your workloads running on there stably. Kubernetes, when left to handle stateless workloads that need to auto-scale, is pretty hard to beat for costs and performance vs. higher level abstraction managed offerings like Fargate with a bit of a learning curve.

    There are some bodies littered on the road behind us though, namely:

    • running your own cluster NOT managed by a major provider on cloud VMs takes a lot of work (doable! but a lot of work.), bare metal is borderline futile
    • YAML was a mistake
    • statefulsets and PVC’s are really provider sensitive and can be a major pain still (there are a couple notable exceptions to this, including CockroachDB running their managed offering on GKE with PVCs)
    • security and hardening are non-trivial though managed providers help a ton here
    • most of the ecosystem not created by Google can be written off

    I’m curious what the long term story around Kubernetes is, but with a solid core of primitives it handles a very specific slice of infrastructure really well, and I’ve overall been happy with it.

    1. 6

      My suspicion about k8s is it’s not interesting for scaling (most people don’t need to scale), but more for a configuration based approach to service management. You could build a modern day MMC or cPanel based on it.

    2. 4

      I suppose this opinion will be unpopular, because proprietary services are supposed to be inherently bad, but I’m disappointed that, AFAIK, neither GCP nor Azure currently have something equivalent to Amazon ECS as an alternative to Kubernetes. ECS isn’t merely a proprietary alternative to Kubernetes; it’s a container scheduler that has a shared, multi-tenant control plane and directly integrates with other AWS services such as VPC and ELB rather than forcing you to use them indirectly through committee-designed, common-denominator APIs (e.g. ingress controllers). If the author had been hosting on AWS rather than GCP, ECS would definitely have been the way to go.

      I want to unpack the bit about the control plane (not control panel as the article said). Kubernetes requires each cluster to have its own control plane, including an etcd cluster, an API server, and other stuff. And, last time I checked, a Kubernetes control plane has fairly high baseline resource consumption. When I installed k3s (stripped-down Kubernetes) on a bare-metal box last year, it had ~10% CPU usage while the cluster wasn’t doing anything. Given this, and the fact that the Kubernetes control plane is designed to be single-tenant, I would not be surprised if the managed Kubernetes services spin up one or more VMs per cluster to run the control plane. Few SaaS developers would choose to implement their product that way when designing for multi-tenant SaaS from the start, but Kubernetes isn’t designed for that. Think of the control plane as being like one of the open-source databases offered by Amazon RDS, Google Cloud SQL, etc. So it’s entirely reasonable that the managed providers charge what they do for the control plane.

      Having said all that, given that the author’s product seems to be compute-bound, and presumably doesn’t require instant startup of containers, a container cluster (whether Kubernetes or something like ECS) might be pure overhead in the first place. IMO the ideal solution would be to use something like GCE managed instance groups or EC2 auto-scaling groups, but with some way of turning something equivalent to docker-compose.yml into an immutable machine image that directly runs the container(s). LinuxKit is the most well-known implementation of this approach, though I’m not sure how well-maintained and suitable for production it is these days.

      If fly.io had VMs with GPUs, it might be a great choice for this use case, particularly with the new Machines API.

      1. 3

        I would not be surprised if the managed Kubernetes services spin up one or more VMs per cluster to run the control plane

        I am not going to contradict you, but it is possible to run a kubernetes control plane in containers (a cluster created with kubeadm does that!) so it’s entirely possible to run the control plane for customers in a kubernetes cluster (or ECS or …).