There are some wilder ideas out there that may help with reselling Kubernetes, but they involve messing with assumptions of vanilla Kubernetes. One is https://github.com/krustlet/krustlet which might let you reduce the overhead of containers and (speculation) maybe overcome the restrictive “110 pods per node” limit.
In contrast, there’s https://github.com/elotl/kip which spawns a whole cloud VM for every pod that is automatically torn down afterward.
To avoid giving each customer their own control plane, there’s https://github.com/loft-sh/vcluster which would give each customer a namespace in a single cluster (one control plane) that “looks” like a standalone cluster, allowing them to use cluster-scoped resources.
not meant to distract from the points you’ve made, but the “110 pods per node” limit is easily adjustable.
Easily adjustable yes, but not safely adjustable.
I have personally raised it (96 cores is too many to waste on only 110 pods) but at 80/90 pods I’m already getting issues with pod launch times and volume mounting timeouts (both PVCs and configmaps/secrets). Kubelet and the container runtime just have to do way too many things way too frequently.
It might be workload dependent: like if all your workloads are long-running and don’t mount many volumes, I could see the limit being raised. But for whatever heterogeneous mix of stuff I’ve got, 110 pods is indeed looking like an upper bound.
Yeah, 110 pods was experimentally determined in load tests.
The way to increase it is to improve the performance of core system components.
Using an alternative kubelet, you’d need to redo the measurements
I’d be interested to trade notes on the topic as I’m not seeing the same thing on our 96 cores servers.
in the world of mobile telecoms, MVNO (mobile virtual network operators) re-sell the mobile infrastructure managed by Tier 1 providers (eg ATTs of the world)
But they (MVNOs) provide their own billing/support/customizations/etc (depending on the level of the reseller).
Some just do marketing, but use the rest from the tier 1s.
Looking back at how that was done – there was a legislation / regulation that needed to make the Tier 1s to support this.
Just wanted to mention the above analogy for your consideration.
I guess, there should be, ideally, more value than just cluster re-selling.
I can think of many add-on value opportunities, I am sure you have some in mind as well.
But I would think of the additional value the reseller providers, this can expand to multi-public-cloud capabilities, etc.