I find it fascinating that existing comments failed to actually examine the author’s point. To be pithy: Kubernetes manages objects which represent running processes. This is not much different from a typical high-level operating system, where processes are objects in a kernel. While we can certainly imagine better implementations of Kubernetes-style systems, they will generally have this same shape, not because Kubernetes is somehow excellently shaped for distributed systems, but because humans build operating systems in this shape, including distributed operating systems.
This is why it doesn’t matter whether the deployment is owned by an individual or a corporation; it doesn’t matter whether the deployment is made of containers, either. The key concept is that the process objects might not reside wholly on a single machine; the state itself is distributed. Distributed computation requires distributed tooling and scaffolding.
Yeah, what I find interesting about k8s is you could do basically configuration management over some kind of RPC like thing - sounds a lot like late 90s Windows administration than /etc hell.
It’s also remarked that OOP programming makes sense in a distributed world - guess what Erlang and k8s are in?
The fact that k8s requires scale to be usable is unfortunate.
The job of the OS is to schedule which software gets what resources at any given time. Kubernetes does the same thing. You have resources on each one of your nodes, and Kubernetes assigns those resources to the software running on your cluster.
ehh, who’s the you here? This is starting from the assumption that I have a lot of nodes, which is only true in the context of me running infrastructure for a corporation; the you here is a corporation.
The first difference is the way you define what software to run. In a traditional OS, you have an init system (like systemd) that starts your software.
again, again, define traditional. Who’s tradition? In what context? In a traditional OS, software starts when you start using it, and then it stops when you stop using it. The idea that everything should be an always-running, fully-managed service is something that’s only traditional in the context of SAAS.
The thing that makes me feel cold about all this stuff is that we’re getting further and further away from building software that is designed for normal people to run on their own machines. So many people that run Kubernetes argue that it doesn’t even make sense unless you have people whose job it is to run Kubernetes itself. So it’s taken for granted that people are writing software that they can’t even run themselves. I dunno. All this stuff doesn’t make me excited, it makes me feel like a pawn.
You’re right, you probably wouldn’t use Kubernetes as an individual.
I’ll take the bait a little bit though and point out that groups of people are not always corporations. For example, we run Kubernetes at the Open Computing Facility at our university. Humans need each other, and depending on other people doesn’t make you a pawn.
Given the millions and millions spent on marketing, growth hacking, and advertising for the k8s ecosystem, I van say with some certainty we are all pawn-shaped.
totally fair criticism. I think “corporation” in my comment could readily be substituted with “enterprise”, “institution”, “organization” or “collective”. “organization” is probably the most neutral term.
Humans need each other, and depending on other people doesn’t make you a pawn.
so I think this is where my interpretation is less charitable, and we could even look at my original comment as being vague and not explicitly acknowledging its own specific frame of reference:
In a traditional OS, software starts when you start using it, and then it stops when you stop using it.
again, who’s tradition, and in what context? Here I’m speaking of my tradition as a personal computer user, and the context is at home, for personal use. When thinking about Kubernetes (or container orchestration generally) there’s another context of historical importance: time-sharing. Now, I don’t have qualms with time-sharing, because time-sharing was a necessity at the time. The time-sharing computing environments of the sixties and seventies existed because the ownership of a home computer was unreasonably expensive: time-sharing existed to grant wider access to computing. Neat!
Circling back to your comment about dependency not inherently making someone a pawn and ask: who is dependent on whom, for what, and why? We might say of time-sharing at a university: a student is dependent on the university for access to computing because computers are too big and expensive for the student to own. Makes sense! The dependent relationship is, in a sense, axiomatic of the technology, and may even describe your usage of Kubernetes. If anything, the university wishes the student wasn’t dependent on them for this because it’s a burden to run.
But generally, Kubernetes is a different beast, and the reason there’s so much discussion of Kubernetes here and elsewhere in the tech industry is that Kubernetes is lucrative. Sure, it’s neat and interesting technology, but so is graphics or embedded programming or cybernetics, etc, etc, etc. There are lots of neat and interesting topics in programming that are very rarely discussed here and elsewhere in programming communities.
Although computers are getting faster, cheaper, and smaller, the computers owned by the majority of people are performing less and less local computation. Although advances in hardware should be making individuals more independent, the SAAS landscape that begat Kubernetes has only made people less independent. Instead of running computation locally, corporations want to run the computation for you and charge you some form of rent. This landscape of rentier computation that is dominating our industry has created dependent relationships that are not inherently necessary, but are instead instruments of profit-seeking and control. This industry-wide turn towards rentier computation is the root of my angst, and I would say is actually the point of Kubernetes.
we’re getting further and further away from building software that is designed for normal people to run on their own machines
This resonates with me a lot. At work, we have some projects that are very easy to run locally and we have some that are much harder. Nearly all the projects that can be run locally get their features implemented more quickly and more reliably. Being able to run locally cuts way down on the feedback loop.
I’m really looking forward to the built-in embed stuff in Go 1.16 for this reason. Yeah, there’s third-party tools that do it, but having it standardized will be great. I write Go servers and one thing I’ve done is implement every storage layer twice: once with a database, and once in process memory. The utility of this has been incredible, because I can compile a server into a single .exe file that I can literally PM to a colleague on Slack that they can just run and they have a working dev server with no setup at all. You can also do this with sqlite or other embedded databases if you need local persistence; I’ve done that in the past but I don’t do it in my current gig.
I write Go servers and one thing I’ve done is implement every storage layer twice: once with a database, and once in process memory.
In my experience the overhead of implementing the logic twice does not pay out since it is very easy to spin up a MySQL or Postgres database, e.g. using docker. Of course this comes with the disadvantage of having to provide another dependency but at least the service then runs in a similar environment to production. Usually spinning up a test database is already documented/automated for testing.
yeah, very often the implementation is just a mess of map[string]*Book, where there’s one Book for every model type and one map for every index, and then you slap a mutex around the whole thing and call it a day. It falls apart when the data is highly relational. I use the in-mem implementation for unit tests and for making debug binaries. I send debug binaries to non-developer staff. Asking them to install Docker alone would be a non-starter.
Suppose that you, an individual, own two machines. You would like a process to execute on either of those machines, but you don’t want to have to manage the detail of which machine is actually performing the computation. At this point, you will need to build something very roughly Kubernetes-shaped.
The difficulty isn’t in having a “lot” of nodes, or in running things “on their own machines”; the difficulty is purely in having more than one machine.
you don’t want to have to manage the detail of which machine is actually performing the computation
…is not a problem which people with two machines have. They pick one, the other, or both, and skip on all the complexity a system that chooses for you would entail.
I struggle with this. My reflex is to want to have that dynamic host management, but fact of the matter is my home boxes have had less pages than my ISP in the past five years. Plain old sysadmin is more than enough in all of my use cases. Docker is still kinda useful to not have to deal with the environment and setup and versions, but like. A lot of difficulty is sidestepped by just avoiding to buy into the complexity.
I wonder if this also holds for “smaller” professional projects.
Unfortunately, I think that your approach is reductive. I personally have had situations where I don’t particularly care which of two identical machines performs a workload; one example is when using CD/DVD burners to produce several discs. A common consumer-focused example is having a dual-GPU machine where the two GPUs are configured as one single logical unit; the consumer doesn’t care which GPU handles which frame. Our operating systems must perform similar logic to load-balance processes in SMP configurations.
I think that you might want to consider the difficulty of being Buridan’s ass; this paradox constantly complicates my daily life.
When I am faced with a situation in which I don’t particularly care which of two identical machines performs a workload, such as your CD burner example, I pick whichever, or both. Flip a coin, and you get out of the buridan’s ass paradox, if you will. Surely the computer can’t do better than that, if it’s truly the buridan’s ass paradox and both choices are equally good. Dual-GPU systems and multicore CPUs are nice in that they don’t really require changing anything from the user’s perspective. Moving from the good old sysadmin way to kubernetes is very much not like that.
I’m sure there’s very valid use-cases for kubernetes, but not having to flip a coin to decide which of my two identical and equally in-reach computers will burn 20 CDs tonight is surely not worth the tradeoff.
To wring one last insight from this line of thought, it’s interesting to note that in the dual-GPU case, a CPU-bound driver chooses which GPU gets which drawing command, based on which GPU is closer to memory which is also driver-managed; while in the typical SMP CPU configuration, one of the CPUs is the zeroth CPU and has the responsibility of booting its siblings. Either way, there’s a delegation of the responsibility of the coin flip. It’s interesting that, despite being set up to manage the consequences of the coin flip, the actual random decision of how to break symmetry and choose a first worker is not part of the system.
And yet, at the same time, both GPU drivers and SMP kernels are relatively large. Even when they do not contain inner compilers and runtimes, they are inherently translating work requests from arbitrary and often untrusted processes into managed low-level actions, and in that translation, they often end up implementing the same approach that Kubernetes takes (and which I said upthread): Kubernetes manages objects which represent running processes. An SMP kernel manages UNIX-style process objects, but in order to support transparent migration between cores, it also has objects for physical memory banks, virtual memory pages, and IPC handles. A GPU driver manages renderbuffers, texturebuffers, and vertexbuffers; but in order to support transparent migration between CPU and GPU memory, it also has objects for GPU programs (shaders), for invoking GPU programs, for fencing GPU memory, and that’s not even getting into hotplugging!
My takeaway here is that there is a minimum level of complexity involved in writing a scheduler which can transparently migrate some of its actions, and that that complexity may well require millions of lines of code in today’s languages.
I mean, that’s not really an abstract thought-experiment, I do have two machines: my computer and my phone. I’d wager that nearly everyone here could say the same. In reality I have more like seven machines: a desktop, a laptop, a phone, two Raspberry Pi’s, a Switch, and a PS4. Each one of these is a computer far more powerful than the one that took the Apollo astronauts to the moon. The situation you’re talking about has quite literally never been a thing I’ve worried about. The only coordination problem I actually have between these machines is how I manage my ever-growing collection of pictures of my dog.
My feelings exactly. Kubernetes is for huge groups. Really huge. If you only have one hundred or so staff, I am not convinced you get much benefit.
If you’re happy in a large company, go wild. Enjoy the kubernets. It isn’t for me - I’m unsure whether I will ever join a group with more than ten or so again, but it won’t be soon.
I kind of see the point of kubernetes as a way for an organization to split developers and admins+devops more cleanly along department lines. This split also lets you outsource some of your devops to google cloud or whoever if you so choose.
I also think if you are a small org, this is not really that useful.
What’s the benefit of running Kubernetes vs. say… very well-behaved systemd services deployed with NixOps? You can do “declarative infrastructure” with both, but Docker-style whole OS containers are pretty bad for composability compared to Nix, even given tools like docker-compose. Still not as bad as trying to compose VMs, though.
I’m not a kubernetes expert but this is an apples and oranges comparison. Systemd doesn’t do any kind of multi system reasoning, no HA, load balancing, etc, for example.
I was specifically wondering about the multi-system deployment you can do with NixOps, not systemd. systemd services managed on multiple hosts with a NixOps declarative network.
It’s one tool among many similar (morph, krops, etc) now, and I was looking for a somewhat “experienced with both” opinion since I’ve mainly had experience with Nix and not kubes. I was sure there were lobsters who have tried both kubes and Nix-style deployment, but maybe I’m wrong.
Yeah, totally fair. Did you check out the manual? FWIW the documentation story with Nix has been gradually improving but lots of stuff is generally in a manual format right now…
Also, NixOps likes keeping local state in the form of a SQLite database. There are alternatives available that don’t require one, but I’ve only ever used NixOps (even though flakes make me more interested in the alternatives)
The “operating system” analogy has always fallen a bit flat for me, to be honest. When I’ve tried to use it, the analogy obscures more than it actually reveals.
Kubernetes provides a mechanism to take a big collection of machines and map some set of workloads onto it, without having to make individual decisions about which workloads go on which machines, or how they’ll communicate.
You can analogize it to an OS, by pointing out that an OS allows you to run programs without specifying individual CPUs and memory. But the analogy leaks a lot when you start talking about the details, such as the individual machines still running their own OS, and the networking and distributed systems details k8s exposes that a “traditional” OS doesn’t.
In my experience, debugging issues with an application on k8s also quickly devolves into regular Linux debugging with all the standard tools. It’s hard to argue that “Kubernetes is your OS now!” when you need to start thinking about iptables again as soon as things break.
At the end of the day, Kubernetes is just another cluster job scheduler. It works in the same space as tools like Nomad and Google Borg, and older tools like batch schedulers such as PBS and Slurm. It solves some problems the other tools don’t, like abstracting over cloud provider load balancer implementations, but adds some complexity to do so. It also solves some problems less well, such as backfill scheduling for batch jobs.
If you want to explain “why Kubernetes”, I’ve had the best luck talking about the operational problems it solves. If you have a hundred machines, it makes it easier to handle problems like what happens if a node fails? Or you need to do maintenance one one of them? How about if the process crashes, or you discover it needs to move because it needs more memory? And how do you make sure that when your service moves to a different machine, all its friends can still find it?
Cluster schedulers give you tools to solve those problems. Which are totally real problems, but also typically only show up when you have 12+ machines to manage. If you don’t have those problems, you probably don’t need a cluster. 🤷♂️ And that’s ok!
(My day job is basically helping customers build and operate compute clusters. But usually my first question is “are you sure you need to do this?” Because they typically introduce some complexity and pain while helping you abstract over other issues.)
The need to defend kubernetes on the face of increasing justified skepticism is quite telling.
I wonder which percentage of people have a setup that doesn’t boil down to nginx/hsproxy in front of replicas of the same http or grpc service. Feels like a whole lot of overengineering for what essentially is a reverse proxy with replicated services behind it.
I have used kubernetes in several companies and not a single one of them even resolve the end to more than one IP. Meaning they still have a single point of failure.
He seems to understand a waypoint on where cloud computing is going. I’m an AWS user, so I can only really speak in those terms. Those nodes will disappear as things migrate into Fargate. EKS will be replaced by ECS. You’ll have a central configuration file of all your resources written for Photon. AWS will roll out more product offerings that will reduce the types of containers you do make until you’re left with only the proprietary code that your business writes and sells as a service. I didn’t really understand what the re:Invent keynotes were about last year. I spent the year fiddling with k8s and containers and it all slowly became clear what AWS was really trying to become. I’ve been worried about resume polishing and maintaining control of things instead of having someone provide well thought out integrated solutions so that there’s more focus on development and adopting new technologies that make the product I sell better at a faster pace. I still love poking around at things at a lower level in my free time, but what I get paid to do is changing entirely and I’m going to have to adapt to that.
I broadly agree with this, although I will say that kube on Google cloud as GKE is already sufficiently managed that for the most part it just works as a place for declarative application deployment. The downside of kube is trying to operate the clusters yourself (for the most part).
If we’re analogizing cloud platforms to operating systems, then AWS is Windows in the 1990s – it’s by far the market leader, and comes with a suite of proprietary APIs and interfaces that people are willing to use because it’s the market leader.
Then the Kubernetes world is like Unix in the 1990s. There are a bunch of different vendors. None of them has enough market share to get people to use their own proprietary interfaces, so they team up and create common interfaces and try to basically standardize as much as possible while still competing with each other. Of course the standardization isn’t perfect and every vendor has its own wrinkles to learn.
If the analogy holds, eventually cloud services will be completely commoditized and essentially free like Linux is. But that is still a few years off.
I doubt very much that cloud services will become “essentially free like Linux is”, unless by that you mean the cost is in the time and expertise required on the part of those who will use the services (i.e., the old “Linux is free if your time costs nothing”).
Hardware and electricity and bandwidth have shown a tendency to trend cheaper over time, but so far nobody has demonstrated the kind of force-multiplier orchestration that would allow a cloud provider to run and offer a huge fleet without the overhead of at least a very large team of high-salary domain experts supporting and maintaining it. Similarly, nobody has demonstrated an alternative to “cloud provider” that allows buying or renting just the hardware/bandwidth/electricity and supplying the orchestration and expertise yourself without significant outlay for humans who understand and can run it. And that’s a very large hurdle to be cleared before the level of commodity pricing you seem to be implying would become possible.
That’s fair, electricity and hardware and real estate will never be free. But in the right circumstances, you could imagine the cost of cloud services shrinking to the cost of dedicated server hosting (like Hetzner) over time.
I find it fascinating that existing comments failed to actually examine the author’s point. To be pithy: Kubernetes manages objects which represent running processes. This is not much different from a typical high-level operating system, where processes are objects in a kernel. While we can certainly imagine better implementations of Kubernetes-style systems, they will generally have this same shape, not because Kubernetes is somehow excellently shaped for distributed systems, but because humans build operating systems in this shape, including distributed operating systems.
This is why it doesn’t matter whether the deployment is owned by an individual or a corporation; it doesn’t matter whether the deployment is made of containers, either. The key concept is that the process objects might not reside wholly on a single machine; the state itself is distributed. Distributed computation requires distributed tooling and scaffolding.
Yeah, what I find interesting about k8s is you could do basically configuration management over some kind of RPC like thing - sounds a lot like late 90s Windows administration than /etc hell.
It’s also remarked that OOP programming makes sense in a distributed world - guess what Erlang and k8s are in?
The fact that k8s requires scale to be usable is unfortunate.
ehh, who’s the you here? This is starting from the assumption that I have a lot of nodes, which is only true in the context of me running infrastructure for a corporation; the you here is a corporation.
again, again, define traditional. Who’s tradition? In what context? In a traditional OS, software starts when you start using it, and then it stops when you stop using it. The idea that everything should be an always-running, fully-managed service is something that’s only traditional in the context of SAAS.
The thing that makes me feel cold about all this stuff is that we’re getting further and further away from building software that is designed for normal people to run on their own machines. So many people that run Kubernetes argue that it doesn’t even make sense unless you have people whose job it is to run Kubernetes itself. So it’s taken for granted that people are writing software that they can’t even run themselves. I dunno. All this stuff doesn’t make me excited, it makes me feel like a pawn.
You’re right, you probably wouldn’t use Kubernetes as an individual.
I’ll take the bait a little bit though and point out that groups of people are not always corporations. For example, we run Kubernetes at the Open Computing Facility at our university. Humans need each other, and depending on other people doesn’t make you a pawn.
Given the millions and millions spent on marketing, growth hacking, and advertising for the k8s ecosystem, I van say with some certainty we are all pawn-shaped.
totally fair criticism. I think “corporation” in my comment could readily be substituted with “enterprise”, “institution”, “organization” or “collective”. “organization” is probably the most neutral term.
so I think this is where my interpretation is less charitable, and we could even look at my original comment as being vague and not explicitly acknowledging its own specific frame of reference:
again, who’s tradition, and in what context? Here I’m speaking of my tradition as a personal computer user, and the context is at home, for personal use. When thinking about Kubernetes (or container orchestration generally) there’s another context of historical importance: time-sharing. Now, I don’t have qualms with time-sharing, because time-sharing was a necessity at the time. The time-sharing computing environments of the sixties and seventies existed because the ownership of a home computer was unreasonably expensive: time-sharing existed to grant wider access to computing. Neat!
Circling back to your comment about dependency not inherently making someone a pawn and ask: who is dependent on whom, for what, and why? We might say of time-sharing at a university: a student is dependent on the university for access to computing because computers are too big and expensive for the student to own. Makes sense! The dependent relationship is, in a sense, axiomatic of the technology, and may even describe your usage of Kubernetes. If anything, the university wishes the student wasn’t dependent on them for this because it’s a burden to run.
But generally, Kubernetes is a different beast, and the reason there’s so much discussion of Kubernetes here and elsewhere in the tech industry is that Kubernetes is lucrative. Sure, it’s neat and interesting technology, but so is graphics or embedded programming or cybernetics, etc, etc, etc. There are lots of neat and interesting topics in programming that are very rarely discussed here and elsewhere in programming communities.
Although computers are getting faster, cheaper, and smaller, the computers owned by the majority of people are performing less and less local computation. Although advances in hardware should be making individuals more independent, the SAAS landscape that begat Kubernetes has only made people less independent. Instead of running computation locally, corporations want to run the computation for you and charge you some form of rent. This landscape of rentier computation that is dominating our industry has created dependent relationships that are not inherently necessary, but are instead instruments of profit-seeking and control. This industry-wide turn towards rentier computation is the root of my angst, and I would say is actually the point of Kubernetes.
This resonates with me a lot. At work, we have some projects that are very easy to run locally and we have some that are much harder. Nearly all the projects that can be run locally get their features implemented more quickly and more reliably. Being able to run locally cuts way down on the feedback loop.
I’m really looking forward to the built-in embed stuff in Go 1.16 for this reason. Yeah, there’s third-party tools that do it, but having it standardized will be great. I write Go servers and one thing I’ve done is implement every storage layer twice: once with a database, and once in process memory. The utility of this has been incredible, because I can compile a server into a single .exe file that I can literally PM to a colleague on Slack that they can just run and they have a working dev server with no setup at all. You can also do this with sqlite or other embedded databases if you need local persistence; I’ve done that in the past but I don’t do it in my current gig.
In my experience the overhead of implementing the logic twice does not pay out since it is very easy to spin up a MySQL or Postgres database, e.g. using docker. Of course this comes with the disadvantage of having to provide another dependency but at least the service then runs in a similar environment to production. Usually spinning up a test database is already documented/automated for testing.
That was my first thought, but upon reflection - the test implementation is really just an array of structs, and adds very little overhead at all.
yeah, very often the implementation is just a mess of
map[string]*Book, where there’s oneBookfor every model type and one map for every index, and then you slap a mutex around the whole thing and call it a day. It falls apart when the data is highly relational. I use the in-mem implementation for unit tests and for making debug binaries. I send debug binaries to non-developer staff. Asking them to install Docker alone would be a non-starter.Suppose that you, an individual, own two machines. You would like a process to execute on either of those machines, but you don’t want to have to manage the detail of which machine is actually performing the computation. At this point, you will need to build something very roughly Kubernetes-shaped.
The difficulty isn’t in having a “lot” of nodes, or in running things “on their own machines”; the difficulty is purely in having more than one machine.
…is not a problem which people with two machines have. They pick one, the other, or both, and skip on all the complexity a system that chooses for you would entail.
I struggle with this. My reflex is to want to have that dynamic host management, but fact of the matter is my home boxes have had less pages than my ISP in the past five years. Plain old sysadmin is more than enough in all of my use cases. Docker is still kinda useful to not have to deal with the environment and setup and versions, but like. A lot of difficulty is sidestepped by just avoiding to buy into the complexity.
I wonder if this also holds for “smaller” professional projects.
Unfortunately, I think that your approach is reductive. I personally have had situations where I don’t particularly care which of two identical machines performs a workload; one example is when using CD/DVD burners to produce several discs. A common consumer-focused example is having a dual-GPU machine where the two GPUs are configured as one single logical unit; the consumer doesn’t care which GPU handles which frame. Our operating systems must perform similar logic to load-balance processes in SMP configurations.
I think that you might want to consider the difficulty of being Buridan’s ass; this paradox constantly complicates my daily life.
When I am faced with a situation in which I don’t particularly care which of two identical machines performs a workload, such as your CD burner example, I pick whichever, or both. Flip a coin, and you get out of the buridan’s ass paradox, if you will. Surely the computer can’t do better than that, if it’s truly the buridan’s ass paradox and both choices are equally good. Dual-GPU systems and multicore CPUs are nice in that they don’t really require changing anything from the user’s perspective. Moving from the good old sysadmin way to kubernetes is very much not like that.
I’m sure there’s very valid use-cases for kubernetes, but not having to flip a coin to decide which of my two identical and equally in-reach computers will burn 20 CDs tonight is surely not worth the tradeoff.
To wring one last insight from this line of thought, it’s interesting to note that in the dual-GPU case, a CPU-bound driver chooses which GPU gets which drawing command, based on which GPU is closer to memory which is also driver-managed; while in the typical SMP CPU configuration, one of the CPUs is the zeroth CPU and has the responsibility of booting its siblings. Either way, there’s a delegation of the responsibility of the coin flip. It’s interesting that, despite being set up to manage the consequences of the coin flip, the actual random decision of how to break symmetry and choose a first worker is not part of the system.
And yet, at the same time, both GPU drivers and SMP kernels are relatively large. Even when they do not contain inner compilers and runtimes, they are inherently translating work requests from arbitrary and often untrusted processes into managed low-level actions, and in that translation, they often end up implementing the same approach that Kubernetes takes (and which I said upthread): Kubernetes manages objects which represent running processes. An SMP kernel manages UNIX-style process objects, but in order to support transparent migration between cores, it also has objects for physical memory banks, virtual memory pages, and IPC handles. A GPU driver manages renderbuffers, texturebuffers, and vertexbuffers; but in order to support transparent migration between CPU and GPU memory, it also has objects for GPU programs (shaders), for invoking GPU programs, for fencing GPU memory, and that’s not even getting into hotplugging!
My takeaway here is that there is a minimum level of complexity involved in writing a scheduler which can transparently migrate some of its actions, and that that complexity may well require millions of lines of code in today’s languages.
I mean, that’s not really an abstract thought-experiment, I do have two machines: my computer and my phone. I’d wager that nearly everyone here could say the same. In reality I have more like seven machines: a desktop, a laptop, a phone, two Raspberry Pi’s, a Switch, and a PS4. Each one of these is a computer far more powerful than the one that took the Apollo astronauts to the moon. The situation you’re talking about has quite literally never been a thing I’ve worried about. The only coordination problem I actually have between these machines is how I manage my ever-growing collection of pictures of my dog.
My feelings exactly. Kubernetes is for huge groups. Really huge. If you only have one hundred or so staff, I am not convinced you get much benefit.
If you’re happy in a large company, go wild. Enjoy the kubernets. It isn’t for me - I’m unsure whether I will ever join a group with more than ten or so again, but it won’t be soon.
I kind of see the point of kubernetes as a way for an organization to split developers and admins+devops more cleanly along department lines. This split also lets you outsource some of your devops to google cloud or whoever if you so choose.
I also think if you are a small org, this is not really that useful.
Nomad is also a cleaner way to do this, especially when fewer humans are involved.
[Comment removed by author]
What’s the benefit of running Kubernetes vs. say… very well-behaved systemd services deployed with NixOps? You can do “declarative infrastructure” with both, but Docker-style whole OS containers are pretty bad for composability compared to Nix, even given tools like docker-compose. Still not as bad as trying to compose VMs, though.
I’m not a kubernetes expert but this is an apples and oranges comparison. Systemd doesn’t do any kind of multi system reasoning, no HA, load balancing, etc, for example.
I was specifically wondering about the multi-system deployment you can do with NixOps, not systemd. systemd services managed on multiple hosts with a NixOps declarative network.
The NixOps page on the Nix wiki is “sparse”.
It’s one tool among many similar (morph, krops, etc) now, and I was looking for a somewhat “experienced with both” opinion since I’ve mainly had experience with Nix and not kubes. I was sure there were lobsters who have tried both kubes and Nix-style deployment, but maybe I’m wrong.
Oh, mostly I was really interested in learning about it and a blank page made me sad.
Yeah, totally fair. Did you check out the manual? FWIW the documentation story with Nix has been gradually improving but lots of stuff is generally in a manual format right now…
Also, NixOps likes keeping local state in the form of a SQLite database. There are alternatives available that don’t require one, but I’ve only ever used NixOps (even though flakes make me more interested in the alternatives)
(Disclaimer, this got longer than I meant it to…)
The “operating system” analogy has always fallen a bit flat for me, to be honest. When I’ve tried to use it, the analogy obscures more than it actually reveals.
Kubernetes provides a mechanism to take a big collection of machines and map some set of workloads onto it, without having to make individual decisions about which workloads go on which machines, or how they’ll communicate.
You can analogize it to an OS, by pointing out that an OS allows you to run programs without specifying individual CPUs and memory. But the analogy leaks a lot when you start talking about the details, such as the individual machines still running their own OS, and the networking and distributed systems details k8s exposes that a “traditional” OS doesn’t.
In my experience, debugging issues with an application on k8s also quickly devolves into regular Linux debugging with all the standard tools. It’s hard to argue that “Kubernetes is your OS now!” when you need to start thinking about iptables again as soon as things break.
At the end of the day, Kubernetes is just another cluster job scheduler. It works in the same space as tools like Nomad and Google Borg, and older tools like batch schedulers such as PBS and Slurm. It solves some problems the other tools don’t, like abstracting over cloud provider load balancer implementations, but adds some complexity to do so. It also solves some problems less well, such as backfill scheduling for batch jobs.
If you want to explain “why Kubernetes”, I’ve had the best luck talking about the operational problems it solves. If you have a hundred machines, it makes it easier to handle problems like what happens if a node fails? Or you need to do maintenance one one of them? How about if the process crashes, or you discover it needs to move because it needs more memory? And how do you make sure that when your service moves to a different machine, all its friends can still find it?
Cluster schedulers give you tools to solve those problems. Which are totally real problems, but also typically only show up when you have 12+ machines to manage. If you don’t have those problems, you probably don’t need a cluster. 🤷♂️ And that’s ok!
(My day job is basically helping customers build and operate compute clusters. But usually my first question is “are you sure you need to do this?” Because they typically introduce some complexity and pain while helping you abstract over other issues.)
The need to defend kubernetes on the face of increasing justified skepticism is quite telling.
I wonder which percentage of people have a setup that doesn’t boil down to nginx/hsproxy in front of replicas of the same http or grpc service. Feels like a whole lot of overengineering for what essentially is a reverse proxy with replicated services behind it.
I have used kubernetes in several companies and not a single one of them even resolve the end to more than one IP. Meaning they still have a single point of failure.
I keep rewriting this response.
He seems to understand a waypoint on where cloud computing is going. I’m an AWS user, so I can only really speak in those terms. Those nodes will disappear as things migrate into Fargate. EKS will be replaced by ECS. You’ll have a central configuration file of all your resources written for Photon. AWS will roll out more product offerings that will reduce the types of containers you do make until you’re left with only the proprietary code that your business writes and sells as a service. I didn’t really understand what the re:Invent keynotes were about last year. I spent the year fiddling with k8s and containers and it all slowly became clear what AWS was really trying to become. I’ve been worried about resume polishing and maintaining control of things instead of having someone provide well thought out integrated solutions so that there’s more focus on development and adopting new technologies that make the product I sell better at a faster pace. I still love poking around at things at a lower level in my free time, but what I get paid to do is changing entirely and I’m going to have to adapt to that.
I broadly agree with this, although I will say that kube on Google cloud as GKE is already sufficiently managed that for the most part it just works as a place for declarative application deployment. The downside of kube is trying to operate the clusters yourself (for the most part).
[Comment removed by author]
If we’re analogizing cloud platforms to operating systems, then AWS is Windows in the 1990s – it’s by far the market leader, and comes with a suite of proprietary APIs and interfaces that people are willing to use because it’s the market leader.
Then the Kubernetes world is like Unix in the 1990s. There are a bunch of different vendors. None of them has enough market share to get people to use their own proprietary interfaces, so they team up and create common interfaces and try to basically standardize as much as possible while still competing with each other. Of course the standardization isn’t perfect and every vendor has its own wrinkles to learn.
If the analogy holds, eventually cloud services will be completely commoditized and essentially free like Linux is. But that is still a few years off.
I doubt very much that cloud services will become “essentially free like Linux is”, unless by that you mean the cost is in the time and expertise required on the part of those who will use the services (i.e., the old “Linux is free if your time costs nothing”).
Hardware and electricity and bandwidth have shown a tendency to trend cheaper over time, but so far nobody has demonstrated the kind of force-multiplier orchestration that would allow a cloud provider to run and offer a huge fleet without the overhead of at least a very large team of high-salary domain experts supporting and maintaining it. Similarly, nobody has demonstrated an alternative to “cloud provider” that allows buying or renting just the hardware/bandwidth/electricity and supplying the orchestration and expertise yourself without significant outlay for humans who understand and can run it. And that’s a very large hurdle to be cleared before the level of commodity pricing you seem to be implying would become possible.
That’s fair, electricity and hardware and real estate will never be free. But in the right circumstances, you could imagine the cost of cloud services shrinking to the cost of dedicated server hosting (like Hetzner) over time.