I always thought that Greenspun’s tenth rule (Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp) ought to be updated for modern development practices, so here’s Hagelberg’s tenth rule: Any sufficiently complicated kubernetes deployment contains an ad-hoc informally-specified, bug-ridden, slow implementation of half of OTP.
Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.
The root of all of this is that the BEAM ecosystem represents a parallel evolutionary path (in many ways better) to what ended up becoming the dominant tooling in our industry.
For a lot of us, this means that when we need to use that neat tooling we end up having to fight with institutional inertia and dogma. I have a coworker–smart dude otherwise–that in their role as a devops just refuses to exploit the idea that our BEAM stuff doesn’t quite act like the generic python or node or whatever app you would use a “livestock not pets” approach on.
It’s annoying, but we deal with the world as it is.
I have a coworker–smart dude otherwise–that in their role as a devops just refuses to exploit the idea that our BEAM stuff doesn’t quite act like the generic python or node or whatever app you would use a “livestock not pets” approach on.
Can you elaborate on this a little bit more - I am not certain what this means but it sounds good.
I always disliked how the kubernetes model conflicts with the live code deploy features of erlang. I’m sure this post will make some kubernetes purists cry though. :)
Could you elaborate? I don’t see how this breaks Kubernetes’ invariants. The idea of putting code into a ConfigMap is not new; there is a common sidecar, configmap-reload, which abstracts this pattern.
Live code deploy in Erlang happens without losing the process’s memory state. So you can insert a new version of a function or module into the ongoing flow of a running program with stopping the VM.
k8s works with the Unix process model which does not support such things without very strange, hard to maintain magic.
Plenty of cloud-native executables support dynamically loading configuration or code without restarting the process. For example, a common use for the sidecar I linked is to reload Prometheus configuration without restarting. There is no magic here.
Please examine your meme. I’m going to reply instead of flagging you “wrong”, because I think that you should think about the words that you use. I also want less tribalism; this is not a situation where Erlang and Kubernetes should be thought of as disjoint conflicting ecosystems.
To start at the end, Kubernetes isn’t BEAM. Kubernetes doesn’t care whether a process is an interpreter or whether a process is dynamically compiling code; it certainly doesn’t care about the structure of modules within Erlang, Elixir, or any other language within a container. When it comes to container images, Kubernetes supports hot changes to the filesystem, and the sidecar I linked earlier is designed to detect those changes and notify the main process.
Is this ad-hoc? No, configmap-reload is a common tool. Is it informally-specified? Sure, you could complain that its documentation is lacking, but the two notification systems could be formalized relative to POSIX and HTTP respectively. Is it bug-ridden? It’s written in Go, but being written in a language with a high defect rate does not guarantee the presence of bugs. Is it slow? It’s fast enough; I’m not sure what would be slow about it.
You appear to be arguing that because some programs can be written in a way that they can manage their own code loading if properly notified and those can be run in Kubernetes, that this is equivalent to the kind of hot swapping that OTP on the BEAM provides.
The main things that Kubernetes provides that OTP does not are placement of replicas of pods on clusters and running arbitrary Linux containers.
No, you’ve moved the goalposts. I’m not claiming that Kubernetes and BEAM are equivalent, as in another thread. I’m claiming that Kubernetes does not prevent container processes from hot code loading.
If you want to think of Kubernetes as providing features that OTP doesn’t provide, then think of Kubernetes as a construction kit with many optional parts; many of the fancier features are optional too. In the original article, the author shows how to integrate Erlang with Kubernetes in a way that doesn’t use most of the optional parts. This isn’t a conflict and it doesn’t require magic, just a sidecar.
The code in the configmap is just a shell script, it’s not the code itself. That shell script pulls down the latest version of the code and deploys it to the Erlang VM. There’s nothing reproducible about it and it is effectively sidestepping the k8s deployment mechanisms.
Ah, I see; if Kubernetes required container images to be reproducibly tagged, not just content-addressed and immutable, then this would be a dire conflict indeed. I’m not sure how we would combine Erlang and Kubernetes in that case. However, container images don’t need to be reproducible; tags like “:latest” are allowed to change which image they are referring to, and services like Nixery tag individual packages by name instead of reproducible hash.
I agree that Dandelion is an alternative to common Kubernetes tools; in particular, it avoids Deployments. This would be an example of what I mean when I say that Kubernetes is a kit with many optional parts. But some portions of the system could still benefit from Deployments, like any changes to the underlying C code; it’s not a binary choice about which objects to use, but a space of possible arrangements of objects.
Thank you for explaining your reasoning. I’m glad that we could find the root of the misconceptions and synthesize a better approach for all Kubernetes users, regardless of whether they are using BEAM.
That’s one way to look at it, and maybe that’s how it appears for Mnesia users; I’m not sure how Mnesia on Kubernetes would work. For applications with SQL databases or other structured persistence, the database’s state is usually stored using a PersistentVolume.
You’re right about the limitations of ConfigMaps. The amount of code that could be changed is enough for a feature flag flip, but maybe not enough for a complete redeploy of a large monolith. Perhaps this is the “conflict” that neighboring comments are talking about. I don’t see this as a conflict; Kubernetes supports mounting multiple types of objects into Pods, including ConfigMaps, PersistentVolumes, and Secrets, but none of them are mandatory.
what if instead of having configuration management systems, you could hard-code your config in constants that just get rolled out live in less than a minute—
The programmer that wants to get things done quickly loves the idea. The SRE in me screams at the thought of pushing configuration so quickly. Ditto for code. Very easy to take down a system entirely, or worse broadly corrupt persistent state.
This is why Erlang has a notion of incremental updates, where the new and old versions can coexist and you switch over only after determining that the new version is working correctly. Remember, Erlang came from a world where whole seconds of downtime in a year were unacceptable.
If your entire codebase is erlang and all your databases are also erlang you don’t need kubernetes. Kubernetes is the lower common denominator when that is not the case. Most of the time you need to coordinate more than just a set of erlang processes. Kubernetes solves that problem using similar concepts as the erlang runtime does but in a completely language agnostic way which erlang can not do.
I always thought that Greenspun’s tenth rule (Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp) ought to be updated for modern development practices, so here’s Hagelberg’s tenth rule: Any sufficiently complicated kubernetes deployment contains an ad-hoc informally-specified, bug-ridden, slow implementation of half of OTP.
http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-programming.html
I should have known! But a lot has happened since 2008, and I feel like k8s deserves to be called out specifically.
The root of all of this is that the BEAM ecosystem represents a parallel evolutionary path (in many ways better) to what ended up becoming the dominant tooling in our industry.
For a lot of us, this means that when we need to use that neat tooling we end up having to fight with institutional inertia and dogma. I have a coworker–smart dude otherwise–that in their role as a devops just refuses to exploit the idea that our BEAM stuff doesn’t quite act like the generic python or node or whatever app you would use a “livestock not pets” approach on.
It’s annoying, but we deal with the world as it is.
Can you elaborate on this a little bit more - I am not certain what this means but it sounds good.
I live in that rule at work, and quite agree with it.
Where are Hagelberg’s rules 1-9?
(Yes, I’m aware)
I always disliked how the kubernetes model conflicts with the live code deploy features of erlang. I’m sure this post will make some kubernetes purists cry though. :)
Turn the code reloader into an Operator that stores state in a CustomResource and now it’s “cloud native” haha
This probably the idiomatic solution. :)
Could you elaborate? I don’t see how this breaks Kubernetes’ invariants. The idea of putting code into a ConfigMap is not new; there is a common sidecar, configmap-reload, which abstracts this pattern.
Live code deploy in Erlang happens without losing the process’s memory state. So you can insert a new version of a function or module into the ongoing flow of a running program with stopping the VM.
k8s works with the Unix process model which does not support such things without very strange, hard to maintain magic.
Plenty of cloud-native executables support dynamically loading configuration or code without restarting the process. For example, a common use for the sidecar I linked is to reload Prometheus configuration without restarting. There is no magic here.
So they have a ad-hoc informally-specified, bug-ridden, slow implementation hot code loading ;-)
Please examine your meme. I’m going to reply instead of flagging you “wrong”, because I think that you should think about the words that you use. I also want less tribalism; this is not a situation where Erlang and Kubernetes should be thought of as disjoint conflicting ecosystems.
To start at the end, Kubernetes isn’t BEAM. Kubernetes doesn’t care whether a process is an interpreter or whether a process is dynamically compiling code; it certainly doesn’t care about the structure of modules within Erlang, Elixir, or any other language within a container. When it comes to container images, Kubernetes supports hot changes to the filesystem, and the sidecar I linked earlier is designed to detect those changes and notify the main process.
Is this ad-hoc? No,
configmap-reloadis a common tool. Is it informally-specified? Sure, you could complain that its documentation is lacking, but the two notification systems could be formalized relative to POSIX and HTTP respectively. Is it bug-ridden? It’s written in Go, but being written in a language with a high defect rate does not guarantee the presence of bugs. Is it slow? It’s fast enough; I’m not sure what would be slow about it.You appear to be arguing that because some programs can be written in a way that they can manage their own code loading if properly notified and those can be run in Kubernetes, that this is equivalent to the kind of hot swapping that OTP on the BEAM provides.
The main things that Kubernetes provides that OTP does not are placement of replicas of pods on clusters and running arbitrary Linux containers.
No, you’ve moved the goalposts. I’m not claiming that Kubernetes and BEAM are equivalent, as in another thread. I’m claiming that Kubernetes does not prevent container processes from hot code loading.
If you want to think of Kubernetes as providing features that OTP doesn’t provide, then think of Kubernetes as a construction kit with many optional parts; many of the fancier features are optional too. In the original article, the author shows how to integrate Erlang with Kubernetes in a way that doesn’t use most of the optional parts. This isn’t a conflict and it doesn’t require magic, just a sidecar.
Looking back, you’re right. I did. I apologize.
The code in the configmap is just a shell script, it’s not the code itself. That shell script pulls down the latest version of the code and deploys it to the Erlang VM. There’s nothing reproducible about it and it is effectively sidestepping the k8s deployment mechanisms.
Ah, I see; if Kubernetes required container images to be reproducibly tagged, not just content-addressed and immutable, then this would be a dire conflict indeed. I’m not sure how we would combine Erlang and Kubernetes in that case. However, container images don’t need to be reproducible; tags like “:latest” are allowed to change which image they are referring to, and services like Nixery tag individual packages by name instead of reproducible hash.
I agree that Dandelion is an alternative to common Kubernetes tools; in particular, it avoids Deployments. This would be an example of what I mean when I say that Kubernetes is a kit with many optional parts. But some portions of the system could still benefit from Deployments, like any changes to the underlying C code; it’s not a binary choice about which objects to use, but a space of possible arrangements of objects.
Thank you for explaining your reasoning. I’m glad that we could find the root of the misconceptions and synthesize a better approach for all Kubernetes users, regardless of whether they are using BEAM.
Doesn’t this require that all of your code be put into a ConfigMap, and not handle persistence of other state?
That’s one way to look at it, and maybe that’s how it appears for Mnesia users; I’m not sure how Mnesia on Kubernetes would work. For applications with SQL databases or other structured persistence, the database’s state is usually stored using a PersistentVolume.
You’re right about the limitations of ConfigMaps. The amount of code that could be changed is enough for a feature flag flip, but maybe not enough for a complete redeploy of a large monolith. Perhaps this is the “conflict” that neighboring comments are talking about. I don’t see this as a conflict; Kubernetes supports mounting multiple types of objects into Pods, including ConfigMaps, PersistentVolumes, and Secrets, but none of them are mandatory.
The programmer that wants to get things done quickly loves the idea. The SRE in me screams at the thought of pushing configuration so quickly. Ditto for code. Very easy to take down a system entirely, or worse broadly corrupt persistent state.
This is why Erlang has a notion of incremental updates, where the new and old versions can coexist and you switch over only after determining that the new version is working correctly. Remember, Erlang came from a world where whole seconds of downtime in a year were unacceptable.
If your entire codebase is erlang and all your databases are also erlang you don’t need kubernetes. Kubernetes is the lower common denominator when that is not the case. Most of the time you need to coordinate more than just a set of erlang processes. Kubernetes solves that problem using similar concepts as the erlang runtime does but in a completely language agnostic way which erlang can not do.