But, why? Why would I want 50 containers on the same machine?
The most efficient way to do things is to have massive machines with many many jobs on them. This is as it reduces resource overhead (think of all the management daemons that run per machine), makes scheduling easier and allows for over-provisioning.
This makes great sense if you’re the one renting out those massive machines–but again, I’m curious about the developers and products themselves.
Would we make these same decisions if there wasn’t a billion-dollar industry predicated on developer outreach and education/advertising/brainwashing pushing this as a solution?
I’m seeing technology like the Erlang runtime and wondering if maybe people should just write on less bloated platforms. Maybe that could solve the efficiency problem–50 Ruby VMs is hardly more efficient than a 50 C programs, for example.
For us, 50 containers on the same machine are 50 different builds running 50 different test suites. We have a cluster that runs tests in parallel, in containers, across many machines. I can’t imagine getting anything done if I had to run all those tests locally.
People also use sorts of Linux or macOS (with all sorts of Linux VMs). Now the build team doesn’t have to support all the different setups. They also don’t have to do flag days as much, they just update the build container and move on.
I run a couple apps in containers on a DO box, each of which shells out to a few other programs written in a variety of languages (ruby, c, node, java). None of these have to care what OS they’re running on or if the packages for those programs are available, or where those binaries are going to be installed to. I just ship the container and I’m good.
Shipping each app with it’s own isolated environment seems beautiful to me, and I don’t see why you wouldn’t want such an abstraction. It’d be great if all the programs I depended on were all written as libraries for the language I wrote my webapp in, but they weren’t.
The current popular implementation of containers is strictly less useful than packages as they are OS specific. As someone who does not run Linux, using containers is a pain for me so I have to run a Linux VM to run the container. Whereas the source artifacts I could build for my OS. So containers have taken OS-independent languages like Python, Ruby, and Java and made OS-dependent artifacts. Personally, I think our lives would be much much simpler if we standardized on a package manager like Nix rather than standardized on a binary artifact such as a container.
I’ve come around to the idea of containers, now that Kubernetes exists.
The idea is that you ought to be able to submit a program, bounded with memory and disk and bandwidth expectations, and have that program encapsulated in such a way that it is trivial to operate (and ideally has no or few state expectations). Then, scaling and operation become super easy.
It’s not for every use case, but it’s a pretty nice sweet spot with acceptable tradeoffs if you have lots of independent services, large scale, generally stateless operation (or have magic, like Google does), many different development teams and/or styles, an unknown scaling requirement on almost every service, enough professionalism to write and keep API contracts, and an ops team that really doesn’t want to deal with your nonsense and wants a rollback button (and/or wants you to deploy and undeploy your own code without blowing up the world).
The big problem is state; Google’s datacenters are literally magic, so they’ll be fine. Everyone else, currently, that’s a problem.
Those all sound like problems that should be being solved at the application layer, and perhaps with better support at the OS layer.
My current impression–and please do correct me if I’m wrong!–is that containers are nowadays basically tacit acknowledgements that, because people lack the discipline/skill/tooling to write fairly self-contained blobs, we need to provide an artificial boundary sack to hold all their garbage in to reduce the ops headaches they would otherwise be creating.
It would seem to me that the sustainable, long-term solution would be to have developers that understand how not to pull in a clowncar of dependencies (cough node ruby cough) and how to use the tools that their systems provide them out of the box for managing quotas/networking/filesystems and so forth. Of course, that’s expensive, and there isn’t money to be made by just telling devs to RTFM.
because people lack the discipline/skill/tooling to write fairly self-contained blobs
How do I ensure they are actually correctly self-contained? “Be more disciplined” sounds suspiciously like “don’t write bugs”. Containers are to multi-tenant deployment as immutable data structures are to multi-threaded programming.
It would seem to me that the sustainable, long-term solution would be to have developers that understand how not to pull in a clowncar of dependencies (cough node ruby cough)
This is irrelevant. Both node and ruby support full namespace isolation of dependencies, without containers. I could install every version of every rubygem on my laptop and my only problem would be lack of disk space.
how to use the tools that their systems provide them out of the box for managing quotas/networking/filesystems
The advantage of overlayfs was already discussed in Julia Evans' post. As for quotas, containers aren’t really about quotas as much as homogenous deployment tooling and safe multi-tenancy. Quotas are a stability afterthought—so much of an afterthought that they aren’t even that well implemented in public container projects.
As someone who does not run Linux, the modern container movement is a regression for me. With source artifacts, I could build packages for my OS (FreeBSD) and start jails to deploy them on for development. With modern containers, I need to run a Linux VM to run an opaque binary artifact just to do development. This makes my environment more complex rather than simpler. Oh well, I’m about a year or two from just being a curmudgeon yelling at people to get off my OS lawn :)
Yes, they are containers (heck, the first containers). However my point was specifically about how containers are working out right now, which are Linux-specific binary artifacts. At best, one has to emulate a Linux system to run a a docker image.
I believe we’ll have a standard image format soon enough, but you’ll still need to put in the effort to build containers targeting the BSDs, since they presumably won’t run linux binaries. Well, I guess freebsd has emulation, but I’m not sure what the state of it is. Dragonfly and OpenBSD don’t, though I don’t think OpenBSD has jails to begin with.
Jails also unfortunately lack the granularity of namespaces but I don’t know how huge of a deal that is.
Oh let’s do be totally clear: the container ecosystem today is an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang and a tenth of Unix. That said, the number of fellow Erlang programmers I’ve ever physically met could fill a car, so pragmatically there’s something to be said for providing the good guarantees and affordances of Erlang in a form that even a node.js programmer can probably not fuck up.
It would seem to me that the sustainable, long-term solution would be to have developers that understand how not to pull in a clowncar of dependencies (cough node ruby cough)
There are a few problems with this part of your statement:
The cost of switching languages at an established tech company is measured in millions. It would take any company years to retool, retrain or rehire talent, all at the expense of not developing new features to pull something like that off. Also, good luck convincing anyone in management that that’s worthwhile.
Using an abstraction to conveniently package applications for multi-tenancy and to alleviate dependency hell solves actual problems. If you work at a company with thousands of apps, dependency conflicts are (almost - monorepos help solve this) inevitable. If you’re relying on OS packages, multi-tenancy is much harder.
This argument sounds very anti-library. While the dependency sprawl of node is inarguable, going the opposite direction and re-implementing and embedding the same set of functions in every project in order to be self-contained is absurd, and hard to maintain. Again, at the scale of a large company, it makes much more sense to dump common functions into libraries, or make use of the tens of thousands FOSS libraries out there.
and how to use the tools that their systems provide them out of the box for managing quotas/networking/filesystems and so forth
On Linux, containers are partially more or less an abstraction on top of the “modern” way of enforcing quotas
Outside of work, I’m not really a Linux user. However, what I do love about the modern Linux container movement is the idea of producing a self-contained image that’s easy to distribute, run and apply run-time constraints on. I’d love to see this applied to FreeBSD Jails. I’m not aware of any attempt at an implementation of this, other than the apparently abandoned jetpack.
It’s stunningly expensive and time-consuming (decades!) to train people up to the level where they can build simple, reliable software with few dependencies. The training process produces a lot of complex, unreliable, useful working software. Is it sustainable to not use it?
I think you’re correct in pointing out “sustainable” as something in need of better specification.
For me, the current situation is “unsustainable” because:
Small programs (from a user perspective) are requiring more and more infrastructure to support even when it their function is fairly simple and limited.
Ever-growing infrastructure is much like ever-growing use of abstractions…you can solve any problem with more tooling/abstraction, except for the problem of too much tooling/abstraction.
This increased emphasis on “screw it throw it in a container” makes security patching even more of a nightmare than it would be otherwise.
This emphasis on multitenancy is great at transferring wealth from startups to service providers while failing to cultivate useful standalone technical expertise. This only works as long as startups have the funding to slosh around on service providers–which is probably not going to go on forever.
There are absolutely people (in this thread for example) that use Docker et al for things like multi-tenant builds, which is a use-case for where it makes sense.
But the unsustainable thing is when people avoid fixing the shittiness of their tooling (Kafka, ML stuff, database deployments, etc.) by shoving it in a container and pretending they don’t have to learn how it works or how to manage it. And there seems, to me, to be a lot of that in the wind these days.
This increased emphasis on “screw it throw it in a container” makes security patching even more of a nightmare than it would be otherwise
I have found it surprising that some of the same people who are so opposed to static compilation (with cries about how you would have to download updates for many programs instead of just openssl), somehow are just fine with containers.
“The idea is that you ought to be able to submit a program, bounded with memory and disk and bandwidth expectations, and have that program encapsulated in such a way that it is trivial to operate (and ideally has no or few state expectations). ”
That’s a great description of processes running on QNX microkernel circa 1990’s. They also had a desktop on a floppy + self-healing to a degree. Something tells me there’s some middle ground between that and how container platforms are doing same thing with many MB of software. Later on, researching along lines of separation kernels, TU Dresden’s TUDOS demonstrator let me run a new user-mode VM a second for Linux on top of L4 microkernel on crappy hardware. Fire them up as fast as I can click them. That’s just an academic prototype. Real-world stuff from RTOS vendors let you do Linux, Ada runtimes, Java subsets, C apps, and so on with same, efficient microkernel with autogenerate communication stubs for making them work together. Ada was safest but C and Java parts can support all the mainstream stuff. Also, vendors like Aonix supported lightweight VM’s with AOT compilation into self-contained ROM’s you can link in.
I’m with angersock in at least how the modern stuff looks like giant piles of crap on top of piles of crap compared to even 90’s version of better architectures. Let’s not even talk about security where modern stuff’s security for multi-tenancy is like a joke compared to GEMSOS or KeyKOS from the 1980’s or Karger’s security kernel for VAX VMM from 90’s. I’m disagreeing with angersock in that it can be a wise solution to chose one of these piles if you’re working with many other piles in a way that needs isolation and management that can’t be easily done with better architecture. If you have to.
I was thinking about this as I rode my bike this morning: how many abstractions or tools in software development push simplicity or understanding in only 1 direction? For example, containers can allow one who has the discipline to build simple systems and deploy them. They also allow a less disciplined engineer to make piles upon piles. The best example of a tool that only the needle in one direction, and I could be wrong, is a non-turing complete type system. As complex as any type system is, it is always a matter of understanding a static formalism and the language comes with a tool to test if mutations of the application of the type system are still correct. Personally, I think turing completeness is a bad thing and I hope in 100 years we look back at all these turing complete languages as a misstep.
“The best example of a tool that only the needle in one direction, and I could be wrong, is a non-turing complete type system.”
It’s one. These are straight-forward to apply and understand. John Nagle gave another example when the DAO got hit: decision tables. Pointed out both humans and computers could understand that formalism while it was decidable. There’s API’s that are also easy to use correctly. Ethos project investigates those for security. One last example is DSL’s. That leads to next one.
“ Personally, I think turing completeness is a bad thing and I hope in 100 years we look back at all these turing complete languages as a misstep.”
Maybe. There’s a tension between expressiveness and decidability. The DSL’s were able to productively express the problem in a concise way that’s easy to work with. They became a nightmare, esp integrating them, when the problem was beyond the DSL. The Turing-complete languages can handle all of it but can be a nightmare anyway. One middle ground I like is powerful languages with DSL support where you create Turing-incomplete DSL’s for each problem whose underlying properties are similar and integrate well. sklogic’s toolkit on Github’s combinatorylogic page is an example where he wrote a LISP compiler framework then DSL’s for it, including Standard ML. He might code in it in strong typing and restricted expressiveness unless the problem is too hard to solve that way. Then, he can fall back on LISP. If it’s XML or whatever, he’s got a DSL for that too that makes it easy. And so on. We’re seeing something similar occurring with Haskell DSL crowd but theoretically even easier analysis of those.
So, Turing Complete may not be a bad thing so much as overusing the concept. Perhaps a Turing-complete foundation that we always constrain just enough to express current problem makes more sense. Heck, that even sounds like POLA pattern from my field. ;)
If containers are a zero-cost abstraction, is it any different from running 50 programs on your machine?
My understanding is that containers are reaching zero-cost, so being able to, say, easily deploy multiple programs with different deployment requirements easily seems like a major win for ops.
Everything’s a tradeoff, of course. But getting a consistent environment is still a major difficulty due to how Unices deal with so many things like fonts, localisation, etc (Env variables! conf files! daemon behavior!). Being able to fix everything about a program’s environment is awesome.
Try using any PDF rendering tool in an undefined environment and getting consistency of output.
In a world where you rely on software written by other people, being able to isolate those programs and define their dependencies is very helpful! At least in theory.
Excellent writeup, as always.
But, why? Why would I want 50 containers on the same machine?
I’m always put in mind of this image whenever reading about the trials and tribulations of Docker et al.
The most efficient way to do things is to have massive machines with many many jobs on them. This is as it reduces resource overhead (think of all the management daemons that run per machine), makes scheduling easier and allows for over-provisioning.
This makes great sense if you’re the one renting out those massive machines–but again, I’m curious about the developers and products themselves.
Would we make these same decisions if there wasn’t a billion-dollar industry predicated on developer outreach and education/advertising/brainwashing pushing this as a solution?
I’m seeing technology like the Erlang runtime and wondering if maybe people should just write on less bloated platforms. Maybe that could solve the efficiency problem–50 Ruby VMs is hardly more efficient than a 50 C programs, for example.
For us, 50 containers on the same machine are 50 different builds running 50 different test suites. We have a cluster that runs tests in parallel, in containers, across many machines. I can’t imagine getting anything done if I had to run all those tests locally.
People also use sorts of Linux or macOS (with all sorts of Linux VMs). Now the build team doesn’t have to support all the different setups. They also don’t have to do flag days as much, they just update the build container and move on.
I run a couple apps in containers on a DO box, each of which shells out to a few other programs written in a variety of languages (ruby, c, node, java). None of these have to care what OS they’re running on or if the packages for those programs are available, or where those binaries are going to be installed to. I just ship the container and I’m good.
Shipping each app with it’s own isolated environment seems beautiful to me, and I don’t see why you wouldn’t want such an abstraction. It’d be great if all the programs I depended on were all written as libraries for the language I wrote my webapp in, but they weren’t.
The current popular implementation of containers is strictly less useful than packages as they are OS specific. As someone who does not run Linux, using containers is a pain for me so I have to run a Linux VM to run the container. Whereas the source artifacts I could build for my OS. So containers have taken OS-independent languages like Python, Ruby, and Java and made OS-dependent artifacts. Personally, I think our lives would be much much simpler if we standardized on a package manager like Nix rather than standardized on a binary artifact such as a container.
For the sake of clarity, do you think Erlang is bloated, or an example of a non-bloated platform?
I’ve come around to the idea of containers, now that Kubernetes exists.
The idea is that you ought to be able to submit a program, bounded with memory and disk and bandwidth expectations, and have that program encapsulated in such a way that it is trivial to operate (and ideally has no or few state expectations). Then, scaling and operation become super easy.
It’s not for every use case, but it’s a pretty nice sweet spot with acceptable tradeoffs if you have lots of independent services, large scale, generally stateless operation (or have magic, like Google does), many different development teams and/or styles, an unknown scaling requirement on almost every service, enough professionalism to write and keep API contracts, and an ops team that really doesn’t want to deal with your nonsense and wants a rollback button (and/or wants you to deploy and undeploy your own code without blowing up the world).
The big problem is state; Google’s datacenters are literally magic, so they’ll be fine. Everyone else, currently, that’s a problem.
Those all sound like problems that should be being solved at the application layer, and perhaps with better support at the OS layer.
My current impression–and please do correct me if I’m wrong!–is that containers are nowadays basically tacit acknowledgements that, because people lack the discipline/skill/tooling to write fairly self-contained blobs, we need to provide an artificial boundary sack to hold all their garbage in to reduce the ops headaches they would otherwise be creating.
It would seem to me that the sustainable, long-term solution would be to have developers that understand how not to pull in a clowncar of dependencies (cough node ruby cough) and how to use the tools that their systems provide them out of the box for managing quotas/networking/filesystems and so forth. Of course, that’s expensive, and there isn’t money to be made by just telling devs to RTFM.
That’s what containers are.
How do I ensure they are actually correctly self-contained? “Be more disciplined” sounds suspiciously like “don’t write bugs”. Containers are to multi-tenant deployment as immutable data structures are to multi-threaded programming.
This is irrelevant. Both node and ruby support full namespace isolation of dependencies, without containers. I could install every version of every rubygem on my laptop and my only problem would be lack of disk space.
The advantage of overlayfs was already discussed in Julia Evans' post. As for quotas, containers aren’t really about quotas as much as homogenous deployment tooling and safe multi-tenancy. Quotas are a stability afterthought—so much of an afterthought that they aren’t even that well implemented in public container projects.
As someone who does not run Linux, the modern container movement is a regression for me. With source artifacts, I could build packages for my OS (FreeBSD) and start jails to deploy them on for development. With modern containers, I need to run a Linux VM to run an opaque binary artifact just to do development. This makes my environment more complex rather than simpler. Oh well, I’m about a year or two from just being a curmudgeon yelling at people to get off my OS lawn :)
Jails are another name for containers, if you squint a little.
Yes, they are containers (heck, the first containers). However my point was specifically about how containers are working out right now, which are Linux-specific binary artifacts. At best, one has to emulate a Linux system to run a a docker image.
Docker is horrible and linux-specific. That’s a contingent fact about docker, not a fundamental issue with containerization.
Yes, I agree. My wording was not clear it seems. But “modern container movement” I’m referring to docker and friends.
I believe we’ll have a standard image format soon enough, but you’ll still need to put in the effort to build containers targeting the BSDs, since they presumably won’t run linux binaries. Well, I guess freebsd has emulation, but I’m not sure what the state of it is. Dragonfly and OpenBSD don’t, though I don’t think OpenBSD has jails to begin with.
Jails also unfortunately lack the granularity of namespaces but I don’t know how huge of a deal that is.
Oh let’s do be totally clear: the container ecosystem today is an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang and a tenth of Unix. That said, the number of fellow Erlang programmers I’ve ever physically met could fill a car, so pragmatically there’s something to be said for providing the good guarantees and affordances of Erlang in a form that even a node.js programmer can probably not fuck up.
There are a few problems with this part of your statement:
On Linux, containers are partially more or less an abstraction on top of the “modern” way of enforcing quotas
Outside of work, I’m not really a Linux user. However, what I do love about the modern Linux container movement is the idea of producing a self-contained image that’s easy to distribute, run and apply run-time constraints on. I’d love to see this applied to FreeBSD Jails. I’m not aware of any attempt at an implementation of this, other than the apparently abandoned jetpack.
Sustainable in what terms?
It’s stunningly expensive and time-consuming (decades!) to train people up to the level where they can build simple, reliable software with few dependencies. The training process produces a lot of complex, unreliable, useful working software. Is it sustainable to not use it?
I think you’re correct in pointing out “sustainable” as something in need of better specification.
For me, the current situation is “unsustainable” because:
There are absolutely people (in this thread for example) that use Docker et al for things like multi-tenant builds, which is a use-case for where it makes sense.
But the unsustainable thing is when people avoid fixing the shittiness of their tooling (Kafka, ML stuff, database deployments, etc.) by shoving it in a container and pretending they don’t have to learn how it works or how to manage it. And there seems, to me, to be a lot of that in the wind these days.
I have found it surprising that some of the same people who are so opposed to static compilation (with cries about how you would have to download updates for many programs instead of just openssl), somehow are just fine with containers.
I’m sure there’s more of it than there used to be - tech has grown, fast.
I’m somewhat less convinced that the fraction of technology in the category you’ve described has grown.
I don’t understand how else you’re supposed to allow people to run applications in an isolated environment, like AWS.
Install a package? AWS is already running a VM.
“The idea is that you ought to be able to submit a program, bounded with memory and disk and bandwidth expectations, and have that program encapsulated in such a way that it is trivial to operate (and ideally has no or few state expectations). ”
That’s a great description of processes running on QNX microkernel circa 1990’s. They also had a desktop on a floppy + self-healing to a degree. Something tells me there’s some middle ground between that and how container platforms are doing same thing with many MB of software. Later on, researching along lines of separation kernels, TU Dresden’s TUDOS demonstrator let me run a new user-mode VM a second for Linux on top of L4 microkernel on crappy hardware. Fire them up as fast as I can click them. That’s just an academic prototype. Real-world stuff from RTOS vendors let you do Linux, Ada runtimes, Java subsets, C apps, and so on with same, efficient microkernel with autogenerate communication stubs for making them work together. Ada was safest but C and Java parts can support all the mainstream stuff. Also, vendors like Aonix supported lightweight VM’s with AOT compilation into self-contained ROM’s you can link in.
I’m with angersock in at least how the modern stuff looks like giant piles of crap on top of piles of crap compared to even 90’s version of better architectures. Let’s not even talk about security where modern stuff’s security for multi-tenancy is like a joke compared to GEMSOS or KeyKOS from the 1980’s or Karger’s security kernel for VAX VMM from 90’s. I’m disagreeing with angersock in that it can be a wise solution to chose one of these piles if you’re working with many other piles in a way that needs isolation and management that can’t be easily done with better architecture. If you have to.
I was thinking about this as I rode my bike this morning: how many abstractions or tools in software development push simplicity or understanding in only 1 direction? For example, containers can allow one who has the discipline to build simple systems and deploy them. They also allow a less disciplined engineer to make piles upon piles. The best example of a tool that only the needle in one direction, and I could be wrong, is a non-turing complete type system. As complex as any type system is, it is always a matter of understanding a static formalism and the language comes with a tool to test if mutations of the application of the type system are still correct. Personally, I think turing completeness is a bad thing and I hope in 100 years we look back at all these turing complete languages as a misstep.
“The best example of a tool that only the needle in one direction, and I could be wrong, is a non-turing complete type system.”
It’s one. These are straight-forward to apply and understand. John Nagle gave another example when the DAO got hit: decision tables. Pointed out both humans and computers could understand that formalism while it was decidable. There’s API’s that are also easy to use correctly. Ethos project investigates those for security. One last example is DSL’s. That leads to next one.
“ Personally, I think turing completeness is a bad thing and I hope in 100 years we look back at all these turing complete languages as a misstep.”
Maybe. There’s a tension between expressiveness and decidability. The DSL’s were able to productively express the problem in a concise way that’s easy to work with. They became a nightmare, esp integrating them, when the problem was beyond the DSL. The Turing-complete languages can handle all of it but can be a nightmare anyway. One middle ground I like is powerful languages with DSL support where you create Turing-incomplete DSL’s for each problem whose underlying properties are similar and integrate well. sklogic’s toolkit on Github’s combinatorylogic page is an example where he wrote a LISP compiler framework then DSL’s for it, including Standard ML. He might code in it in strong typing and restricted expressiveness unless the problem is too hard to solve that way. Then, he can fall back on LISP. If it’s XML or whatever, he’s got a DSL for that too that makes it easy. And so on. We’re seeing something similar occurring with Haskell DSL crowd but theoretically even easier analysis of those.
So, Turing Complete may not be a bad thing so much as overusing the concept. Perhaps a Turing-complete foundation that we always constrain just enough to express current problem makes more sense. Heck, that even sounds like POLA pattern from my field. ;)
FWIW Google’s datacenters all run on containers.
If containers are a zero-cost abstraction, is it any different from running 50 programs on your machine?
My understanding is that containers are reaching zero-cost, so being able to, say, easily deploy multiple programs with different deployment requirements easily seems like a major win for ops.
Everything’s a tradeoff, of course. But getting a consistent environment is still a major difficulty due to how Unices deal with so many things like fonts, localisation, etc (Env variables! conf files! daemon behavior!). Being able to fix everything about a program’s environment is awesome.
Try using any PDF rendering tool in an undefined environment and getting consistency of output.
In a world where you rely on software written by other people, being able to isolate those programs and define their dependencies is very helpful! At least in theory.