I work at one, and we pushed microcode updates to our complete fleet within hours. Luckily we were able to late load the update (i.e., update the microcode at runtime), or it would have taken us a lot longer.
For us, this was a heartbleed level kind of bug. It was trivial to see the CPU registers of ofher VMs and I have no doubt that we’ll see new tools pop up that are able to make sense of the noise and string together secret keys and the like.
It is not quite clear that this is generally safe, and at least with Intel I think you should have hyper threading disabled, but AMD seems to support this. Rebooting is safer, but in our tests it worked well, and it allowed us to react a lot quicker.
Basically we downloaded the latest microcode binary from the linux firmware repository and ran the command above, while closely monitoring the hosts as we gradually rolled out the fix.
The conflict between the Docker and systemd developers is very interesting to me. Since all the Linux machines I administer already have systemd I tend to side with the Red Hat folks. If I had never really used systemd in earnest before maybe it wouldn’t be such a big deal.
…and with good security if you build your own containers with debootstrap instead of pulling stuff made by random strangers on docker hub.
I was glad to see this comment.
I have fun playing with Docker at home but I honestly don’t understand how anyone could use Docker Hub images in production and simultaneously claim to take security even quasi-seriously. It’s like using random npm modules on your crypto currency website but with even more opaqueness. Then I see people arguing over the relative security of whether or not the container runs as root but then no discussion of far more important security issues like using Watchtower to automatically pull new images.
I’m no security expert but the entire conversation around Docker and security seems absolutely insane.
That’s the road we picked as well, after evaluating Docker for a while. We still use Docker to build and test our containers, but run them using systemd-nspawn.
To download and extract the containers into folders from the registry, we wrote a little go tool: https://github.com/seantis/roots
Inside these spaces, we can launch Linux-based operating systems.
This keeps confusing me. When I first saw containers, I saw them described as light weight VM’s. Then I saw people clarifying that they are really just sandboxed Linux processes. If they are just processes, then why do containers ship with different distros like Alpine or Debian? (I assume it’s to communicate with the process in the sandbox.) Can you just run a container with a standalone executable? Is that desirable?
EDIT
Does anyone know of any deep dives into different container systems? Not just Docker, but a survey of various types of containers and how they differ?
Minimally built containers reduce attack surface, bring down image size, serve as proof that your application builds in a sterile environment and act as a list with all runtime dependencies, which is always nice to have.
Then I saw people clarifying that they are really just sandboxed Linux processes.
This statement is kinda true (my experience is limited to Docker containers). Keep in mind more than one process can run on a container, as containers have their own PID namespace.
If they are just processes, then why do containers ship with different distros like Alpine or Debian?
Because containers are spun up based on a container image, which is essentially a tarball that gets extracted to the container process’ root filesystem.
Said filesystem contains stuff (tools, libraries, defaults) that represents a distribution, with one exception: the kernel itself, which is provided by the host machine (or a VM running on the host machine, à la Docker for Mac).
Can you just run a container with a standalone executable? Is that desirable?
Yes, see my prometheus image’s filesystem, it strictly contains the prometheus binary and a configuration file.
In my experience, minimising a container image’s contents is a good thing, but for some cases you may not want to. Applications written in interpreted languages (e.g. Python) are very hard to reduce down to a few files in the image, too.
I’ve had most success writing minimal container images (check out my GitHub profile) with packages that are either written in Go, or that have been around for a very long time and there’s some user group keeping the static building experience sane enough.
I find the easier something is to put into a docker container, the less point there is. Go packages are the ideal example of this: building a binary requires 1 call to a toolchain which is easy to install, and the result has no library dependencies.
Afaik: Yes, you can and yes, it would be desirable. I think dynamically linked libraries were the reason why people started to use full distributions in containers. For a Python environment you would probably have to collect quite a few different libraries from your OS to copy into the container so that Python can run.
If my words are true then in the Go environment you should see containers with only the compiled binary? (I personally installed all my go projects without containers, because it’s so simple to just copy the binary around)
If you build a pure Go project, this is true. If you use cgo, you’ll have to include the extra libraries you link to.
In practice, for a Go project you might want a container with a few other bits: ca-certificates for TLS, /etc/passwd and /etc/group with the root user (for “os/user”), tzdata for timezone support, and /tmp. gcr.io/distroless/static packages this up pretty well.
You can have very minimal containers. Eg. Nix’s buildLayeredImage builds layered Docker images from a package closure. I use it to distribute some NLP software, the container only contains glibc, libstdc++, libtensorflow, and the program binaries.
I’m personally not a fan of Python async code as it adds visual noise in my opinion, but I can understand why you would chose that model.
Apart from that noise, the task model looks well thought out and approachable. I think that is pretty important in any kind of tool that wants to be an alternative to Puppet, Ansible and what have you. Part of what made Ansible big is probably the number of modules that volunteers added themselves.
Having people contribute back is key to the success of something like this. While Ansible adopted their third-party repository of modules to use, my hope with Pitcrew is to make it more first-party, think homebrew with the “fork and pr” strategy of contributing to it.
I’m also struggling with the noise of async code in python, really wish it could be more like Ruby fibers.
I don’t have hundreds of hosts handy, so not easy for me to benchmark it, but that would be fun work.
Fabric seems to use a process per connection, so seems like a downside instead of using nonblocking io.
It looks like it should be quite similar to Mitogen. Having spent much time in the Yaml forests of Ansible, I can say I don’t personally want live there (see: inner platform effect). Also, Pitcrew should be able to support an inverted control strategy, where you sync the code then execute it local to where it’s running, to avoid roundtrip latency.
Is this a huge issue for cloud providers? Or do they set the “chicken bit” already?
I work at one, and we pushed microcode updates to our complete fleet within hours. Luckily we were able to late load the update (i.e., update the microcode at runtime), or it would have taken us a lot longer.
About late loading see https://www.kernel.org/doc/html/next/x86/microcode.html
For us, this was a heartbleed level kind of bug. It was trivial to see the CPU registers of ofher VMs and I have no doubt that we’ll see new tools pop up that are able to make sense of the noise and string together secret keys and the like.
Can you do that on the fly or do you reboot the entire fleet for that?
You can do that on the fly:
It is not quite clear that this is generally safe, and at least with Intel I think you should have hyper threading disabled, but AMD seems to support this. Rebooting is safer, but in our tests it worked well, and it allowed us to react a lot quicker.
Basically we downloaded the latest microcode binary from the linux firmware repository and ran the command above, while closely monitoring the hosts as we gradually rolled out the fix.
For shell scripts on my local machine I use https://elv.sh, a shell with a rather neat scripting language that borrows from all kinds of languages.
With the built-in container support in SystemD you don’t even need new tools:
https://blog.selectel.com/systemd-containers-introduction-systemd-nspawn/
…and with good security if you build your own containers with debootstrap instead of pulling stuff made by random strangers on docker hub.
The conflict between the Docker and systemd developers is very interesting to me. Since all the Linux machines I administer already have systemd I tend to side with the Red Hat folks. If I had never really used systemd in earnest before maybe it wouldn’t be such a big deal.
I was glad to see this comment.
I have fun playing with Docker at home but I honestly don’t understand how anyone could use Docker Hub images in production and simultaneously claim to take security even quasi-seriously. It’s like using random npm modules on your crypto currency website but with even more opaqueness. Then I see people arguing over the relative security of whether or not the container runs as root but then no discussion of far more important security issues like using Watchtower to automatically pull new images.
I’m no security expert but the entire conversation around Docker and security seems absolutely insane.
That’s the road we picked as well, after evaluating Docker for a while. We still use Docker to build and test our containers, but run them using systemd-nspawn.
To download and extract the containers into folders from the registry, we wrote a little go tool: https://github.com/seantis/roots
From your link:
This keeps confusing me. When I first saw containers, I saw them described as light weight VM’s. Then I saw people clarifying that they are really just sandboxed Linux processes. If they are just processes, then why do containers ship with different distros like Alpine or Debian? (I assume it’s to communicate with the process in the sandbox.) Can you just run a container with a standalone executable? Is that desirable?
EDIT
Does anyone know of any deep dives into different container systems? Not just Docker, but a survey of various types of containers and how they differ?
Containers are usually Linux processes with their own filesystem. Sandboxing can be good or very poor.
Not desirable. An advantage of containers over VMs is in how easily the host can inspect and modify the guest filesystem.
Minimally built containers reduce attack surface, bring down image size, serve as proof that your application builds in a sterile environment and act as a list with all runtime dependencies, which is always nice to have.
May I ask why isn’t it desirable?
You can attach to a containerized process just fine from the host, if the container init code doesn’t go out of it’s way to prevent it.
gdbaway.I’m not sure if it’s as deep as you’d like, but https://www.ianlewis.org/en/tag/container-runtime-series might be part of what you’re looking for.
This looks great! Thank you for posting it.
This statement is false, indeed.
This statement is kinda true (my experience is limited to Docker containers). Keep in mind more than one process can run on a container, as containers have their own PID namespace.
Because containers are spun up based on a container image, which is essentially a tarball that gets extracted to the container process’ root filesystem.
Said filesystem contains stuff (tools, libraries, defaults) that represents a distribution, with one exception: the kernel itself, which is provided by the host machine (or a VM running on the host machine, à la Docker for Mac).
Yes, see my prometheus image’s filesystem, it strictly contains the
prometheusbinary and a configuration file.In my experience, minimising a container image’s contents is a good thing, but for some cases you may not want to. Applications written in interpreted languages (e.g. Python) are very hard to reduce down to a few files in the image, too.
I’ve had most success writing minimal container images (check out my GitHub profile) with packages that are either written in Go, or that have been around for a very long time and there’s some user group keeping the static building experience sane enough.
I find the easier something is to put into a docker container, the less point there is. Go packages are the ideal example of this: building a binary requires 1 call to a toolchain which is easy to install, and the result has no library dependencies.
They’re not just processes: they are isolated process trees.
Why Alpine: because the images are much smaller than others.
Why Debian: perhaps because reliable containers for a certain application happen to be available based on it?
Afaik: Yes, you can and yes, it would be desirable. I think dynamically linked libraries were the reason why people started to use full distributions in containers. For a Python environment you would probably have to collect quite a few different libraries from your OS to copy into the container so that Python can run.
If my words are true then in the Go environment you should see containers with only the compiled binary? (I personally installed all my go projects without containers, because it’s so simple to just copy the binary around)
If you build a pure Go project, this is true. If you use cgo, you’ll have to include the extra libraries you link to.
In practice, for a Go project you might want a container with a few other bits: ca-certificates for TLS, /etc/passwd and /etc/group with the root user (for “os/user”), tzdata for timezone support, and /tmp.
gcr.io/distroless/staticpackages this up pretty well.You can have very minimal containers. Eg. Nix’s
buildLayeredImagebuilds layered Docker images from a package closure. I use it to distribute some NLP software, the container only contains glibc, libstdc++, libtensorflow, and the program binaries.Author here, I’d greatly appreciate any feedback, ideas, critiques! Thanks :)
I’m personally not a fan of Python async code as it adds visual noise in my opinion, but I can understand why you would chose that model.
Apart from that noise, the task model looks well thought out and approachable. I think that is pretty important in any kind of tool that wants to be an alternative to Puppet, Ansible and what have you. Part of what made Ansible big is probably the number of modules that volunteers added themselves.
Having people contribute back is key to the success of something like this. While Ansible adopted their third-party repository of modules to use, my hope with Pitcrew is to make it more first-party, think homebrew with the “fork and pr” strategy of contributing to it.
I’m also struggling with the noise of async code in python, really wish it could be more like Ruby fibers.
How does this compare to fabric? Or ansible with mitogen?
I don’t have hundreds of hosts handy, so not easy for me to benchmark it, but that would be fun work.
Fabric seems to use a process per connection, so seems like a downside instead of using nonblocking io.
It looks like it should be quite similar to Mitogen. Having spent much time in the Yaml forests of Ansible, I can say I don’t personally want live there (see: inner platform effect). Also, Pitcrew should be able to support an inverted control strategy, where you sync the code then execute it local to where it’s running, to avoid roundtrip latency.