I think there’s one more potential attack that’s missing: a vulnerability or misconfiguration in etcd (or wherever else you store your secrets) that allows reading without requiring root access or physical access to the machine. In that scenario, encrypted secrets can provide another layer of security assuming the decryption is done via a separate mechanism.
I also don’t think “most people aren’t using feature X of Vault” is that strong an argument. You can’t dismiss a tool by insisting people aren’t using it correctly.
Yeah the dismissiveness of vault is mainly just me ranting. Maybe should have been an aside or footnote because the argument doesn’t rely on people misusing Vault. A properly configured Vault instance is what I ultimately compared to plain Kubernetes Secrets.
I agree that Vault is a complicated beast (I used to manage Vault at a previous employer) but USP for Vault must be the dynamic secret with TTLs right?
So even if you could read the secret from the RAMdisk on the node/pod it would not be usable unless you timed it so that you read it exactly before the service read it but after the injector well injected it.
My understanding is that while Vault can perform automatic password rotation, it can’t e.g. configure Redis or MySQL to change the password automatically. You could build something that does that for every secret-consuming application, but now vault is relegated to being a random password generator and again could be replaced with plain kubernetes secrets, /dev/urandom, and a cronjob.
I think the real value from Vault is the policies, not just storage. If a deployment is not taking advantage of that, then yes it’s no better than etcd or anything else.
Kubernetes and security don’t belong in the same sentence. That entire cottage industry is a marketing department larping as a software engineer. While I totally agree your average company is consuming too much software to scale to the traditional unix security model of users/groups/etc. k8s comes in and murders the vm isolation that has existed for the past 20 years. Own a single node in k8s and you now probably have access to all infrastructure - not just the instance you popped. Lateral traversal becomes orders of magnitude easier. Linux has always had major security issues but k8s really rubs me the wrong way and makes the 9x malware scene look like a toy shop.
IMO container isolation is generally “good enough” for most purposes, especially with recent efforts to avoid running as root.
Despite that, though…k8s can multiple compatible container runtimes, including Firecracker which uses VMs or gVisor which uses a userland kernel implementation with a heavy focus on sandboxing.
The whole problem with containers is that they were never designed from a security standpoint. They were designed so you could compile an application with a correct build environment. Then the marketing people started ramping up the false “security” claims and so a lot of people have this very very false notion that they have some sort of security inside them. They don’t. Kubernetes takes this idea and expands it across an infrastructure fleet compounding the problem considerably.
Containers are an inherently broken construct and can not be “fixed” with the addition of so-called devsecops software. All the static and dynamic analysis in the world doesn’t fix it.
Firecracker provides some value although it isn’t appropriate for all workloads. As for gVisor - it’s basically a non-starter unless you have your own hardware or are using nested virtualization or things like nitro.
Oh man, that bit on Vault Sidecar Injector hit hard. My team uses Vault (self-hosted, so you can guess where this is going), and it’s basically a glorified key-value store. We talk about using the value-adds that Vault brings, but never make progress because instead of letting one person dive deep and become an expert, we shuffle ticket assignments around and let someone learn just enough to say “yeah, this is feasible but I don’t know how to get us there”.
Good writeup.
I think there’s one more potential attack that’s missing: a vulnerability or misconfiguration in etcd (or wherever else you store your secrets) that allows reading without requiring root access or physical access to the machine. In that scenario, encrypted secrets can provide another layer of security assuming the decryption is done via a separate mechanism.
I also don’t think “most people aren’t using feature X of Vault” is that strong an argument. You can’t dismiss a tool by insisting people aren’t using it correctly.
Yeah the dismissiveness of vault is mainly just me ranting. Maybe should have been an aside or footnote because the argument doesn’t rely on people misusing Vault. A properly configured Vault instance is what I ultimately compared to plain Kubernetes Secrets.
I agree that Vault is a complicated beast (I used to manage Vault at a previous employer) but USP for Vault must be the dynamic secret with TTLs right? So even if you could read the secret from the RAMdisk on the node/pod it would not be usable unless you timed it so that you read it exactly before the service read it but after the injector well injected it.
Are we talking about https://www.hashicorp.com/resources/painless-password-rotation-hashicorp-vault ?
My understanding is that while Vault can perform automatic password rotation, it can’t e.g. configure Redis or MySQL to change the password automatically. You could build something that does that for every secret-consuming application, but now vault is relegated to being a random password generator and again could be replaced with plain kubernetes secrets, /dev/urandom, and a cronjob.
I think the real value from Vault is the policies, not just storage. If a deployment is not taking advantage of that, then yes it’s no better than etcd or anything else.
Kubernetes and security don’t belong in the same sentence. That entire cottage industry is a marketing department larping as a software engineer. While I totally agree your average company is consuming too much software to scale to the traditional unix security model of users/groups/etc. k8s comes in and murders the vm isolation that has existed for the past 20 years. Own a single node in k8s and you now probably have access to all infrastructure - not just the instance you popped. Lateral traversal becomes orders of magnitude easier. Linux has always had major security issues but k8s really rubs me the wrong way and makes the 9x malware scene look like a toy shop.
IMO container isolation is generally “good enough” for most purposes, especially with recent efforts to avoid running as root.
Despite that, though…k8s can multiple compatible container runtimes, including Firecracker which uses VMs or gVisor which uses a userland kernel implementation with a heavy focus on sandboxing.
The whole problem with containers is that they were never designed from a security standpoint. They were designed so you could compile an application with a correct build environment. Then the marketing people started ramping up the false “security” claims and so a lot of people have this very very false notion that they have some sort of security inside them. They don’t. Kubernetes takes this idea and expands it across an infrastructure fleet compounding the problem considerably.
Containers are an inherently broken construct and can not be “fixed” with the addition of so-called devsecops software. All the static and dynamic analysis in the world doesn’t fix it.
Firecracker provides some value although it isn’t appropriate for all workloads. As for gVisor - it’s basically a non-starter unless you have your own hardware or are using nested virtualization or things like nitro.
Oh man, that bit on Vault Sidecar Injector hit hard. My team uses Vault (self-hosted, so you can guess where this is going), and it’s basically a glorified key-value store. We talk about using the value-adds that Vault brings, but never make progress because instead of letting one person dive deep and become an expert, we shuffle ticket assignments around and let someone learn just enough to say “yeah, this is feasible but I don’t know how to get us there”.
Not just attack surface, but probability of misconfiguring or inadvertently neutering the solution by introducing a regression.