Before laying out my question, let me put it into context. It seems that lately attackers are more successful by targeting administrators or developers than attacking the actual deployments and servers. Meanwhile some attacks social engineer employees into credential theft or reset, other attacks go after operational credentials that are used in semi-automated operations or development scripts.
To be clear, I don’t want to focus on the issue of secure online credential management (I hope we all use some form of password manager), or the security of online authentication protocols (even OAuth seems to be lacking), or the security of MFAs, not even the security of cloud API keys that are actively used by unattended running services (if the attacker has compromised the actual server he most likely already has access to all the sensitive data he wants).
What I want to focus on is those credentials that are used mainly for semi-automated operational or development scripts. Like for example AWS secret keys used to deploy or to gain access to the underlying buckets or backups; or passwords used to access the SQL database for maintenance; or other such secrets that are used outside the browser or specialized applications like SSH.
Also, let’s assume we are speaking in the context of a small company with at most 10 employees. Large companies probably have 10 people on the security team tasked only with this topic, thus perhaps this problem is solved in more complex ways.
So, how do the administrators and developers in this community manage such secrets when writing semi-automatic scripts?
I’m interested especially on the following aspects:
grep
-ing the scripts code for random looking tokens?/tmp
?scp
it to /tmp
?Sorry for the lengthy question, but I do wonder about this for some time…
I have a bone to pick with secret management strategies.
IMO there are two options for secret management: either use a hardware key as your root of trust for everything, or lie to yourself. No one does the former perfectly, so everyone’s doing the latter.
Developers are constantly executing arbitrary un-sandboxed code on their computers due to dependencies, so you can just assume any file and any block of memory on your computer can be stolen without you knowing. As a result, we can conclude:
Any combination of those tools still depends on some root of trust that an attacker can exfil. The only solution is a key stored in hardware that physically can’t be exfil’d. Use something like PKCS11 to send challenges to your hardware key to access everything.
This assumes a perfect world where you have no legacy systems of course, but I’m shocked at the amount of engineering time thrown at alternate solutions that just add complexity.
In absolute terms you might be right: everything is broken in one way or another outside (hopefully?) hardware keys. However that doesn’t mean we should all just give up and create a large
PASSWORDS.txt
file on the desktop.Every additional step and measure might make the life harder for the attacker. (Hopefully harder than it makes life for us.) :)
To clarify, I’m not advocating an all or nothing approach; more just complaining that if every company who built all the half-baked secret management systems mentioned in this thread had been using their brains, we would have actually secure secrets by now.
I completely agree on this. For example I bought in 2012 two ePass2003 RSA tokens (2048 bit max) and wanted (at that time) to use them for TLS authentication plus SSH and GnuPG. I failed because the support (in Linux) was quite unreliable and I didn’t even manage to have it working with GnuPG. Thus I’ve shelved them.
At the end of 2022 (10 years later, and still the “current” offer from Feitian) I took them for a ride hoping things have changed… Nop… I though that at least I can use them in my own custom tools via PKCS#11 interface… Nop… Just looking at the documentation made me dizzy… :)
So yeah… If all these companies would have just spent 10% of the time and effort to actually make PKCS#11 compatible tokens work the story would have been quite different… (Or just come up with a working alternative, but standardize on something and support it long term…)
And looking at the whole YubiKey tokens ecosystem doesn’t give me much hope… Sure there is https://github.com/str4d/age-plugin-yubikey but that’s not enough…
Absolute security in the presence of an adversary is unattainable but what happens more often is credentials leak. Someone committed a hardcoded API key. Some script dumped its env into a logfile for debugging and it ended up full of secrets. A service has access to more credentials than it needs and used the wrong one to access (a wrong) db or a production API instead of staging. Those kind of things.
At Determinate Systems, we use vault for secrets. HCP Vault is fine. We don’t share secrets, we get them from Vault and store them in the environment as needed. They all expire on a regular basis (maximum duration of ~12h.) Users get credentials based on their role / group membership in our Google org. All our access is gated on roles:
I can “become” the “appX_developer” role, which allows me to get credentials for appX. Without that step, though, I can’t get appX credentials. Each of these steps creates audit logs.
Vault feels heavy to start with, but it is really a wonderful tool and the documentation is very good.
The problem with HashiCorp’s Vault (and I’m sure there are other tools like it) is that it works in a centralized environment: there is “the vault server”, and to get there you need to be either to be on premise or VPN in. If your internet is spotty or you are traveling – which seems to be happen more often than not with smaller startups – I don’t think will work as nice.
Moreover, I bet everyone that uses Vault still has a “master password list” tucked away somewhere, because if Vault goes down (bug, infrastructure issue, etc.) and there is an emergency and you need to get into your infrastructure then what do you do? :)
Great questions.
Our Vault is actually open to the internet, though using something like tailscale ephemeral tokens makes it a lot easier to lock that down like that.
We don’t have a master password list. What we can do is log in to AWS’s console using our Google accounts and get AWS tokens to go poking around. We still don’t get long-term tokens through that.
Anyway, I’m not sure these are “the problem” with Vault so much as an inherent tension of secret management where you need to make choices.
This depends on the type of secret, but for infrastructure we have a central repository to house all of our infrastructure-specific secrets (e.g. the administrative password to get into some SaaS offering that marketing uses, back-up passphrases, etc.) and we use Mozilla’s SOPS for state, which hooks in nicely with Google for authorizing the people who can even decrypt the main file. SOPS also plays delightfully with Ansible, which is a huge plus in my eyes as well as it then loses any friction that may normally come with working with secrets.
To share secrets between employees if they aren’t in our team (e.g. “hey, here’s the reset password for service X”) we set up our own instance of Password Pusher and it’s been great as it takes care of deleting the secret after a set (customizable) period of time. Our practice (albeit not a good one) prior to Password Pusher was to send something in Slack and as soon as the other person gave the green light we deleted the original message containing the secret. We had no allusions that this would actually wipe it from Slack’s nooks and crannies though.
1password has a CLI
Interesting tool; it mostly focuses on environment variables and template files. I’ve seen a similar one from Mozilla, https://github.com/mozilla/sops.
However both are what I would call file-system leaky… These tools might be wonderful for developer onboarding, by providing a way to bootstrap one’s credentials, but afterwards the resulting files (based on the templates) remain on the file-system. And with today’s Btrfs’s and ZFS’s once it reached the file-system, it will “forever” be leaked…
What would leak?
Let’s take their example from https://developer.1password.com/docs/cli/secrets-config-files/:
Now your secrets are in plain on the disk inside
config.yml
. It might be excluded from Git, or you might delete it afterwards, but if the attacker gets into your laptop, he can justdd if=/dev/sda | tr -c ' -~' '\n' | grep -C 100 -F -e 'username:'
and now he can start looking at older versions of these files.The only protection against this type of attack is to make sure that the
config.yml
is stored on aramfs
(which in Linux doesn’t leak in swap) or intmpfs
with encrypted swap.The safer alternative with this tool is to use the environments approach:
I loved Keybase for this, shame it’s on the green mile now.
I could keep my credentials there and just tell my local development instance of whatever I’m working on to load its .env from K:\
Don’t write scripts. Ever.
That’s my hard-line rule for teams and projects I’ve run. Scripts are, by nature, temporary. And there’s nothing more permanent than temporary.
My advice to people over the years has been to always build or use a product that solves your problem and treat it with the same practices you would other products in the domain. So, don’t write a deployment “script”, use a pre-existing tool that’s backed by a quality company/team (preferably written in a “proper language” not a “scripting language”.) Secrets have a very small surface area that they are allowed to exist in: GitHub Actions variables are one of those areas for example. Sticking to some simple, strongly enforced rules like this means you’ve only got a couple of places where secrets are accessible from.
Oh and cycle credentials frequently, make them expire, force people to get into the habit of updating configurations with the new creds if needed, treat it like a fire drill.
Although I would really like to agree with you, it’s just practically impossible…
Well, a script matches perfectly this description: someone has built an in-house product to solve one’s problem; it just happens that the developer has chosen
bash
to do it and not say Go. Certainly the developer, just like the other professional tool developers, has treated this in-house product with the up-most care and attention. :)Getting serious, regardless of how many pre-existing tools one uses, at one point one would definitively want to automate some chaining of some tools that don’t support it out-of-the-box. Then what? Write another tool?
And this is exactly part of the question I was asking: how does one “update configurations” with new credentials? By hand? With some other tool?
My point was more about the mental model of what a “script” is - you can write a “script” in any language, it just has vibes of hackyness and carelessness. I’ve written scripts in Go that ignore all errors because they’re use-once. I’ve also encountered these kinds of Go programs in-the-wild doing important tasks.
It’s about practices not language/tools. When I say “don’t write scripts” I’m really saying don’t solve problems as if they’re one-time-thing “I just gotta quickly automate this” scripts. If you treat the underlying problem as a product problem that’s a permanent fixture of your environment, you can build a proper solution not a hacky script.
As for updating config, I generally do that manually and ensure many people on the team also know how and when to do it. But I’ve also not worked at real large scale so that probably changes once you hit hundreds of services running globally, and this is the point where my experience ends.
Your question really covers two different tiers of things. For things were there is any way to avoid using a shared secret, avoid it. With AWS and like https://github.com/99designs/aws-vault – the secret is stored in OS protected space (which has whole teams of engineers securing it) and you get temporary creds, which obviously solves basically all your concerns.
But, the real issue is with shared keys you have no way to vary per developer. The first protection is don’t ever let the developers see the production keys. They live somewhere in your AWS Secrets (both in prod and dev) – but the developers can only ever see the dev keys, and devops can see the prod keys. Far from perfect but helpful in limiting damage scope.
Indeed with AWS that provides a way to work with temporary credentials (are there other cloud providers that do this?) the solution you’ve highlighted might work. But 99% of the other providers just give you a random “secret API key” and are done with it…
The major players have similar solutions. But, speaking to the situation where you don’t have such a system – you can still leverage a secrets store. Gives you a single place to manage secrets, gives you a single place to kill / rotate creds and can keep your developers from ever seeing the production keys.
I’ve been using an encrypted JSON file to hold credentials. The script prompts for a passphrase, then uses that to decrypt and parse the json file.
Nothing fancy: Locally I’m using a combination of gopass + direnv. I use this setup for secrets that I generate or (more likely) I am going to be the only consumer. For secrets that are meant to be used by the rest of my team we rely on vault.
direnv allows me to just automatically have my secrets available when I’m in the repository for a given project as environment variables. gopass usually requests me to confirm the loading of the secrets either via Touch ID or with a Yubikey.
Since my team mostly deploys into our Nomad cluster, we rely on vault for making those secrets available in our containers.
For my personal projects I use a combination of sops and age. I decrypt and copy the secrets I need for the project to an .env file, and once I am done I delete it.
Some AWS-specific points, though I assume other cloud providers do similar things:
sts:AssumeRoleWithWebIdentity
to get short-lived credentials. This means that if an attacker runs code on CI, they can’t exfiltrate keys and use them forever, but of course they can still do a lot of damage.sts:AssumeRole
API can take a TOTP code; the AWS CLI and other tools can use this to grant temporary credentials only when you provide a 2FA token. This limits the amount of havoc that can be caused from a lost set of developer AWS Keys.