How do you securely manage secrets in scripts and development environments?

6

How do you securely manage secrets in scripts and development environments? ☶ ask security
authored by ciprian_craciun 2 months ago | 24 comments

24

Before laying out my question, let me put it into context. It seems that lately attackers are more successful by targeting administrators or developers than attacking the actual deployments and servers. Meanwhile some attacks social engineer employees into credential theft or reset, other attacks go after operational credentials that are used in semi-automated operations or development scripts.

To be clear, I don’t want to focus on the issue of secure online credential management (I hope we all use some form of password manager), or the security of online authentication protocols (even OAuth seems to be lacking), or the security of MFAs, not even the security of cloud API keys that are actively used by unattended running services (if the attacker has compromised the actual server he most likely already has access to all the sensitive data he wants).

What I want to focus on is those credentials that are used mainly for semi-automated operational or development scripts. Like for example AWS secret keys used to deploy or to gain access to the underlying buckets or backups; or passwords used to access the SQL database for maintenance; or other such secrets that are used outside the browser or specialized applications like SSH.

Also, let’s assume we are speaking in the context of a small company with at most 10 employees. Large companies probably have 10 people on the security team tasked only with this topic, thus perhaps this problem is solved in more complex ways.

So, how do the administrators and developers in this community manage such secrets when writing semi-automatic scripts?

I’m interested especially on the following aspects:

where do you store the secrets? in plain-text files, protected with GPG or Age, or even Google Drive?
how do you share the secrets between employees? just throw the sensitive material in Slack or email? (can one actually safely remove something from Slack or GMail?)
how do you dispose of the secrets once they aren’t needed (or have been exposed)? (because if they are in Git, good-luck with that…)
how do you keep track of what credentials are still required and in use? grep-ing the scripts code for random looking tokens?
how do you know who has access to what?
how often, and how do you actually rotate such secrets? (I admit I don’t want to answer this question…)
how do you actually provide it to the scripts? via environment variables? via command line arguments? via world-readable files in /tmp?
is anyone actually using any of the PKCS#11-related tools out there?
what if you need to pass such a secret to a tool that runs on a remote system which you access via SSH? do you just scp it to /tmp?
do you use any existing tool for this purpose? or is everything built in house?

Sorry for the lengthy question, but I do wonder about this for some time…

6
cetera edited 2 months ago | link
I have a bone to pick with secret management strategies.

IMO there are two options for secret management: either use a hardware key as your root of trust for everything, or lie to yourself. No one does the former perfectly, so everyone’s doing the latter.

Developers are constantly executing arbitrary un-sandboxed code on their computers due to dependencies, so you can just assume any file and any block of memory on your computer can be stolen without you knowing. As a result, we can conclude:
- Putting keys in files that decrypt other files is pointless
- Using a local password manager that decrypts a local vault into memory is pointless
- Using a cloud-based password manager that uses browser cookies (unencrypted files stored on disk) to authenticate you is pointless
- Using short-lived tokens that refresh (e.g. OIDC/AWS STS) is pointless (think an attacker can only strike outside of 9-5 when you’re logged out?)
Any combination of those tools still depends on some root of trust that an attacker can exfil. The only solution is a key stored in hardware that physically can’t be exfil’d. Use something like PKCS11 to send challenges to your hardware key to access everything.

This assumes a perfect world where you have no legacy systems of course, but I’m shocked at the amount of engineering time thrown at alternate solutions that just add complexity.
1. 3
  
  ciprian_craciun 2 months ago | link
  
  In absolute terms you might be right: everything is broken in one way or another outside (hopefully?) hardware keys. However that doesn’t mean we should all just give up and create a large PASSWORDS.txt file on the desktop.
  
  Every additional step and measure might make the life harder for the attacker. (Hopefully harder than it makes life for us.) :)
  1. 1
    
    cetera 2 months ago | link
    
    To clarify, I’m not advocating an all or nothing approach; more just complaining that if every company who built all the half-baked secret management systems mentioned in this thread had been using their brains, we would have actually secure secrets by now.
    1. 4
      
      ciprian_craciun 2 months ago | link
      
      I completely agree on this. For example I bought in 2012 two ePass2003 RSA tokens (2048 bit max) and wanted (at that time) to use them for TLS authentication plus SSH and GnuPG. I failed because the support (in Linux) was quite unreliable and I didn’t even manage to have it working with GnuPG. Thus I’ve shelved them.
      
      At the end of 2022 (10 years later, and still the “current” offer from Feitian) I took them for a ride hoping things have changed… Nop… I though that at least I can use them in my own custom tools via PKCS#11 interface… Nop… Just looking at the documentation made me dizzy… :)
      
      So yeah… If all these companies would have just spent 10% of the time and effort to actually make PKCS#11 compatible tokens work the story would have been quite different… (Or just come up with a working alternative, but standardize on something and support it long term…)
      
      And looking at the whole YubiKey tokens ecosystem doesn’t give me much hope… Sure there is https://github.com/str4d/age-plugin-yubikey but that’s not enough…
2. 2
  
  pointlessone 2 months ago | link
  
  Absolute security in the presence of an adversary is unattainable but what happens more often is credentials leak. Someone committed a hardcoded API key. Some script dumped its env into a logfile for debugging and it ended up full of secrets. A service has access to more credentials than it needs and used the wrong one to access (a wrong) db or a production API instead of staging. Those kind of things.
3

grahamc 2 months ago | link

At Determinate Systems, we use vault for secrets. HCP Vault is fine. We don’t share secrets, we get them from Vault and store them in the environment as needed. They all expire on a regular basis (maximum duration of ~12h.) Users get credentials based on their role / group membership in our Google org. All our access is gated on roles:

I can “become” the “appX_developer” role, which allows me to get credentials for appX. Without that step, though, I can’t get appX credentials. Each of these steps creates audit logs.

Vault feels heavy to start with, but it is really a wonderful tool and the documentation is very good.
1. 1
  
  ciprian_craciun 2 months ago | link
  
  The problem with HashiCorp’s Vault (and I’m sure there are other tools like it) is that it works in a centralized environment: there is “the vault server”, and to get there you need to be either to be on premise or VPN in. If your internet is spotty or you are traveling – which seems to be happen more often than not with smaller startups – I don’t think will work as nice.
  
  Moreover, I bet everyone that uses Vault still has a “master password list” tucked away somewhere, because if Vault goes down (bug, infrastructure issue, etc.) and there is an emergency and you need to get into your infrastructure then what do you do? :)
  1. 1
    
    grahamc 2 months ago | link
    
    Great questions.
    
    Our Vault is actually open to the internet, though using something like tailscale ephemeral tokens makes it a lot easier to lock that down like that.
    
    We don’t have a master password list. What we can do is log in to AWS’s console using our Google accounts and get AWS tokens to go poking around. We still don’t get long-term tokens through that.
    
    Anyway, I’m not sure these are “the problem” with Vault so much as an inherent tension of secret management where you need to make choices.
3

usrme edited 2 months ago | link

This depends on the type of secret, but for infrastructure we have a central repository to house all of our infrastructure-specific secrets (e.g. the administrative password to get into some SaaS offering that marketing uses, back-up passphrases, etc.) and we use Mozilla’s SOPS for state, which hooks in nicely with Google for authorizing the people who can even decrypt the main file. SOPS also plays delightfully with Ansible, which is a huge plus in my eyes as well as it then loses any friction that may normally come with working with secrets.

To share secrets between employees if they aren’t in our team (e.g. “hey, here’s the reset password for service X”) we set up our own instance of Password Pusher and it’s been great as it takes care of deleting the secret after a set (customizable) period of time. Our practice (albeit not a good one) prior to Password Pusher was to send something in Slack and as soon as the other person gave the green light we deleted the original message containing the secret. We had no allusions that this would actually wipe it from Slack’s nooks and crannies though.
2

kellogh 2 months ago | link

1password has a CLI
1. 1
  
  ciprian_craciun 2 months ago | link
  
  Interesting tool; it mostly focuses on environment variables and template files. I’ve seen a similar one from Mozilla, https://github.com/mozilla/sops.
  
  However both are what I would call file-system leaky… These tools might be wonderful for developer onboarding, by providing a way to bootstrap one’s credentials, but afterwards the resulting files (based on the templates) remain on the file-system. And with today’s Btrfs’s and ZFS’s once it reached the file-system, it will “forever” be leaked…
  1. 1
    
    st3fan edited 2 months ago | link
    
    What would leak?
    1. 1
      
      ciprian_craciun 2 months ago | link
      
      Let’s take their example from https://developer.1password.com/docs/cli/secrets-config-files/:
      
      ## config.yml.tpl database: host: http://localhost port: 5432 username: op://prod/mysql/username password: op://prod/mysql/password op inject -i config.yml.tpl -o config.yml
      
      Now your secrets are in plain on the disk inside config.yml. It might be excluded from Git, or you might delete it afterwards, but if the attacker gets into your laptop, he can just dd if=/dev/sda | tr -c ' -~' '\n' | grep -C 100 -F -e 'username:' and now he can start looking at older versions of these files.
      
      The only protection against this type of attack is to make sure that the config.yml is stored on a ramfs (which in Linux doesn’t leak in swap) or in tmpfs with encrypted swap.
      
      The safer alternative with this tool is to use the environments approach:
      
      op run --env-file="./prod.env" -- aws
  2. 1
    
    southclaws 2 months ago | link
    
    I loved Keybase for this, shame it’s on the green mile now.
    
    I could keep my credentials there and just tell my local development instance of whatever I’m working on to load its .env from K:\
2

southclaws edited 2 months ago | link

Don’t write scripts. Ever.

That’s my hard-line rule for teams and projects I’ve run. Scripts are, by nature, temporary. And there’s nothing more permanent than temporary.

My advice to people over the years has been to always build or use a product that solves your problem and treat it with the same practices you would other products in the domain. So, don’t write a deployment “script”, use a pre-existing tool that’s backed by a quality company/team (preferably written in a “proper language” not a “scripting language”.) Secrets have a very small surface area that they are allowed to exist in: GitHub Actions variables are one of those areas for example. Sticking to some simple, strongly enforced rules like this means you’ve only got a couple of places where secrets are accessible from.

Oh and cycle credentials frequently, make them expire, force people to get into the habit of updating configurations with the new creds if needed, treat it like a fire drill.
1. 6
  
  ciprian_craciun 2 months ago | link
  
  Don’t write scripts. Ever.
  
  Although I would really like to agree with you, it’s just practically impossible…
  
  My advice to people over the years has been to always build or use a product that solves your problem and treat it with the same practices you would other products in the domain.
  
  Well, a script matches perfectly this description: someone has built an in-house product to solve one’s problem; it just happens that the developer has chosen bash to do it and not say Go. Certainly the developer, just like the other professional tool developers, has treated this in-house product with the up-most care and attention. :)
  
  Getting serious, regardless of how many pre-existing tools one uses, at one point one would definitively want to automate some chaining of some tools that don’t support it out-of-the-box. Then what? Write another tool?
  
  Oh and cycle credentials frequently, make them expire, force people to get into the habit of updating configurations with the new creds if needed, treat it like a fire drill.
  
  And this is exactly part of the question I was asking: how does one “update configurations” with new credentials? By hand? With some other tool?
  1. 2
    
    southclaws 2 months ago | link
    
    Although I would really like to agree with you, it’s just practically impossible… In my experience I’ve not found this to be particularly difficult, but YMMV.
    
    My point was more about the mental model of what a “script” is - you can write a “script” in any language, it just has vibes of hackyness and carelessness. I’ve written scripts in Go that ignore all errors because they’re use-once. I’ve also encountered these kinds of Go programs in-the-wild doing important tasks.
    
    It’s about practices not language/tools. When I say “don’t write scripts” I’m really saying don’t solve problems as if they’re one-time-thing “I just gotta quickly automate this” scripts. If you treat the underlying problem as a product problem that’s a permanent fixture of your environment, you can build a proper solution not a hacky script.
    
    As for updating config, I generally do that manually and ensure many people on the team also know how and when to do it. But I’ve also not worked at real large scale so that probably changes once you hit hundreds of services running globally, and this is the point where my experience ends.
1
robertmeta 2 months ago | link
Your question really covers two different tiers of things. For things were there is any way to avoid using a shared secret, avoid it. With AWS and like https://github.com/99designs/aws-vault – the secret is stored in OS protected space (which has whole teams of engineers securing it) and you get temporary creds, which obviously solves basically all your concerns.

But, the real issue is with shared keys you have no way to vary per developer. The first protection is don’t ever let the developers see the production keys. They live somewhere in your AWS Secrets (both in prod and dev) – but the developers can only ever see the dev keys, and devops can see the prod keys. Far from perfect but helpful in limiting damage scope.

where do you store the secrets? in plain-text files, protected with GPG or Age, or even Google Drive?
- In local OS secured space or Cloud Secrets space.
how do you share the secrets between employees? just throw the sensitive material in Slack or email? (can one actually safely remove something from Slack or GMail?)
- You either don’t or do it (per employee creds) or via a lookup to a secrets share at runtime.
how do you dispose of the secrets once they aren’t needed (or have been exposed)? (because if they are in Git, good-luck with that…)
- Log into secrets manager, rotate it, done. If at employee level, they rotate them.
how do you keep track of what credentials are still required and in use? grep-ing the scripts code for random looking tokens?
- Audits. Secrets are in AWS Secrets or a part of your IAM role letting you generate them.
how do you know who has access to what?
- Audits again, same technique.
how often, and how do you actually rotate such secrets? (I admit I don’t want to answer this question…)
- You just rotate them.
how do you actually provide it to the scripts? via environment variables? via command line arguments? via world-readable files in /tmp?
- They either come in via env variable in the better setup, or in a pure shared secret situation they are pulled in from the cloud secret provider directly via the script.
is anyone actually using any of the PKCS#11-related tools out there?
- No clue.
what if you need to pass such a secret to a tool that runs on a remote system which you access via SSH? do you just scp it to /tmp?
- Again, you either use your role (IAM) on remote to build temp ones for env or they get it out of secret store that they have access to via a role on that machine.
do you use any existing tool for this purpose? or is everything built in house?
- Building such tools gets very complex, very fast. Use open source ones like awsvault.
1. 1
  
  ciprian_craciun 2 months ago | link
  
  Indeed with AWS that provides a way to work with temporary credentials (are there other cloud providers that do this?) the solution you’ve highlighted might work. But 99% of the other providers just give you a random “secret API key” and are done with it…
  1. 1
    
    robertmeta 2 months ago | link
    
    The major players have similar solutions. But, speaking to the situation where you don’t have such a system – you can still leverage a secrets store. Gives you a single place to manage secrets, gives you a single place to kill / rotate creds and can keep your developers from ever seeing the production keys.
1

posix_cowboy 2 months ago | link

I’ve been using an encrypted JSON file to hold credentials. The script prompts for a passphrase, then uses that to decrypt and parse the json file.
1

jorgelbg 2 months ago | link

Nothing fancy: Locally I’m using a combination of gopass + direnv. I use this setup for secrets that I generate or (more likely) I am going to be the only consumer. For secrets that are meant to be used by the rest of my team we rely on vault.

direnv allows me to just automatically have my secrets available when I’m in the repository for a given project as environment variables. gopass usually requests me to confirm the loading of the secrets either via Touch ID or with a Yubikey.

Since my team mostly deploys into our Nomad cluster, we rely on vault for making those secrets available in our containers.
1

threkk 2 months ago | link

For my personal projects I use a combination of sops and age. I decrypt and copy the secrets I need for the project to an .env file, and once I am done I delete it.
1
jackdk 2 months ago | link
Some AWS-specific points, though I assume other cloud providers do similar things:
- You can use OIDC with many CI providers to get temporary credentials for each CI run. The CI runner will give your job a token that it can use with sts:AssumeRoleWithWebIdentity to get short-lived credentials. This means that if an attacker runs code on CI, they can’t exfiltrate keys and use them forever, but of course they can still do a lot of damage.
- The sts:AssumeRole API can take a TOTP code; the AWS CLI and other tools can use this to grant temporary credentials only when you provide a 2FA token. This limits the amount of havoc that can be caused from a lost set of developer AWS Keys.