1. 52
  1. 9

    You should rotate ssh keys. Also, all ssh keys should be protected with a passphrase. To solve this problem, yet maintain your convenience, Consider generating your ssh credentials on yubikey or similar device. According to Yubico, it’s impossible to exfiltrate these private key from the yubikey. While I don’t believe that anything is impossible, I’m convinced that exfiltrating these keys is beyond the abilities of anyone short of governments.

    The basic steps for using a yubikey for ssh authentication are:

    • Install and setup the full gnupg package including all smart card utilities on your laptop or from whence you plan to originating ssh.
    • Make sure to setup gnupg as your ssh agent.
    • Use the gpg card setup commands to generate a key on the card.
    • Use ssh-add -L | grep card to get your public key.
    • Install that key on the target systems where you need to have ssh sessions.
    • Fix your .profile on your laptop so that gpg-agent is always available.

    In practice you will want to have two yubikeys. I have a low profile one that’s always there and convenient and a backup stored in a safe place. When you attempt to setup an ssh session, gpg-agent will prompt you for the yubikey’s pin code to unlock the key for a configurable amount of time.

    Passphrase or not, the problem with ssh private keys in files is that they are non-appropriable. That means that an ssh token stored in a file can be copied and used by someone else with or without your permission. Since a rsa private key on a yubikey device can’t be exfiltrated, the only way for someone to appropriate it is would be to take possession of the yubikey. You can assume that a yubikey that’s gone missing has been compromised and you can invalidate the public keys associated with it.

    Finally, Openssh also allows you to generate certificates for authentication. While I prefer Yubikeys and manual rotation myself, ssh certificates should be considered because they are self expiring.

    1. 4

      I settled on something a bit more convenient (IMO), but not as secure. I generate all my ssh private keys with passphrases. I store both the keys and the passphrases in pass, which is encrypted with my GPG key. I have a hodge podge of shell scripts that will add my ssh keys to my agent by only typing the password for my GPG key. It’s convenient and avoids using unprotected ssh keys. I guess the downside is that I’ve put all my eggs in one basket. But it’s good enough for my personal threat model. For a business, I’d do something stronger.

      1. 1

        I take an alternative approach with a similar result: generating SSH keys that are tied to a FIDO device (such as a Yubikey). This is natively supported in OpenSSH; no gpg necessary. Without the hardware, exfiltration is useless—a much nicer situation than described in the article.

        I also keep SSH certificates on all my machines, clients and servers, with an expiration date. As part of my normal setup process for any new SSH sever—disabling password logins and enforcing keypair auth only—I also pull off the host keys to sign with the CA, and enable a cronjob to fetch a newly signed certificate for expiration. And whenever I generate keypairs on a new client, I specify my trusted host CA at the same time. This has the benefit of completely eliminating host key warnings; I haven’t seen one on my own servers since I started this practice two years ago.

      2. 7

        A good start for people using static keys (not SSH certs) is to use keys per source host, and to keep a logbook of all the places that need to be updated when the key changes. This will tell you … a lot.

        Also, understand your threat model: are you using SSH agent forwarding? By default? To boxes out on the Internet? I see a lot of people who forward agents to bastion hosts and then ssh from the bastion onwards, meaning that a compromise of the bastion (or malicious admin there) can connect onwards as you while you’re connected, typically without prompting you (confirm-on-use is available for SSH but anyone who does bulk connections (ansible etc) won’t enable that). Just using ProxyJump in newer OpenSSH and NEVER forwarding the agent unless you absolutely have to will win here.

        Modern OpenSSH has Ed25519 on yubikeys via the sk-ssh-ed25519@openssh.com type; figuring out high-value targets vs regular usage and only allowing touch-key-to-continue usage can work great (up until aforementioned parallel usage via ansible).

        If you work from a personal device, make sure you have different SSH keys for work and personal use, because any key you use for work might belong to your employer and the private key might be disclosable upon demand. Rather than worry about if this is valid or not, or what the likelihood is, just use a different key for work vs personal. You can typically have up to 6 keys tried in a connection before it fails. This way, if your employer ever does demand this, you won’t give up access to personal systems.

        Consider what will happen if there ever is a crypto break and strongly consider using two different algorithms for your current set of keys, and treat a bundle of keys from the same host as equivalent. That way, if a crypto break means folks have to mass-deploy an PubkeyAcceptedAlgorithms / PubkeyAcceptedKeyTypes sshd config change, you won’t get locked out. Don’t demand this of everyone, it will just confuse many, but as long as enough key (!) employees do this, you have business continuity.

        Consider separate keys for Prod vs Code. Sooner or later, you’ll find that some kind of prod access does require forwarding an agent somewhere less than ideal. A compromise should not grant access to all the crown jewels of your company. A broken VCS server with people using agent-forwarding should not grant access to prod. Note that this also then means separate keyrings to actually have protection, since any key in the agent is available for signing operations for remote hosts. I’m using an overly complicated setup via an ssh-role shim script (and ssh shim wrapper) to support parallel agents started via a systemd template. So I have xyz-code and xyz-prod agents for an employer xyz. It’s not perfect but if you have admin access to “a lot” then it’s worth taking care to limit blast radius of an event.

        And then you get into SSH Certificate Authorities, which are orthogonal to every issue above except the work vs personal split.

        1. 1

          Also, understand your threat model: are you using SSH agent forwarding? By default? To boxes out on the Internet? I see a lot of people who forward agents to bastion hosts and then ssh from the bastion onwards, meaning that a compromise of the bastion (or malicious admin there) can connect onwards as you while you’re connected,

          I’ve always assumed that a malicious admin on the bastion could extract and reconstruct keys from the forwarded agent - not merely make use of them to connect on while the forwarding is active,

          I’m not sure how wise it is to trust keys, even with passphrases on machines with an untrusted admin. For example, I use passwords rather than keys to log on to my home system from work on the assumption that it’d be harder to exploit. The admins at work, being people that actually know me are probably especially untrustworthy because they might have reason to want to get onto my systems. If I used an ssh key they would at least have easy access to the encrypted key file.

          1. 5

            I’ve always assumed that a malicious admin on the bastion could extract and reconstruct keys from the forwarded agent - not merely make use of them to connect on while the forwarding is active,

            No. This is precisely what ssh-agent is designed to avoid. The only way that they could do this is if they were able to break RSA (or whatever public key algorithm SSH is using in a particular instance). The ssh-agent protocol is very simple. The intermediary just forwards the signing request to the host that owns the key and then passes the response back. To be able to extract the key, it would have to be able to reconstruct a private key from some known (or chosen) plaintext, the public key, and the cyphertext. If an attacker can do that, then they can reconstruct ssh keys from observing the cypertext of any ssh session.

            1. 2

              This feels… all wrong, and yet I can’t prove that it is. At least, with the searching I’ve done up to this point.

              That’s bad in and of itself. There surely is a good document detailing the SSH threat model somewhere.

          2. 6

            My 13-year-old son pointed out to me that calling this “rotating” your keys makes it sound like a really bad idea; you should replace your keys, not “rotate” them thru your systems! It’s a weird bit of terminology.

            1. 1

              Ever since I first heard it used, I imagined a wheel of sorts, or maybe something related to crew rotation.

              I have no idea where crew rotations come from, they’re not kebab, but computing has other similar terms like deployment configuration. Maybe in IT they’re not originally about the deployment and configuration of a fleet of ships, but maybe they are!

              Anyone know, as I couldn’t find anything quickly enough for this?

            2. 4

              Wait, everyone doesn’t add passphrases to their SSH keys & use their SSH agent to not have to type it in all the time?! Yeah that’s a disaster waiting to happen. Add passphrases to your keys! Use ssh-add!

              1. 2

                Here is Pinterest’s 2016 write up of similar solutions to Knox, their open-source key storage tool: https://github.com/pinterest/knox/wiki/Similar-Solutions

                1. 2

                  In the past I’ve been guilty of having a single all-purpose key. Last year I started reducing the scope of my keys. I now have a key for each combination of source computer, destination system and year. When I replace a laptop then I’ll be able to invalidate all my keys from that source.

                  1. 2

                    The author has a valid point on rotation - the threat model that applies to passwords, applies to ssh keys as well.

                    1. 2

                      Keeps me wonder why no-one ever talks about rotating ssh host keys, as most people see it as something that should never change - hence I can’t even remember GitHub or GitLab updating their keys…

                      1. 1

                        Because unless you have a model to manage deployment of systemwide known-hosts files, or are using an ssh certificate authority for host keys, or use DNSSEC-signed DNS and all clients have validating resolvers and configure clients to trust DNS over local TOFU cache, unless you have one of those … the TOFU store (~/.ssh/known_hosts by default) will mean that rotating host keys causes pain.

                        This is probably why GitHub is still on RSA+DSA, with no Ed25519 host-keys: the act of adding a new keytype will cause real-world breakage for many clients, many of whom won’t be aware that they’re using SSH under the hood. They’re accumulating technical debt by being unable to change and not working with their client community to manage a process to update with minimal disruption.

                        1. 2

                          Adding an Ed25519 wouldn’t be breaking though, just dropping an old key.

                          And actual rotation is a bit easier since UpdateHostKeys was introduced, but you still need to keep the old key around long enough for a sufficient amount of clients to have connected and updated. What amount is “sufficient” for any given use case is harder to determine though.

                          1. 1

                            Different versions of different SSH clients handle this with varying levels of grace.

                            In some versions of OpenSSH, adding a new key of a preferred type would immediately try to verify that host-key, ignoring the known ones, and hard-fail because it wasn’t already known but host keys of other types were.

                            Now look at how long some OS distributions linger for, with real-world users using ancient OpenSSH with backported features.

                            That’s the problem which services like GitHub have to consider: the end-users who are best able to understand the issue and adapt with help are the least likely to be stuck using ancient OS releases.

                      2. 1

                        I’d like to ask for an informal survey here…

                        • For those that use them, what have your experiences been with key management systems? [1]
                        • What were your technical and organizational drivers? How satisfied are you with what you (your org) picked?

                        [1] https://en.wikipedia.org/wiki/Key_management#Management_and_compliance_systems

                        1. 2

                          Can’t say much, and I have no idea about how alternatives hold up, but when I was paid to care, SSH Corporation’s UKM did its job well enough.

                          I do think certs are a better fit for many, but if you need that UKM-type of key management, my experience says to look into it.

                        2. 1

                          I confess that I do it whenever I get a new hard drive/do a new OS install and have to generate a new key to replace one lost. The slight downside to this is that I recently realized that my main terminal server still would, in theory, accept SSH logins from my account on a particular computer in a particular lab at my grad school.