It could be worse, the user has the old rsa host key present alongside newer ed / ecdsa keys, they may never rotate out the rsa one. A future mitm simply only advertises the rsa key, and mitm passes.
Users will need to actively remove the old rsa key in order to be safe.
Okay, I tested this and on a new enough OpenSSH client, the RSA key gets replaced using the mechanism described here: https://lwn.net/Articles/637156/ (if you connect using a key other than RSA).
But this is also a very bad thing in the other direction. If you are MITM’d first, they can “update” the other keys if you connect with RSA first, right?
I mean, it doesn’t really make any difference? The only situation where this makes any difference is if the user takes no action to update the keys manually, and in that case the MITM will continue as long as the man is in the middle, whether this mechanism exists or not. And then once you connect to actual GitHub, the fact that you got MITMed will be more noticeable.
Call this SRE cynicism, but there’s usually a long tail of builds that you do at various companies. Some of those builds may not run very often, and somebody may have pinned the GitHub RSA key thinking that it would never change. It’s going to be interesting to see with the long tail of less often run builds is at some companies.
I see your point and nice mention of the long tail. Things will likely be discovered and break like you said. I’d call this failing safe 🤷🏻♂️. If I revoked an SSL cert with a Certificate Revocation List (CRL), things might break (if clients don’t read the CRL). But I revoked the cert. This is what I wanted. The problem with SSH keys (as mentioned here by df) is that SSH keys don’t have an expiration and there’s no CRL.
I know how I should rotate passwords and the impulse could be my password leaked out on my Twitch stream (or something). A leak is possible and I know how to rotate a password. I know what the impact is (my old password stops working). I know how to rotate an SSL cert. There’s an automated way to do it with certbot. I might do it every week. A leak of SSH keys is possible so what is the procedure? It’s this very story. There is no other option. There is no CRL.
So in this way, SSH has no explicit mechanism because … it’s a circle of trust for servers and hostnames to me that is unrelated to a terminal user on Github. I mean, I don’t really get an interactive shell on Github but this is what SSH started out as. I can have a git repo on a normal Linux box but I am granted a user account. This is different (in this context) than the HTTPS access method on Github for the same feature (Github access).
It’s similar to the TLS circle of trust but there’s no private key infrastructure by itself. So it’s nice how simple SSH is (no registrars, no fees, no PKI, no CRLs, no expiration) but … how do you rotate an SSH key like you would a password or a cert? You do it because you want to fail safe but you make the news. 🌻
This is an argument for using HTTPS whenever possible. It’s not perfect, but there’s an established process for key revocation and rotation, and also there are good ways for large-scale services to use those with an HSM to make accidentally leaking the private key much harder. There’s a reason we don’t see private key leaks with normal websites very often.
I see the point. OTOH, using http auth with github requires downloading some piece of proprietary (or otherwise GitHub branded) software since they disabled password auth. So currently, the only way to authenticate with GitHub by using only standard git tooling is SSH.
As yetanotherjosh mentions on the orange site, please take a moment to verify the SHA256 of the new key after you remove the old one and try connecting to github.com again.
$ ssh -o VerifyHostKeyDNS=yes github.com
The authenticity of host 'github.com (140.82.121.3)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
No matching host key fingerprint found in DNS.
No.
SSHFP is a flawed design. The idea here is that it is protected by DNSSEC. However DNSSEC only protects the connection between the authoritative nameserver and the resolver. Unless you’re using ssh on a server that also runs its own DNS resolver, the SSHFP record will not be protected from man in the middle attacks, and is therefore not a security mechanism.
However DNSSEC only protects the connection between the authoritative nameserver and the resolver.
It doesn’t have to be that way. In major linux distributions, resolv.conf is managed which systemd-resolved, and it proxies DNS records after validate DNSSEC records, before handing them over (locally) to the libc or the application reading looking up the resolver address in resolv.conf.
So yes, in the hold times clients were not protected by DNSSEC, but since 2-3 years, in Debian/Fedora/(most distro with systemd) DNSSEC is enabled, thus increasing security.
Please note that this issue was not the result of a compromise of any GitHub systems or customer information. Instead, the exposure was the result of what we believe to be an inadvertent publishing of private information.
It was in a public repo, so might have been downloaded. After that, there’s no way of knowing if it ended up in the hands of anyone malicious. If someone is able to fake your GitHub DNS reply (e.g. on a public WiFi network) then they can redirect you to a server that will accept your git pushes to private repos.
As raggi points out on the orange website:
Okay, I tested this and on a new enough OpenSSH client, the RSA key gets replaced using the mechanism described here: https://lwn.net/Articles/637156/ (if you connect using a key other than RSA).
But this is also a very bad thing in the other direction. If you are MITM’d first, they can “update” the other keys if you connect with RSA first, right?
I mean, it doesn’t really make any difference? The only situation where this makes any difference is if the user takes no action to update the keys manually, and in that case the MITM will continue as long as the man is in the middle, whether this mechanism exists or not. And then once you connect to actual GitHub, the fact that you got MITMed will be more noticeable.
This is gonna be a huge pain to deal with for years. I understand why they rotated it, but I almost wish they didn’t.
Why do you think this will last years?
Call this SRE cynicism, but there’s usually a long tail of builds that you do at various companies. Some of those builds may not run very often, and somebody may have pinned the GitHub RSA key thinking that it would never change. It’s going to be interesting to see with the long tail of less often run builds is at some companies.
now that i’ve been doing infra work in a non-hobbyist capacity for literally any amount of time i don’t call SREs cynical anymore, just realistic.
I see your point and nice mention of the long tail. Things will likely be discovered and break like you said. I’d call this failing safe 🤷🏻♂️. If I revoked an SSL cert with a Certificate Revocation List (CRL), things might break (if clients don’t read the CRL). But I revoked the cert. This is what I wanted. The problem with SSH keys (as mentioned here by
df) is that SSH keys don’t have an expiration and there’s no CRL.I know how I should rotate passwords and the impulse could be my password leaked out on my Twitch stream (or something). A leak is possible and I know how to rotate a password. I know what the impact is (my old password stops working). I know how to rotate an SSL cert. There’s an automated way to do it with certbot. I might do it every week. A leak of SSH keys is possible so what is the procedure? It’s this very story. There is no other option. There is no CRL.
So in this way, SSH has no explicit mechanism because … it’s a circle of trust for servers and hostnames to me that is unrelated to a terminal user on Github. I mean, I don’t really get an interactive shell on Github but this is what SSH started out as. I can have a git repo on a normal Linux box but I am granted a user account. This is different (in this context) than the HTTPS access method on Github for the same feature (Github access).
It’s similar to the TLS circle of trust but there’s no private key infrastructure by itself. So it’s nice how simple SSH is (no registrars, no fees, no PKI, no CRLs, no expiration) but … how do you rotate an SSH key like you would a password or a cert? You do it because you want to fail safe but you make the news. 🌻
Sure there is another way https://cybervillains.com/@djm/110077675622139561
This is an argument for using HTTPS whenever possible. It’s not perfect, but there’s an established process for key revocation and rotation, and also there are good ways for large-scale services to use those with an HSM to make accidentally leaking the private key much harder. There’s a reason we don’t see private key leaks with normal websites very often.
SSH servers can store the host keys in a HSM as well, FWIW. It seems like they just didn’t do that here.
I see the point. OTOH, using http auth with github requires downloading some piece of proprietary (or otherwise GitHub branded) software since they disabled password auth. So currently, the only way to authenticate with GitHub by using only standard git tooling is SSH.
It’s also possible to use multiple private keys, as long as they were signed for your domain, you don’t have to have that one key on every server
As yetanotherjosh mentions on the orange site, please take a moment to verify the SHA256 of the new key after you remove the old one and try connecting to github.com again.
You can also just copy and paste their keys from the docs
They didn’t update their DNS SSHFP records ?
What a security nightmare…
I have to admit, until your comment I had never even heard of SSHFP records, much less the need to update them.
Or the change just didn’t propagate yet?
This is the exact comment they posted on the orange website, to which someone already replied that
And that operation is nonsensical anyway as that’s the ED25519 key, which has not changed (the RSA key was the compromised one).
The lack of SSHFP records in the first place should be concerning :/
No. SSHFP is a flawed design. The idea here is that it is protected by DNSSEC. However DNSSEC only protects the connection between the authoritative nameserver and the resolver. Unless you’re using ssh on a server that also runs its own DNS resolver, the SSHFP record will not be protected from man in the middle attacks, and is therefore not a security mechanism.
It doesn’t have to be that way. In major linux distributions,
resolv.confis managed which systemd-resolved, and it proxies DNS records after validate DNSSEC records, before handing them over (locally) to the libc or the application reading looking up the resolver address inresolv.conf.So yes, in the hold times clients were not protected by DNSSEC, but since 2-3 years, in Debian/Fedora/(most distro with systemd) DNSSEC is enabled, thus increasing security.
Not true. systemd-resolved checks DNSSEC, but doesn’t enforce it by default. It has no security value unless you manually configure it differently. See: https://www.freedesktop.org/software/systemd/man/resolved.conf.html
“DNSSEC= […] Defaults to “allow-downgrade”.”
If your server is not running its own validating resolver, then your server is wrong
It is irrelevant what the server does. The key gets validated on the client.
I was replying to this. You should indeed be doing this.
If what you meant was “maybe you’re using ssh not on a server” then sure, but you can/should also run a validating resolver for your local internet.
I try not to find the matters of DNS records raise my adrenaline levels no matter my personal preference.
How can exposing a private key not be compromise?
Because it was presumably a mistake by someone working at Github, and not an external party deliberately trying to get hold of the private key.
To be clear, the exposing of a key itself was a compromise, but it was not the result of a separate compromise.
It was in a public repo, so might have been downloaded. After that, there’s no way of knowing if it ended up in the hands of anyone malicious. If someone is able to fake your GitHub DNS reply (e.g. on a public WiFi network) then they can redirect you to a server that will accept your git pushes to private repos.
Shame they didn’t enable secret scanning on that repo