There was a site called shhgit that continuously streamed credentials that were accidentally checked into Github… it was amazing, there was a near-continuous stream of credentials showing up for everything in existence. I just checked and the site is now a pricy enterprise service!
I’m fairly sure GitHub does some monitoring on its own as well by the way; a while ago I committed a bogus example token and I got an email from GitHub telling me about it.
I wonder how these attackers manage to find these secrets so quickly. Sounds like some pretty hefty resources behind this.
Wasn’t there a blog post some years ago from github working together with google cloud etc to revoke such tokens on sight ? (Which itself also seems bad because youre whole system might come down that second)
Some years ago, a client experienced this. A contractor accidentally uploaded some AWS keys to their personal Github account while (legitimately!) blogging about work they’d done for the client.
Anyway, what you’d expect happened, happened: Amazon contacted us to say they’d discovered keys in the wild, we rotated the keys in question, and checked that nothing untoward had happened in the account. We also contacted the contractor, who was mortified and highly apologetic.
The interesting thing was that on the morning of the leak, the client’s head of IT received a phone call from someone purporting to be from Amazon, offering assistance with the incident. The call was very polished and professional sounding, but utterly bogus - it was an attempt to get said head of IT to provide our root account credentials o_O
I got mugged a while ago, and they took my phone. As soon as I got home, I triggered a remote lock and put my phone number on it in case it somehow found its way into the hands of an honest person.
I didn’t get any offers to get my phone back, but I did get a lot of helpful messages saying that Apple recovered my phone and I needed to go to apple.recovery.biz or similar to prove it was mine. I doubt that the people that stole the phone were running that site; I always wondered what the economy for that looks like. Do they sell it for cheap to someone who sends out the phishing messages in hopes that they can get my credentials, unlock it, and sell it as an actual phone and not just parts?
Many years ago, I was using a vim plugin to to gist buffers. At some point, I must have changed the default behaviour from private to public. I was working with CloudFormation and the huge JSON files were not rendering properly, so I was using github to double-check my linter worked properly. As you can imagine there were credentials there, so … my credentials were leaked. Nearly 60s after the gist, we were notified by github. An email org’s email and quickly we rotated the creds.
About 1.5 years later a new hire, found the leaked file on another website that was copying and storying public gists. The website was behind cloudflare. I contacted the owner of the website to bring down the gist, mentioning GDPR. But not reply. The website was most likely abandoned. GDPR had just entered into action, so I sent a GDPR at cloudflare. The reply borderline ridiculous: “We can’t bring down your gist, because our systems don’t work like that blah blah blah”. We exchanged few emails, so I got tired and gave them a notice period of five days to bring down the website or else I’ll be contacting the German GDPR authority, explaining the website was behind cloudflare. Two days later the website, magically, came down. They sent no notice to me, I just went to visit the website and there was nothing there. Perfect timing? Maybe.
I also use GDPR on shitty companies. For example Atlassian. I had a registration that got interrupted half way through and I could not continue or cancel the process. I reached out if support who did not believe me even though I sent them screenshots. After a while I got fed up and sent them a GDPR delete requests and after it went through I could finally register with that email. I plan to do the same with Google snd a Gmail address that is stuck sort of same way. I cannot change the phone number and the 2fa. Theres is no gmail support.
I don’t understand. If you had already rotated the credentials 1.5 years ago, why did you care that there was a site out there that was displaying your old credentials?
Was it that you wanted to hide even the existence of some the services you were running?
This makes me want to set up a honeypot ssh container and leak connection info on the internet where any ip which gets a shell and issues a command logged to https://complaint.ic3.gov/default.aspx
Credential leaks, along with compromised sites, are a good reason to invest in second-factor authentication (2FA).
Requiring that “something you have” as well as a password means that this sort of thing is more embarrassing than actually damaging.
This applies to “human” credentials, but I think in most of those cases people accidentally commit machine credentials, e.g. AWS Access Keys and their secrets, which from what I understand cannot use 2FA.
Well you could let your system provide a TOTP besides the API token, but you’ll have to provide that too somewhere in the configs - if you don’t want to have an interactive shell for that on every instance. So now you can leak an API key and the TOTP secret with your configuration files, double the fun.
Bravo! At our company it is a firing offense to use anything else than a credential file (~/.aws) or an instance profile. I also strongly discourage any FOSS developer to include examples including using credentials directly (not through a profile).
You can go even further and limit the scope of credentials to IP ranges which you know your developers are working from.
Been here done that… Exact same story. It was amazing how quickly the creds were picked up and put into use. While I was somewhat embarrassed by my incident, no actual harm was done and it serves as a great warning to others. Thanks for sharing your experience.
😌 it’s good to know it’s happened to others as well. Key management has been forever changed from something for other people in other situations to something that must be understood.
The irony with key/cred management is that it’s inherently a very private thing. I don’t care what blog posts there are about it, the people best at it, don’t publicize it.
Like i’m watching what I say right now. I might let an attack vector slip.
These are the lessons you have to learn on your own, and viscerally.
Akin to how you are better at unix after you “rm -rf *” in the wrong place once.
If creds are pushed they must be removed from use, and yesterday. You can’t reliably clean up history.
Before things go to prod you must ask how can undo any creds/keys/passwords fast.
This leads to questions like “can I stop payment and/or payment sources fast”.
In other words in some cases it’s already too late, so what are you second tier mitigation strategies.
Being agile/startup etc. does not take away this responsibility. It actually enhances it. You’re the targets.
I say that because non tech ppl with the money control might not want to hear about this in early stages.
But early stages is where this belongs.
Does anyone use honeytokens like https://github.com/thinkst/canarytokens? It seems like we could do a better job of always expecting this to happen and just include it as part of most infrastucture scaffolding.
There was a site called shhgit that continuously streamed credentials that were accidentally checked into Github… it was amazing, there was a near-continuous stream of credentials showing up for everything in existence. I just checked and the site is now a pricy enterprise service!
I’m fairly sure GitHub does some monitoring on its own as well by the way; a while ago I committed a bogus example token and I got an email from GitHub telling me about it.
I wonder how these attackers manage to find these secrets so quickly. Sounds like some pretty hefty resources behind this.
Wasn’t there a blog post some years ago from github working together with google cloud etc to revoke such tokens on sight ? (Which itself also seems bad because youre whole system might come down that second)
Some years ago, a client experienced this. A contractor accidentally uploaded some AWS keys to their personal Github account while (legitimately!) blogging about work they’d done for the client.
Anyway, what you’d expect happened, happened: Amazon contacted us to say they’d discovered keys in the wild, we rotated the keys in question, and checked that nothing untoward had happened in the account. We also contacted the contractor, who was mortified and highly apologetic.
The interesting thing was that on the morning of the leak, the client’s head of IT received a phone call from someone purporting to be from Amazon, offering assistance with the incident. The call was very polished and professional sounding, but utterly bogus - it was an attempt to get said head of IT to provide our root account credentials o_O
I got mugged a while ago, and they took my phone. As soon as I got home, I triggered a remote lock and put my phone number on it in case it somehow found its way into the hands of an honest person.
I didn’t get any offers to get my phone back, but I did get a lot of helpful messages saying that Apple recovered my phone and I needed to go to apple.recovery.biz or similar to prove it was mine. I doubt that the people that stole the phone were running that site; I always wondered what the economy for that looks like. Do they sell it for cheap to someone who sends out the phishing messages in hopes that they can get my credentials, unlock it, and sell it as an actual phone and not just parts?
Sorry to hear you got mugged, & that you didn’t recover your phone.
Fascinating about the ‘helpful’ messages, though. I wonder if it’d be worth an iPhone honeypot to try to track down the people doing this.
… which leads me to wonder whether the Police run operations like that, and if not, why not.
My story with GDPR and CloudFlare.
Many years ago, I was using a vim plugin to to gist buffers. At some point, I must have changed the default behaviour from private to public. I was working with CloudFormation and the huge JSON files were not rendering properly, so I was using github to double-check my linter worked properly. As you can imagine there were credentials there, so … my credentials were leaked. Nearly 60s after the gist, we were notified by github. An email org’s email and quickly we rotated the creds.
About 1.5 years later a new hire, found the leaked file on another website that was copying and storying public gists. The website was behind cloudflare. I contacted the owner of the website to bring down the gist, mentioning GDPR. But not reply. The website was most likely abandoned. GDPR had just entered into action, so I sent a GDPR at cloudflare. The reply borderline ridiculous: “We can’t bring down your gist, because our systems don’t work like that blah blah blah”. We exchanged few emails, so I got tired and gave them a notice period of five days to bring down the website or else I’ll be contacting the German GDPR authority, explaining the website was behind cloudflare. Two days later the website, magically, came down. They sent no notice to me, I just went to visit the website and there was nothing there. Perfect timing? Maybe.
It’s always fun how many things are only technically impossible until the threat of legal attention gets involved.
I also use GDPR on shitty companies. For example Atlassian. I had a registration that got interrupted half way through and I could not continue or cancel the process. I reached out if support who did not believe me even though I sent them screenshots. After a while I got fed up and sent them a GDPR delete requests and after it went through I could finally register with that email. I plan to do the same with Google snd a Gmail address that is stuck sort of same way. I cannot change the phone number and the 2fa. Theres is no gmail support.
I don’t understand. If you had already rotated the credentials 1.5 years ago, why did you care that there was a site out there that was displaying your old credentials?
Was it that you wanted to hide even the existence of some the services you were running?
The infra was way different by that time, wasn’t a big issue. I was embarrassed I suppose.
This makes me want to set up a honeypot ssh container and leak connection info on the internet where any ip which gets a shell and issues a command logged to https://complaint.ic3.gov/default.aspx
Credential leaks, along with compromised sites, are a good reason to invest in second-factor authentication (2FA). Requiring that “something you have” as well as a password means that this sort of thing is more embarrassing than actually damaging.
This applies to “human” credentials, but I think in most of those cases people accidentally commit machine credentials, e.g. AWS Access Keys and their secrets, which from what I understand cannot use 2FA.
Fair enough, that makes sense thanks!
Well you could let your system provide a TOTP besides the API token, but you’ll have to provide that too somewhere in the configs - if you don’t want to have an interactive shell for that on every instance. So now you can leak an API key and the TOTP secret with your configuration files, double the fun.
Bravo! At our company it is a firing offense to use anything else than a credential file (~/.aws) or an instance profile. I also strongly discourage any FOSS developer to include examples including using credentials directly (not through a profile).
You can go even further and limit the scope of credentials to IP ranges which you know your developers are working from.
Been here done that… Exact same story. It was amazing how quickly the creds were picked up and put into use. While I was somewhat embarrassed by my incident, no actual harm was done and it serves as a great warning to others. Thanks for sharing your experience.
😌 it’s good to know it’s happened to others as well. Key management has been forever changed from something for other people in other situations to something that must be understood.
I bet whoever watches Github for credentials like this makes a LOT of money. I wonder how many cloud providers they support?
The irony with key/cred management is that it’s inherently a very private thing. I don’t care what blog posts there are about it, the people best at it, don’t publicize it.
Like i’m watching what I say right now. I might let an attack vector slip.
These are the lessons you have to learn on your own, and viscerally. Akin to how you are better at unix after you “rm -rf *” in the wrong place once.
If creds are pushed they must be removed from use, and yesterday. You can’t reliably clean up history. Before things go to prod you must ask how can undo any creds/keys/passwords fast. This leads to questions like “can I stop payment and/or payment sources fast”.
In other words in some cases it’s already too late, so what are you second tier mitigation strategies.
Being agile/startup etc. does not take away this responsibility. It actually enhances it. You’re the targets. I say that because non tech ppl with the money control might not want to hear about this in early stages. But early stages is where this belongs.
(maybe leave some passwords in source control …)
@mthwsjc Just an FYI, small typographical error here:
Guessing it should be “leaked”, not “leaded”
I generate all my credentials unleaded. Better for the environment.
thanks
Does anyone use honeytokens like https://github.com/thinkst/canarytokens? It seems like we could do a better job of always expecting this to happen and just include it as part of most infrastucture scaffolding.
https://github.com/paralax/awesome-honeypots has a few that could probably be cheap/free to run on heroku or other free tier services.
This is a thing https://docs.github.com/en/code-security/secret-security/about-secret-scanning