1. 16
  1. 11

    TLDR: Never store private keys in git. Not even for private repositories.

    1. 11

      “Unlike the title of this post suggests, this is actually a problem with the GitHub extension that ships with Visual Stuido 2015. ”

      1. 1

        Correct, here’s the bug report. It’s a combination of a GitHub bug and the author’s process issue.

      2. 8

        Per the author’s conclusion, how did the miners continue to create instances after their keys were revoked?

        1. 1

          My guess would be that the attacker created a new IAM user with full access prior to the revocation, and that the author only revoked access keys on their own user.

        2. 8

          How putting my keys in git cost me $6500 in a few hours.

          Seriously now, when will people stop putting keys in git? This is only going to keep happening as long as they do, and the smallest violin maker doesn’t produce enough fiddles to keep up with demand already.

          1. 6

            Putting private keys in a private repo sounds safe enough on the surface.

            Amazon could trivially offer a variety of features which would effectively mitigate this; they’ve chosen to keep putting the burden on their users.

            A few things they could do:

            • Add an IAM setting to automatically revoke compromised keys (after all, they detected it), and enable it by default for new accounts.
            • Add a setting to require a positive balance before starting new instances.
            1. 4

              This could really easily be solved by a pre-payment scheme: for typical individual users, you put in a few dollars each month to run whatever you’ve got running for that month. If an attacker gets in, they can spend the rest of your balance, but not run up a multi-thousand-dollar bill.

              This particular case is a whole cascade of failures:

              • The user put credentials in source control
              • Github’s Visual Studio plugin behaved unexpectedly and unsafely
              • Amazon completely failed to mitigate the attack (even though they were able to detect it!)
              1. 5

                I strongly prefer that payment style also, for other reasons too— in academia money tends to come in bursts and requires bureaucracy to spend, so I’d prefer to put $500 into a cloud-computing account now and then, at a time of my own choosing, rather than receive an invoice every month. It also makes it safer to give students access to things with assurance that in the worst case all they can do is run down the budget, not spend over-budget.

                I currently use a small Italian cloud provider that lets you pay that way. It doesn’t have AWS levels of services (or reliability), just a pretty standard Apache CloudStack install on a cluster of machines + NAS appliance. But for an individual user it has what I need, is quite a bit cheaper, and lets me recharge the account when I want. (Would be interested in hearing if there are other cloud providers people are happy with who allow this style of payment.)

                1. 1

                  digitalocean.com provides a prepay option via PayPal only. DO provides VPS paid by the hour and has an API to manage them. Not sure of its feature parity with AWS but I’ve never had reliability problems with them.

            2. 4

              Perhaps git should throw an error if you try to add a directory named .ssh/? If you really did want to upload .ssh/, you could simply pass a parameter to git add, overriding this protection.

              I’ve read at least three of these stories. And I’d say if it can be protected with technology it should be. Asking users to be safe should be a last resort :P

              1. 1

                As much as I hate to admit this, neither Microsoft nor GitHub are directly responsible for the data breach. The fact that the key was in the repo was the developer’s fault and no one else’s, it looks like he is just looking for a target to lash out at.

                1. 7

                  As @craigstuntz linked, Microsoft tracked and fixed it as a VisualStudio bug. It is not user error. I agree with you that putting keys in source control is a bad pratice, but “developer’s fault and no one else’s” is simply incorrect.

                  1. 8

                    The key disclosure isn’t even the only disclosure here, like yes secrets in git is a terrible practice and it’s reasonable for like one person to show up in the thread saying so, and not like a jerk, but c'mon have some chill, people. This dude was also worried about his source and he may be out 6.5 grand. That sucks and he might just be writing the blog post to sway some Amazon robots into doing something about the bill, too.

                    1. 2

                      OK, understood. Sorry for the flippant remark, didn’t mean to be incendiary.

                      @meridith Thanks for reminding me of the other factors he has to deal with, for some reason they hadn’t completely sunk in when I wrote that. ;-)

                  2. 2

                    To the people who dv-ed as incorrect: My point is not that MS and GitHub are entirely free from blame, but that they are not directly responsible. You shouldn’t even be putting keys in private repos.

                    1. 1

                      I disagree. The guy wanted to put private data in a private repo. He clicked the “private repository” checkbox, and the plugin made the repo public. Clearly a software bug.

                      I agree private keys in a repo might not be a good idea, but it’s irrelevant here. People put a lot of valuable private data in source control (like proprietary source code).

                      I’m almost surprised GitHub didn’t pick up the bill just to avoid the embarrassment of having this all over tech news sites. Clearly they didn’t test it very well…