The everything authors did nothing wrong, I fully support their future endeavors, and the whole incident was hilarious.
Did they inconvenience others? Yes, and as soon as they realized that, they tried to remove it. When that didn’t work, they reached out to NPM so it could be forcibly removed.
That’s what I was wondering about, is there a technical reason why you needed to upload the packages ? Seems to me like more work than using local packages, and would have avoided the whole mess.
is there a technical reason why you needed to upload the packages
Not really. We were curious if any package manager could handle it, thought it would be cool to call it npm install everything instead of npx everything or whatever (otherwise it wouldn’t be a “hack”, which is a creative use of existing tools), and thought it would be easier to avoid writing our own package manager if we didn’t need to. Well, hindsight is 20/20.
Their rebuttals are pretty reasonable. In fact, once they realized they had created a serious problem, they started reaching out repeatedly to amend the harm.
I do think there’s something to “fuck around, find out”, but ultimately the massive corporation running the crucial infrastructure should own a significant chunk of the responsibility: abuse is well known, and these were things that any reasonable product security team should have seen coming. It sounds like NPM has been under-invested in, at least on that front. These sorts of “attacks” are nothing new.
They got lucky that it was someone fooling around, instead of a concentrated attack to create a prolonged outage (which sounds feasible judging by npm and github’s responses).
Apart from all the defenses of the OP’s conduct that have already been made in this thread:
GitHub support makes some pretty egregious claims here, let’s walk through them one by one:
Harassing other users of the Service is never tolerated, whether via public or private media.
We did not harass anybody, and we are essentially unable to prove it now. It’s now the word of them (a trillion dollar corporation) against us (a few random netizens with an average age of 20). Luckily, I have the full issue history saved in screenshots. These will be attached at the end of the article.
is pretty egregious behavior on the part of Github. I’m personally already of the opinion that Github is an untrustworthy force and no one should depend on it, and I’ve said as much multiple times - I host a lot of my own code off-github and only use my Github account reluctantly, to contribute to other peoples’ open-source projects that host on Github. I think projects like Radicle that are trying to build meaningful alternatives to the social infrastructure side of Github are valuable projects for programmers to work on (multiple meaningful alternatives to the Git forge side of Github already exist, thankfully; the thing that needs to be combatted is Github’s network effects)
To the author: this is one of the most interesting technical pieces I’ve read in awhile. Kudos. Probably because it’s a mix of tech, hacking, and politics. A little bit of FAFO helps, as well.
Thank you Matt, that’s incredibly generous and it’s hugely appreciated. It’s been gratifying to learn in the past few days (or, rather, weeks) that people do appreciate my writing
I’m finding it hard to sympathize with the author. “Let’s see what will happen!” is not an acceptable justification for the abuse of a public service used by millions. There is no timeline where “depend on everything” is a real world use.
I think that’s an unnecessarily unsympathetic take.
I don’t know about you, but my default assumption when thinking about dependency systems is that they operate as an append-only DAG, and so it seems reasonable to assume that adding dependent packages shouldn’t meaningfully impact dependencies.
Obviously, this assumption is incorrect (at least, in the case of NPM) and a more accurate model recognises that there’s more going on here: but I still have sympathy with those that unknowingly expect technology to behave in a manner consistent with the manner in which it is commonly advertised.
I also think that the folks at Microsoft/GitHub have egg on their face. This is a publicly available service on the Internet owned by one of the biggest tech companies and used by a relatively large fraction of the world’s developers. It is their job to think through how their product works including edge cased like “huh, if a third party referencing a package prevents the original package owner from doing something I wonder if that could be abused?” My most charitable guess is that this was brought up by an engineer and permanently backlogged until they entered the “find out” stage of running a public Internet service.
Both parties acted pretty unprofessionally, in different ways, but I expect script kiddies to be unprofessional and I expect npm/GitHub to do better.
npm is not a public service; it is privately owned by (a subsidiary of a subsidiary of) Microsoft. The authors didn’t anticipate that their antics would lead to a denial of service or any other degradation, and they weren’t doing anything prohibited (at the time) by the EULA or TOS; I’m not sure “abuse” is the right word for what they did.
MS/GH reaction to good-faith actors finding an issue tells more about GH/MS than about those actors in any case. Those folks weren’t even intentionally trying to trigger any issues — while legitimate security researchers may and it’s uncontroversial that they do it.
In all fairness, legitimate security researchers wouldn’t impact others as part of their research, whereas we did by accident. That does not change the fact that this problem, now that it’s discovered, should be addressed
Honestly the message I get from MS/GH/npm here is if you ever do anything that accidentally takes out one of their services out you should come up with a catchy name, buy a domain, and say you found a security vulnerability.
Then forward it to the press and CC their press contact.
Why report anything going wrong to a company that will go to the press falsely claiming malice and failing to report that you contacted them about the issue? It’s only marginally different from when companies used to sue people for publishing security vulnerabilities.
[edit: they only took this path of libel here because they could use it to distract from their incompetence, and the solution in future is to not give them that opportunity]
My friend who doesn’t publish his real name had no trouble deciding to make a post. Me, I’m pretty young and my real name is attached to this, so I had to think long and hard before hitting publish. On one hand, once this hit the news, all I wanted was for it to die down. On the other, I had a cool story to tell and I wanted my side out there, for the reasons you stated. Ultimately I chose the latter, and nothing really came of it till today (it’s funny how the internet works like that). I think I made the right choice. It feels like closure.
[Edit: All this is really in response to your last statement — that they should not be given opportunity. I should add that the calculus would have been different if I had any sort of a platform. The question I really was grappling with is “what do I do when a large company is upset at us and I want my voice heard”]
I don’t know, perhaps also my perspective is influenced by where I’m from (Germany), and possibly other countries aren’t handling things as badly as our authorities here.
With news stories like this one (excuse that it’s in German; essentially the person reporting the vulnerability got their home searched by police and their PC, five Laptops, and a phone taken away for more than a year, until the company suing them lost their case in court and the court ruled they could get their devices back), I would never have the balls to disclose any security vulnerability unless I’d be doing it anonymously, or for a well-known company with a good track record of not displaying such adversarial behavior.
Well, you never know what the impact is going to be. If you find that sending a request with a certain sequence of characters can crash my web server, the vulnerability is still my problem. The only difference between a good-faith actor and a malicious one is that the malicious one would try to use it against me, while a white-hat person would try to help me learn about the problem and fix it.
Nah I think legitimate security researchers might well impact others as part of their research - quite possibly justifiably, or in a context where it’s genuinely unclear which party is in the right.
I tend to disagree – this sounds to me like “art project gone haywire.” The original goal was a gonzo “what if” experiment that just happened to trigger an unforeseen vulnerability in npm. If anything, the author did a public service by accidentally discovering this before a bad actor did.
No, they went “can we make a project that depends on everything” which I think is an interesting question.
They did not realize - because of a policy change NPM instantiated in response to the response from a developer the last time they attacked the community members on whom their entire product depends - that it would cause problems. Then despite being a large corporation who’s actual product is provided entirely by unpaid volunteers, NPM provided no response, and then went to the media and again attacked the community members, and as far as I can make out straight out lied to the press claiming that this was intentional and malicious abuse.
I’m not sure why you have decided to join team “blame the young people who made a seemingly minor mistake” and side with the corporation making false claims about them, but I think you should take some time to reflect on why you feel unsympathetic to someone young making a minor mistake in doing something that is a reasonable thing to try, and should not have been harmful in any well run project, that ends up going so wrong because of how badly the billion dollar corporation handled it that they were called by reporters.
I don’t feel sympathetic to the profiteering corporation making false claims and libeling people to cover for their own incompetence - all because of a problem that only existed because of their previous shitty behaviour - and it’s bizarre to me that you think this is the author’s fault.
There is no timeline where “depend on everything” is a real world use.
This could, in fact, be a community service activity. As an easy example one could depend on everything and then run a static analyzer against the whole thing to try to find security problems. Or spin up a clean VM, npm install everything, and watch for unexpected filesystem modifications and divide-and-conquer to determine if there’s packages in the registry that do malicious things. Or maybe someone’s developing e.g. node-gyp and wants to ensure that the changes they’re making don’t break any existing package builts; npm install everything, make changes, npm install everything again and see if the behaviour changes.
Indeed. If they wanted to screw around like this, then clone the registry on your own system, and try this sort of crap on your own, without bothering anyone else.
I don’t think people should be allowed to delete packages (I think rust’s solution is perhaps the best). What I do know is that people expected to be able to delete them, and we broke that.
The
everythingauthors did nothing wrong, I fully support their future endeavors, and the whole incident was hilarious.Did they inconvenience others? Yes, and as soon as they realized that, they tried to remove it. When that didn’t work, they reached out to NPM so it could be forcibly removed.
Strongly agreed; this was 100% a problem with the design of NPM, not their fun hack.
We might do this again but without actually publishing the packages. It feels like an unfortunate end
That’s what I was wondering about, is there a technical reason why you needed to upload the packages ? Seems to me like more work than using local packages, and would have avoided the whole mess.
Not really. We were curious if any package manager could handle it, thought it would be cool to call it
npm install everythinginstead ofnpx everythingor whatever (otherwise it wouldn’t be a “hack”, which is a creative use of existing tools), and thought it would be easier to avoid writing our own package manager if we didn’t need to. Well, hindsight is 20/20.Please do, and let us know what happens!
Their rebuttals are pretty reasonable. In fact, once they realized they had created a serious problem, they started reaching out repeatedly to amend the harm.
I do think there’s something to “fuck around, find out”, but ultimately the massive corporation running the crucial infrastructure should own a significant chunk of the responsibility: abuse is well known, and these were things that any reasonable product security team should have seen coming. It sounds like NPM has been under-invested in, at least on that front. These sorts of “attacks” are nothing new.
They got lucky that it was someone fooling around, instead of a concentrated attack to create a prolonged outage (which sounds feasible judging by npm and github’s responses).
I wish there was a little more here about the “NPM was out of office” aspect. Because…what??
Apart from all the defenses of the OP’s conduct that have already been made in this thread:
is pretty egregious behavior on the part of Github. I’m personally already of the opinion that Github is an untrustworthy force and no one should depend on it, and I’ve said as much multiple times - I host a lot of my own code off-github and only use my Github account reluctantly, to contribute to other peoples’ open-source projects that host on Github. I think projects like Radicle that are trying to build meaningful alternatives to the social infrastructure side of Github are valuable projects for programmers to work on (multiple meaningful alternatives to the Git forge side of Github already exist, thankfully; the thing that needs to be combatted is Github’s network effects)
To the author: this is one of the most interesting technical pieces I’ve read in awhile. Kudos. Probably because it’s a mix of tech, hacking, and politics. A little bit of FAFO helps, as well.
Thank you Matt, that’s incredibly generous and it’s hugely appreciated. It’s been gratifying to learn in the past few days (or, rather, weeks) that people do appreciate my writing
I’m finding it hard to sympathize with the author. “Let’s see what will happen!” is not an acceptable justification for the abuse of a public service used by millions. There is no timeline where “depend on everything” is a real world use.
I think that’s an unnecessarily unsympathetic take.
I don’t know about you, but my default assumption when thinking about dependency systems is that they operate as an append-only DAG, and so it seems reasonable to assume that adding dependent packages shouldn’t meaningfully impact dependencies.
Obviously, this assumption is incorrect (at least, in the case of NPM) and a more accurate model recognises that there’s more going on here: but I still have sympathy with those that unknowingly expect technology to behave in a manner consistent with the manner in which it is commonly advertised.
On the other hand, it’s crazy to me that this is an attack vector in a major package manager, and that part is entirely the fault the npm.
I think the people who did this were dumb (it’s less “hacker spirit” and more “huh, I could make a suicide plug I wonder what will happen?). https://twitter.com/bshoup/status/1336423185999388678
I also think that the folks at Microsoft/GitHub have egg on their face. This is a publicly available service on the Internet owned by one of the biggest tech companies and used by a relatively large fraction of the world’s developers. It is their job to think through how their product works including edge cased like “huh, if a third party referencing a package prevents the original package owner from doing something I wonder if that could be abused?” My most charitable guess is that this was brought up by an engineer and permanently backlogged until they entered the “find out” stage of running a public Internet service.
Both parties acted pretty unprofessionally, in different ways, but I expect script kiddies to be unprofessional and I expect npm/GitHub to do better.
npmis not a public service; it is privately owned by (a subsidiary of a subsidiary of) Microsoft. The authors didn’t anticipate that their antics would lead to a denial of service or any other degradation, and they weren’t doing anything prohibited (at the time) by the EULA or TOS; I’m not sure “abuse” is the right word for what they did.MS/GH reaction to good-faith actors finding an issue tells more about GH/MS than about those actors in any case. Those folks weren’t even intentionally trying to trigger any issues — while legitimate security researchers may and it’s uncontroversial that they do it.
In all fairness, legitimate security researchers wouldn’t impact others as part of their research, whereas we did by accident. That does not change the fact that this problem, now that it’s discovered, should be addressed
Honestly the message I get from MS/GH/npm here is if you ever do anything that accidentally takes out one of their services out you should come up with a catchy name, buy a domain, and say you found a security vulnerability.
Then forward it to the press and CC their press contact.
Why report anything going wrong to a company that will go to the press falsely claiming malice and failing to report that you contacted them about the issue? It’s only marginally different from when companies used to sue people for publishing security vulnerabilities.
[edit: they only took this path of libel here because they could use it to distract from their incompetence, and the solution in future is to not give them that opportunity]
My friend who doesn’t publish his real name had no trouble deciding to make a post. Me, I’m pretty young and my real name is attached to this, so I had to think long and hard before hitting publish. On one hand, once this hit the news, all I wanted was for it to die down. On the other, I had a cool story to tell and I wanted my side out there, for the reasons you stated. Ultimately I chose the latter, and nothing really came of it till today (it’s funny how the internet works like that). I think I made the right choice. It feels like closure.
[Edit: All this is really in response to your last statement — that they should not be given opportunity. I should add that the calculus would have been different if I had any sort of a platform. The question I really was grappling with is “what do I do when a large company is upset at us and I want my voice heard”]
“used to”? I thought that’s still common practice, no?
fair call, while most companies have realized at this point that that results in the Streisand effect, there are still a few idiot companies though :D
I don’t know, perhaps also my perspective is influenced by where I’m from (Germany), and possibly other countries aren’t handling things as badly as our authorities here.
With news stories like this one (excuse that it’s in German; essentially the person reporting the vulnerability got their home searched by police and their PC, five Laptops, and a phone taken away for more than a year, until the company suing them lost their case in court and the court ruled they could get their devices back), I would never have the balls to disclose any security vulnerability unless I’d be doing it anonymously, or for a well-known company with a good track record of not displaying such adversarial behavior.
Well, you never know what the impact is going to be. If you find that sending a request with a certain sequence of characters can crash my web server, the vulnerability is still my problem. The only difference between a good-faith actor and a malicious one is that the malicious one would try to use it against me, while a white-hat person would try to help me learn about the problem and fix it.
Nah I think legitimate security researchers might well impact others as part of their research - quite possibly justifiably, or in a context where it’s genuinely unclear which party is in the right.
I tend to disagree – this sounds to me like “art project gone haywire.” The original goal was a gonzo “what if” experiment that just happened to trigger an unforeseen vulnerability in npm. If anything, the author did a public service by accidentally discovering this before a bad actor did.
“Dont do things with no real world use” isn’t one of the rules.
No, they went “can we make a project that depends on everything” which I think is an interesting question.
They did not realize - because of a policy change NPM instantiated in response to the response from a developer the last time they attacked the community members on whom their entire product depends - that it would cause problems. Then despite being a large corporation who’s actual product is provided entirely by unpaid volunteers, NPM provided no response, and then went to the media and again attacked the community members, and as far as I can make out straight out lied to the press claiming that this was intentional and malicious abuse.
I’m not sure why you have decided to join team “blame the young people who made a seemingly minor mistake” and side with the corporation making false claims about them, but I think you should take some time to reflect on why you feel unsympathetic to someone young making a minor mistake in doing something that is a reasonable thing to try, and should not have been harmful in any well run project, that ends up going so wrong because of how badly the billion dollar corporation handled it that they were called by reporters.
I don’t feel sympathetic to the profiteering corporation making false claims and libeling people to cover for their own incompetence - all because of a problem that only existed because of their previous shitty behaviour - and it’s bizarre to me that you think this is the author’s fault.
This could, in fact, be a community service activity. As an easy example one could depend on everything and then run a static analyzer against the whole thing to try to find security problems. Or spin up a clean VM, npm install everything, and watch for unexpected filesystem modifications and divide-and-conquer to determine if there’s packages in the registry that do malicious things. Or maybe someone’s developing e.g. node-gyp and wants to ensure that the changes they’re making don’t break any existing package builts; npm install everything, make changes, npm install everything again and see if the behaviour changes.
Indeed. If they wanted to screw around like this, then clone the registry on your own system, and try this sort of crap on your own, without bothering anyone else.
Save your moral indignation for literally anything else that’s actually materially significant in the world.
Why should anyone be able to delete an npm package, though? What difference would it really make if it were all append-only?
I guess I don’t really understand the harm aspect. Do package authors have some right to not be depended-upon by others?
Anyhow, cool project. You can easily download all of Wikipedia, so why not npm?
I don’t think people should be allowed to delete packages (I think rust’s solution is perhaps the best). What I do know is that people expected to be able to delete them, and we broke that.
Relevant XKCD: https://xkcd.com/1172/
[Comment removed by author]