1. 85
  1.  

  2. 41

    Wow, that’s pretty terrible.

    On the other hand, I can’t help but to feel sorry about Dominic, we all make mistakes, this public shaming is pretty violent.

    I guess we should sometimes take some time off to read the license before using a library:

    THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

    (F)OSS is not a consumer good.

    1. 11

      I agree that shaming people is toxic and unproductive. No one wants to be shamed and no one is perfect.

      But I see another dimension to the negative responses Dominic has received. Non-hierarchical, self-governing communities like open source software are organized by social norms. Social norms work through peer pressure - community members conform to the norms of the community not because they are compelled to by law but because it would cost them standing in the community not to. This isn’t inherently good. Some norms are toxic and self-policing via peer pressure can lead to shaming. What I see in some of the critical comments addressed to Dominic is an attempt to establish a clear social norm about what to do when you are ready to abandon a package. The norm is desirable because it increases the general level of trust. Even if the landscape is generally untrustworthy, you can have some confidence that people aren’t handing their packages off to strangers because it’s the norm not to do that. The desire for some norm here, whatever it is in the end, is reasonable.

      Ending the discussion with “don’t worry about it Dominic, everyone makes mistakes, and anyways you’re not liable for it” signals to everyone that they’re not responsible for the consequences of what they do. In a strictly legal sense, that might be true. Even then, I’m skeptical that the warranty clause would cover negligence in the distribution of the software rather than the software itself. But in either case, don’t we want a community where people do feel responsible for the actions they take and are open to receiving feedback when an action they’ve taken has a bad result? This dialogue can occur without shaming, without targeting anyone personally, and can be part of the same give-and-take process that produces the software itself.

      1. 6

        Blaming people in any security issue is toxic, no matter what happens. In any organization with paid people where you should expect better, the most important rule of a post-mortem is to remain blameless. It doesn’t get anyone anywhere and doesn’t get remotely close to actual root cause. Instead of asking about why Dominic gave away a critical package, people should be asking why some random maintainer were able to give away a critical package.

        Ending the discussion with “don’t worry about it Dominic, everyone makes mistakes, and anyways you’re not liable for it” signals to everyone that they’re not responsible for the consequences of what they do.

        By putting blame on Dominic, people are not taking responsibilities. The main issue is that many core libraries in the JavaScript ecosystems still depends on external, single-file, non-core, likely unmaintained library. People who should take responsabilities are the ones who chose to add a weak single point of failure by depending on event-stream.

        1. 2

          It depends what you mean by blame. If you mean assigning moral responsibility, especially as a pretext for shaming them, then I agree it’s toxic. I think I was clear that I agree this shouldn’t happen. But if blame means asserting a causal relationship between Dominic’s actions and this result, it’s hard to argue that there isn’t such a relationship. The attack was only possible because Dominic transferred the package. This doesn’t mean he’s a bad person or that he should be “in trouble” or that anything negative should happen to him as a consequence. A healthy social norm would be to avoid transferring packages to un-credentialed strangers when you’re ready to abandon the package because we’ve seen this opens an attack vector. Then what’s happened here is instructive and everyone benefits from the experience. And yes, ideally these dilemmas are prohibited by the system. Until that is the case, it helps to have norms around the best way to act.

          1. 1

            I understand you don’t condone the attacks and shaming going around. However I would argue that even if you agree that the blaming is toxic, that building some social norm around it is better than nothing, I believe that even hinting that it was somehow Dominic’s fault is a net negative.

            The attack was only possible because Dominic transferred the package.

            This is exactly what I’m condoning. By looking at individual and their action you scope the issue at that level. The attack was taking over a dependancy. It is possible to do so in so many way, especially for packages such as Dominic’s. This time it was a case of social engineering, next time it might as well be a case of credential hijacking, phishing or maintainer going rogue.

            A healthy social norm would be to avoid transferring packages to un-credentialed strangers when you’re ready to abandon the package because we’ve seen this opens an attack vector.

            I would say pushing this rethoric is actually unhealty and only lead people to rely on those social norm and use it as an excuse to disown their accountability. It would be much healthier to set expectation right and learn proper risk assessment about dependancies management.

            Then what’s happened here is instructive and everyone benefits from the experience. And yes, ideally these dilemmas are prohibited by the system. Until that is the case, it helps to have norms around the best way to act.

            The same issue have come up so many time in the past few years, especially in the NPM ecosystem, it should be well past the “learn from the experience” and I believe it’s time the relevant actors actually move toward a solution.

      2. 17

        I’ve done a similar thing before. After leaving the Elm community, I offered to transfer most of my repos over to the elm-community organisation. They accepted the most popular ones, but not elm-ast (and maybe one or two others). A few months later I received an e-mail from @wende asking if he could take over so I took a look at his profile and stuff he’s done in the past and happily gave him commit access thinking users would continue getting updates and improvements without any hassle. Now, @wende turns out to be a great guy and I’m pretty sure he hasn’t backdoored anyone using elm-ast, but I find it hilarious that people somehow think that maintainers should be responsible for vetting who they hand over control of their projects to or that they could even do a good job of it OR that it would even make any sort of a difference. Instead of trusting one random dude on the internet (me) you’re now trusting another.

        Don’t implicitly trust random people on the internet and run their code. Vet the code you run and keep your dependency tree small.

        1. 25

          Vet the code you run

          Or trust well-known, security-oriented distributions.

          keep your dependency tree small

          Yes, and stay away from environment, frameworks, languages that force dependency fragmentation on you.

          1. 4

            Or trust well-known, security-oriented distributions.

            That too! :D

            1. 3

              and stay away from […] frameworks

              I wouldn’t say that as absolutely for the web. I suspect that things would likely go a lot more haywire if people started handling raw HTTP in Python or Ruby or what have you. There’s a lot of stuff going on under the hood such as content security policies, CSRF protection and the like. If you’re not actively, consciously aware of all of that, a web framework will probably still end up providing a net security benefit.

              1. 5

                Please don’t quote words without context:

                […] that force dependency fragmentation on you

                Frameworks and libraries with few dependencies and a good security track record are not the problem. (If anything, they are beneficial)

                1. 2

                  I interpreted “Yes, and stay away from environment, frameworks, languages that force dependency fragmentation on you.” as (my misunderstandings in brackets) “Yes, and stay away from [(a) integrated development] environments, [(b)] frameworks, [(c)] languages that force dependency fragmentation on you.” with a and b being separate from the “that” in c.

                  I apologize for the misunderstanding caused.

              2. 2

                Isn’t it the case that reputable, security-focused distributions acquire such status and the continuity thereof by performing extensive vetting of maintainers?

                The responsible alternative being abandoning the project and letting the community fork it if they want to.

                1. 1

                  Or trust well-known, security-oriented distributions.

                  Then how do You deal with things like that: “The reason the login form is delivered as web content is to increase development speed and agility” ?

                  1. 2

                    As a distribution? Open a bug upstream, offer a patch, and sometimes patch the packaged version.

                    1. 1

                      That’s a good idea in general but sometimes the bug is introduced downstream.

              3. 9

                Most proprietary software also comes with pretty much the same warranty disclaimer. For example, see section 7c of the macOS EULA:

                https://images.apple.com/legal/sla/docs/macosx107.pdf

                I mean, have we held accountable Apple or Google or Microsoft or Facebook in any substantial ways for their security flaws?

                1. 4

                  In many other products accountability is enforced by law and it overrides any EULA. And that is tied to profit in the broad sense: sales or having access to valuable customer data & so on.

                  Software companies got away with zero responsibility and this only encourages bad software.

                  1. 1

                    And how have we enforced that by law for those companies, regardless of what those EULAs have said? When macOS allowed anyone to log in as root, what were the legal consequences it faced?

                    1. 3

                      other products

                      e.g. selling cars without safety belts, electrical appliances without grounding…

                2. 2

                  It is a security disaster given how easy it is for js stuff to hijack cookies and sessions.

                  1. 1

                    It really isn’t if a well thought out CORS policy is defined.

                3. 19

                  Some lessons I can gather from this:

                  • Distributing a second minified version should not be a thing, npm should minify for you, the individual packages should not have the ability to do it
                  • There should be an option to unlist yourself as maintainer without adding another maintainer.

                  I suspect that leaving a package unmaintained is still an improvement over handing it to an unknown person. Someone who wants to pick it up can do so under a different name. The alternate proposal of handing it over to a team of volunteers that is vetted in some way is also good.

                  1. 7

                    I agree with the first point but I’d generalize “minification” with “any source code transformation”. In my opinion minification isn’t the problem itself, we compile other programming languages (C, C++, Rust, …) into binary gibberish for years and I think any kind of TypeScript or JSX translation can hide as many nasty things as minification: the generated code is not very readable and no one reads it. In my opinion, the real problem is that there is no real, trusted link between the actual source code of a library on GitHub and its package on npmjs.org. The package itself on npm is a simple tarball uploaded from the maintainer’s computer and AFAIK there is no easy way to verify that this tarball is an authentic build from the original source code. You must trust the maintainer.

                    Of course, lots of other package managers for other langages should be vulnerable to the same issue.

                    1. 1

                      I agree, unless a project owner can absolutely trust the people they are handing repository rights to they should mark the project as abandoned and allow the nature course of forking take the project in to new maintained directions.

                    2. 18
                      1. 27

                        I think people talking about inspecting the source before installing dependencies are being unreasonable to some degree.

                        1. The malicious code was present only in the minified version of the code. I suppose the red flag that tipped the reporter was the lack of history/popularity of the repository in question, but it doesn’t have to be like that
                        2. It can be released to npm in a way that’s not evident to casually browsing the github repo
                        3. There isn’t even any guarantee that the code on npm matches what’s on github at all

                        Meaning the ways to be safe are:

                        1. Hand-inspect the code in your node_modules directory (including — or especially— those that may be minified); or
                        2. Don’t use npm at all.

                        I don’t see these people (nor myself) doing either. From which it follows:

                        Any company desiring to buy into the so-called “modern” front end development (be it for productivity, performance or hiring purposes) does so by making itself vulnerable to attacks such as this.

                        I don’t know if that’s a reasonable price to pay to use, say, React, but it sure isn’t reasonable to me to pay that to use Node (versus, say, Golang, which can reasonably be used to build the same kinds of apps using little more than the standard library).

                        1. 21

                          The malicious code was present only in the minified version of the code. I suppose the red flag that tipped the reporter was the lack of history/popularity of the repository in question, but it doesn’t have to be like that

                          One more reason for reproducible builds… minified JS should be treated like compiled code and automated mechanisms should check if it matches the unminified version…

                          1. 6

                            This, a thousand times this. I can’t comprehend the reasoning that goes into committing derived code into source control. It’s a pain to remember to update it every time you commit, it’s hard to verify the code matches the original source and just pollutes the history. Diffing is mostly undoable too.

                            1. 3

                              I think the reasoning is to avoid build dependency. For some time, it was a usual practice to include Autoconf-derived configure script in release artifacts, so that users can avoid installing Autoconf.

                              1. 1

                                Yeah, that’s annoying too (and a lot of projects still do it even though it’s not really good practice), but at least configure scripts don’t tend to/need to change with every single code change like these minified files do.

                                1. 1

                                  generated autoconf configure scripts are pretty easy to read, I can say there were times I preferred them over the m4 source.

                            2. 11

                              It would be really nice if the package repositories (npm/pypi/rubygems/etc) did something:

                              • try to automatically detect obfuscated code
                              • stop letting maintainers upload packages from their dev machines, make sure any compilation happens on a public CI environment from a known git tag (this would also encourage non-compiled packages, i.e. just direct snapshots of git tags)
                              • have some popularity threshold for packages beyond which manual review from a trusted group of reviewers is required for each new release
                              • (also why not require the git tags to be gpg signed for these popular packages)
                              • maybe rethink the whole package handover thing, maybe only allowing “deprecation in favor of [a fork]” (i.e. requiring every update to be manual) is good
                              1. 3

                                I wouldn’t even check the node_modules output either as package installation can execute arbitrary code (node-gyp, other node bindings producing code)

                                1. 4

                                  I agree with you!

                                  People seems to like to hit on npm, but I don’t see how the core issue is different than say Pypi, Cargo or Go (Other than the issues you raised). I personnaly take easy and simple dependancies management over C/C++ fragmented package management because most of my project are not security critical anyway or my threat model doesn’t include targeted code injection in my stack.

                                  I find it annoying when people look at those issues and some fault is put on the maintainers. Maybe the issue is not that one of your application’s thousands of dependancies compromition, but the fact that your risk management for your wallet application relies on thousands of unvetted dependancies…

                                  Meaning the ways to be safe are:

                                  I guess a first start would be to gather a bunch of useful and common repositories and ensure they and all their dependancies are well vetted and signed by the maintainers for each release and prevent any new dependancies from being pulled in without proper review and ensuring those dependancies use the same process. Documenting and enforcing such process for a subset of widely used dependancies would allow to trust a few organization and avoid to code review any dependancies I pull in in my own project. I guess most distribution core repositories has similar process like Arch maintained packages vs AUR.

                                  1. 8

                                    Pypi absolutely has the the same potential issues, though in practice I think the dependency trees for popular projects are way smaller than what you get in the node ecosystem. So you’re much less likely to be hit by a transitive vulnerability. To me this is one of the advantages of a fairly comprehensive standard library, and a relatively small number (compared to node, at least) of popular, high quality third-party libraries that get a lot of eyeballs.

                                    1. 11

                                      On top of that, a lot of Python code is deployed to production by system engineers. Often it’s vetted, built, tested and baked in by distributions - and the same is true for other non-web languages.

                                      javascript, on the other hand, is more often deployed by the upstream developer and thrown at web browsers straight away without any 3rd party review.

                                      1. 3

                                        Definitely! But that somehow happened to be this way. It would be nice to look at the social side as to why Python ended up this way while nothing prevented it from ending up like NPM. Maybe some key aspect of the tooling drive the trend one way or the other or it might be just the community (Python being much older and the tooling has seen a lot of changes over the years).

                                        I would be looking forward to a someone doing a graph analysis of a few package repositories across languages and find some way to rate them and put some risk on packages. How many and how deep does their dependancies go? How many of them are maintained by external maintainer? Sounds like I found myself a new week-end project…

                                        1. 12

                                          Python has a decent class library. Good libraries that have general use migrate back into that class library, in some fashion or another. Thus, third party libraries don’t have to have long dependency chains to do anything.

                                          What NPM forgot was that this was the fundamental idea that made package management useful. This stretches back to the early days of Java, at least, and I’m sure you can find other similar examples. By having a rich class library which already provides most of what you need, you’re simply going to layer on dependencies to adapt that framework to your specific business needs. Java, .NET, Ruby, Python- they all have that going for them. JavaScript simply does not. So half the Internet unwittingly depends on leftpad because a dependency of a dependency of a dependency needed it, and there wasn’t anything in the core library which could do it.

                                          1. 1

                                            Maybe some key aspect of the tooling drive the trend one way or the other or it might be just the community (Python being much older and the tooling has seen a lot of changes over the years).

                                            I think this is a big part of it — Python’s tooling is generally less capable than the Node ecosystem’s.

                                            To this day Pip doesn’t have a dependency resolver, so the result of installing a dependency tree with conflicts at the transitive dependency level isn’t an error, but an arbitrary version getting installed. You can only have a single version of a Python module installed, too, because they are global state. Contrast with how npm has historically (still does?) install multiple versions of a package, effectively vendoring each dependency’s dependency tree, transitively.

                                            Additionally, publishing Python packages has long been messy and fraught. Today there is decent documentation but before that you were forced to rely on rumors, hearsay, and Stack Overflow. Putting anything nontrivial on PyPI (e.g., a C extension module) is asking for a long tail of support requests as it fails to install in odd environments.

                                            I think the end result was a culture that values larger distributions to amortize packaging overhead. For example, the popular Django web framework long had a no-dependencies policy (if dependencies were required, they were vendored — e.g. simplejson before it entered the standard library).

                                            Regardless of the reasons for it, I think that this is healthier than the Node culture of tiny dependencies with single contributors. More goes into distributing software than just coding and testing — documentation, support, evangelism, and legal legwork are all important — but tiny libraries have such limited scope that they’ll never grow a social ecosystem which can persist in the long term (of course, even Django has trouble with that).

                                            1. 1

                                              You can only have a single version of a Python module installed, too, because they are global state.

                                              That’s actually a pretty good point I think. I have fought myself a few time against Pip due to conflicting versions. It does benefits library with fewer dependancies.

                                        2. 1

                                          While I’m not generally a fan of it, I think minimal version selection that’s planned for the future go package manager would make this attack spread much more slowly.

                                      2. 8
                                        1. 4

                                          If I understand correctly the suspected malware was trying to snoop crypto keys from the environment? Obviously it sucks that this happened, but surely how you run your node application is also a big part of the problem.

                                          If you’re going to run it with sensitive information about other software available in the environment, isn’t that a bad practice to begin with? Likewise, if you run it with elevated privileges then aren’t you also making a mistake, from a defense-in-depth standpoint? I think we (as application developers using the node ecosystem) all need to take a bit more collective responsibility for letting issues like this affect us.

                                          Somewhat-relatedly, Ryan Dahl, the creator of Node, is now working on deno “A secure TypeScript runtime built on V8”. One of the features is

                                          File system and network access can be controlled in order to run sandboxed code. Defaults to read-only file system access and no network access. Access between V8 (unprivileged) and Rust (privileged) is only done via serialized messages defined in this flatbuffer. This makes it easy to audit. To enable write access explicitly use –allow-write and –allow-net for network access.

                                          1. 2

                                            If you’re going to run it with sensitive information about other software available in the environment, isn’t that a bad practice to begin with?

                                            In a server context yes, but node is also in use in a fair amount of desktop software as well where that’d be the norm rather than the exception.

                                            1. 2

                                              That’s a good point, I had forgotten about how common that use case is. And now I actually find that the most concerning of the three, for these kinds of vulnerabilities ( the others being: browser - fairly well sandboxed; server node app - securable by developer / devops / security policies).

                                          2. 3

                                            The JavaScript crowd gets a bad rep for these incidents, but it seems to me that there is nothing at the moment, except perhaps culture and convention, that would prevent these issues in the package systems of other languages (e.g., Ruby Gems, Python PiPy, Rust Cargo, etc.)

                                            We face a difficult problem: not using third-party libraries is not a realistic option (I will concede that the JavaScript takes it further than most communities; see left-pad or nice-try) and auditing every version of every dependency we use does not scale.

                                            A cool project that was announced on /r/rust a couple of months ago is crev. It’s a system to share code reviews and form webs of trust. I like this approach because it’s distributed, so one team does not need to review all their dependencies, they can rely on other reviewers that they trust. It’s also a language agnostic solution, so all communities benefit. I don’t know if crev is going to be the git of code reviews (meaning, quickly and widely adopted), but no matter the exact tool, signed and distributed code reviews is going to be a very good counter to the kind of issues that happened in NPM this week.

                                            1. 1

                                              The difference between javascript and ruby ecosystems is that a small rails apps dependencies have ~40 unique maintainers whereas a new create-react-app codebase uses dependencies from ~400.

                                              My numbers are inexact (I counted 6 months ago and don’t recall perfectly, but it was something like that).

                                              Crucially, I can mentally track the reputation of the members of the ruby community, thanks to the smaller total.

                                            2. 3

                                              When I originally pitched the idea for CommonJS, I had imagined the result of the effort as being like Python’s standard library. Node and npm came along, and all seemed good (even better, because of npm’s dependency handling), despite the fact that server-side JavaScript didn’t end up with a bigger standard library.

                                              Today, given that we have ES6 modules and better build tooling, it seems like JavaScript would be better off with bigger packages so that there weren’t so many dependencies and transitive dependencies, all maintained by a huge number of individuals. A standard library that offered a bunch of common functionality (akin to that of Python or Go) in a well-supported way (infrequent major versions, with clear deprecation warnings in the previous releases) could reduce the number of dependencies required dramatically.

                                              Granted, there’s no perfect way to draw the boundaries, but the lack of a perfect solution doesn’t mean there couldn’t be something better than what exists today. Unfortunately, the ship has sailed to some extent and it’s not clear to me that someone could make a more full-featured standard library that would get traction.

                                              1. 2

                                                This is why npm packages need some form of namespacing; something like username/package-name This would prevent package name squatting or situations like this where the dev hands over maintainership of the package to someone else.

                                                1. 1

                                                  Using URLs directly would help with this too.

                                                2. 2

                                                  Package managers are a rug over dirt.

                                                  I honestly wish we added our dependencies directly to our git trees. Updating a package should look like a source diff, your team should review it. I’d want tooling to automate the download, and maybe de-duplicate shared transitive dependencies, but that’s it! git commit is your package-lock.json.

                                                  Note that I’m not talking about auditing every single line of code: even a cursory review would reveal things like one of your dependencies being massive and having a huge attack surface, etc. It’s about surfacing costs so the incentives are better aligned.

                                                  1. 2

                                                    The cryptocurrency blog CCN reports that the repo has been used in numerous Bitcoin wallets:

                                                    https://www.ccn.com/breaking-numerous-bitcoin-wallets-may-have-been-compromised-by-rogue-developer/

                                                    1. 2

                                                      As long as people are installing complete modules, and updating them without vetting their code (which is not going to change any time soon, as I see it) this is going to continue, and get worse. Billions of dollars worth of code is relying on infrastructure and and a handful of volunteers, and basically just hoping they’re not the ones to get pwn3d.