This isn’t news to anybody who has to be involved with Node.js development. This is also an NPM issue, not a Node.js issue, despite how tightly coupled they are. There’s alternative package managers out there (Yarn) that try and solve these problems.
Do the alternate package managers use an alternate package repo?
An “ecosystem” that encourages and embraces “modules” to check if a number is even, and then another one to check if a number is odd, needs more than a different CLI tool to download the same shit code.
I don’t disagree with that last statement of yours at all. I’m just saying Node.js isn’t really the problem in and of itself, and there’s people in the Node.js world who are actively looking for better solutions to these problems.
And to be fair, its not really shit code, its just run of the mill average code.
This microoptimization is silly because JavaScript runtimes recognize modulus by a power of 2 and change the generated code accordingly.
The V8 code to do this is here.
The ChakraCore code for this is here, search for isModByPowerOf2.
SpiderMonkey appears to do something similar but the code is harder to track down.
You raise a good point. In that module’s repo, there’s a comment that reads:
Since it is ending up as a dependency of things like Webpack now (insane as that is), I would hope it would actually behave as expected given the name.
Isn’t this just a frustration with the fact that so many packages depend on so many packages? Because the same rogue actor stuff would apply to any tree dependency self-published system. See your local Gemfile. Do you really know what’s being installed?
So what’s the proposed fix? Stop depending on so many packages?
Perhaps there should be a line drawn for the major packages, to use less of these tiny modules.
But this is also part of the beauty of single responsibility packages. Composing extremely simple parts is all we really need, right?
Think about all of the tiny programs you run on your Unix machine of choice. Perhaps those are also at fault for the “too small to be worth it” sin.
The elm package manager will refuse to publish a package that violates semantic versioning. So if you’re a malicious actor, changing the API of some package for nefarious purposes, you’d be doing a bit of a hard job because since functions can’t change their signatures without you bumping the major version and since most elm-packages.json files will not update major versions automatically, it becomes harder to ship evil code. Also, a simple elm-package diff command would tell you what changed between versions, making it easier for you to audit the code. There is a lot of beauty in it. It is not evil proof, but it makes it a bit harder.
I wonder if you can push this to an extreme, where each public function is independently versioned. Then you could upgrade to a new version even if there are breaking changes, as long as they’re not in any part of the API you use. I’m not sure if it would be useful but its a fascinating idea.
I just took a look at the Gemfile for one of my websites and there are a lot of gems but ~80% come from the rails team and most of the rest are trusty sources like amazon. This is not even close to as bad as when I created a blank Vue project that already had thousands of packages many about 3 lines long.
Ruby seems to have lots of mega libs that do big tasks but JS devs seem to have taken the path of making a new package for every function. I do wonder how often JS devs add 2 packages that do the same thing because they forgot about the last one. The result is also that questions online about ruby often tell you to use methods built in to ruby or ActiveSupport where as JS questions either get you to build it yourself or use jquery.
In defense of ActiveSupport, you can include individual pieces of it at whatever granularity you like, and the documentation explicitly calls this out to you at the very beginning.
Even in projects that are not web-related, I sometimes bring ActiveSupport along but will only require 'active_support/core_ext' and add things later if I need them as the project grows (often things like time_with_zone or number_helper).
I think that it’s a good way to balance “provides a lot of stuff” and “forces you to use all of its stuff”. I want to complain that the rest of Rails’ sub-projects don’t provide the level of utility when included gradually, but I also appreciate that that is the whole bargain: Rails’ perceived productivity boosts are derived from a tight coupling that pays dividends, you as long as you go with the Rails Way.
Edit/Afterthought: Over the years, some of the better Ruby libs to hit this balance of providing a lot but allowing for gradual or clean interop (with other libs, or with your application code) have been those authored or worked on by Piotr Solnica.
What I do in my Python work is generally be willing to depend on large, tested projects with well established, thriving communities that have a good history of responding to security incidents. These packages usually do not depend on other packages or when they do, on well established ones (e.g. requests library).
I might include other, less known packages, if I intend to use most of their functionality and if it is not easy to replicate on my own. Otherwise I write my own version or copy with attribution and license only the relevant parts.
This is not perfect, but gives me some confidence that things will be fine.
It is also an approach that completely breaks down in my Javascript/Typescript work because of so many packages being pulled in and everything seemingly depending on everything else (yes, I am exaggerating).
The problem already starts with licenses. For many apps for example you’ll have to list the licenses of all used libraries, and the libraries, including all transitive dependencies in the UI.
For my Android app that’s a few dozen overall, and it took only a few hours to do this. For even the smallest Angular project, this is insanity.
So if you want compliance, you’ll need to throw the entire ecosystem out right now. Or you just violate the licenses. Often even projects such as angular have dependencies that have no license.
I’ll see your ‘duplicating the standard lib’ and raise you a ‘defining the alphabet’[1] and a ‘defining each ANSI colour individually’[2] :
The first one is ridiculous. JS apparently has no range() method, perhaps that would have been worthwhile writing as a module. But no. Three arrays of characters.
The second one is just ridiculous modularisation. A single ansi-colours (or -colors if you like your modules “de-queened”) module could export functions and constants (e.g.: ansi(message, ansi.YELLOW) or similar).
This is the same guy who wrote the ‘isEven’/‘isOdd’ I mentioned earlier. He has ~1400 packages on NPM, and based on the ones I’ve looked at in the last few days, just his own packages have the typical nodejs dependency tree that mimics crab grass.
[about is-odd] Going through the dependents tree, I found hundreds of projects depending on this, including some major players are depending on this but more interestingly also the big players including Webpack, BrowserSync and Babel.
They’re depending on it via a glob matcherby the same author. Pretty much no one else uses that package. That’s the thing with tiny packages, you can often consider them parts of a bigger package by the same author.
And there’s not that many of these tiny packages.
Every once in a while someone writes another version of the same tired rant about the npm ecosystem that extremely exaggerates the problem and repeats the same “omg horrible code we can’t code 1000 layers deep millions of dependencies aaaaaaaa left pad left pad” mantra. COME ON. It’s not true. The dependency trees in Node projects are not that deep and not that wide, and one line packages are not that common.
I hate how it’s trendy to hate on JavaScript/Node/npm now. They absolutely do not deserve it. PHP actually sucked because of awful design. JS historically was closer to having no design, which does mean more dependencies for common simple things, which does have its problems, but is not a “disaster” or “shit” or anything like that. Actually that has its upsides too. “The standard library is where libraries go to die”, so not having a huge one means less old junk floating around that you’ll be tempted to use instead of a good module since that one is Built In™.
The author of this blog post is the same guy who ripped off James Iry’s classic A brief, and mostly wrong, history of programming languages with only a kind of backhanded attribution at the end of his (much less funny) post. He basically writes garbage blogs about programming, and somehow manages to get people talking about them. This post was probably taken directly from the Reddit discussion of this topic from a few days ago - certainly he hasn’t covered anything that wasn’t in the most upvoted comments of that thread.
I hate how it’s trendy to hate on JavaScript/Node/npm now. They absolutely do not deserve it. PHP actually sucked because of awful design.
PHP still has warts but a lot of improvements have been made to solve actual real world issues (i.e. type hinting, performance, removing unsafe/hard-to-use functionality, etc).
JavaScript still has the core problems it’s had since I was a student, at the beginning of this century. How can I write a javascript method that accepts two integers and.. well, knows they’re integers?
Oh. Right, we’re back to that guys “is-number” package, aren’t we?
List of package for a project I am working on which has ~50 dependencies in package.json and <20 dev dependencies (most of which required by couple of frameworks I use):
> ls node_modules|wc -l
857
My typical Python project usually has less than 20 external dependencies. That’s an order of magnitude difference. Admittedly these are not directly comparable projects, but it is also quite typical that my Javascript/Typescript work that depends on external frameworks quickly accumulates other packages numbered in few hundreds.
I find it difficult to argue that JS developers (this is really not a Node issue) are not more likely to depend on other packages than some other programming communities. It is effectively impossible to keep a current understanding of what packages are included in your project, what all of them do and what their state is so describing it as chaotic does not feel that wrong to me.
Even that is misleading. Try npm ls --parseable | wc -l.
On a random (old) client project I had (which lists 18 deps and 27 dev deps), the count balloons out from 619 using ls node_modules | wc -l to 1024 using npm ls --parseable | wc -l and its not even “complete” because it failed to compile something, so who know how many dependencies aren’t actually installed.
Could one of the most prominent actors in the Javascript/NodeJS community step-up and build a curated standard-lib module that contains all the utility functions many projects rely on? The Guava of Javascript kind of?
For somebody on the sidelines of Node, what’s the latest recommendation to avoid this? I’ve heard NPM supports namespaced modules like Go does, but is that predominant? Can it reference a package checksum/signature? Is there any way to control 3rd and 4th tier deps?
This is pretty much it. Working with NodeJS/NPM even at distance is like having a romantic dinner in a fucking dumpster fire.
I’ve had the displeasure of working (backend/ops) on some projects where they insisted on flavour of the week frontend stuff. So there was node, npm. A dozen other interdependent build tools. I ended up helping the guy running the project track down a bug - basically a downstream library had changed something and it broke stuff. That same module was included somewhere like 11 or 12 times, all slightly different versions. The dependency tree for that project’s JS files (keeping in mind this was frontend only) was like one of those “European royalty” family tress: everything is somehow related, and just looking at it makes you want to scrub your eyes with tobasco.
This isn’t news to anybody who has to be involved with Node.js development. This is also an NPM issue, not a Node.js issue, despite how tightly coupled they are. There’s alternative package managers out there (Yarn) that try and solve these problems.
Do the alternate package managers use an alternate package repo?
An “ecosystem” that encourages and embraces “modules” to check if a number is even, and then another one to check if a number is odd, needs more than a different CLI tool to download the same shit code.
I don’t disagree with that last statement of yours at all. I’m just saying Node.js isn’t really the problem in and of itself, and there’s people in the Node.js world who are actively looking for better solutions to these problems.
And to be fair, its not really shit code, its just run of the mill average code.
I’d argue that it is shit code. Shit doesn’t mean its faulty. Good code is both functional and uncomplicated.
The
isEven/isOddshit is deliberately written to use a bitwise operation which makes it a lot less obvious what it’s doing, for this reasoning:And frankly, JavaScript itself is part of the problem - “is-number” has 10 million downloads for the last 7 days.
This microoptimization is silly because JavaScript runtimes recognize modulus by a power of 2 and change the generated code accordingly.
The V8 code to do this is here.
The ChakraCore code for this is here, search for isModByPowerOf2.
SpiderMonkey appears to do something similar but the code is harder to track down.
You raise a good point. In that module’s repo, there’s a comment that reads:
Yarn is an alternative dependency resolver, but it uses the exact same package ecosystem as NPM.
Isn’t this just a frustration with the fact that so many packages depend on so many packages? Because the same rogue actor stuff would apply to any tree dependency self-published system. See your local Gemfile. Do you really know what’s being installed?
So what’s the proposed fix? Stop depending on so many packages?
Perhaps there should be a line drawn for the major packages, to use less of these tiny modules.
But this is also part of the beauty of single responsibility packages. Composing extremely simple parts is all we really need, right?
Think about all of the tiny programs you run on your Unix machine of choice. Perhaps those are also at fault for the “too small to be worth it” sin.
Again, where do we draw lines?
The elm package manager will refuse to publish a package that violates semantic versioning. So if you’re a malicious actor, changing the API of some package for nefarious purposes, you’d be doing a bit of a hard job because since functions can’t change their signatures without you bumping the major version and since most elm-packages.json files will not update major versions automatically, it becomes harder to ship evil code. Also, a simple elm-package diff command would tell you what changed between versions, making it easier for you to audit the code. There is a lot of beauty in it. It is not evil proof, but it makes it a bit harder.
I wonder if you can push this to an extreme, where each public function is independently versioned. Then you could upgrade to a new version even if there are breaking changes, as long as they’re not in any part of the API you use. I’m not sure if it would be useful but its a fascinating idea.
This is super interesting. I wonder how much this friction reduces velocity and quick progress, especially for younger developers. But I like this.
I just took a look at the Gemfile for one of my websites and there are a lot of gems but ~80% come from the rails team and most of the rest are trusty sources like amazon. This is not even close to as bad as when I created a blank Vue project that already had thousands of packages many about 3 lines long.
Good heuristic. Still possible to see bad stuff come down the pipe, but perhaps not as common just by way of community.
Could we imagine the same thing happening, though, in a parallel universe? I think so.
Then again, standard libs for Ruby do a lot of the small stuff for you. Even/odd come to mind.
Ruby seems to have lots of mega libs that do big tasks but JS devs seem to have taken the path of making a new package for every function. I do wonder how often JS devs add 2 packages that do the same thing because they forgot about the last one. The result is also that questions online about ruby often tell you to use methods built in to ruby or ActiveSupport where as JS questions either get you to build it yourself or use jquery.
In defense of ActiveSupport, you can include individual pieces of it at whatever granularity you like, and the documentation explicitly calls this out to you at the very beginning.
Even in projects that are not web-related, I sometimes bring ActiveSupport along but will only
require 'active_support/core_ext'and add things later if I need them as the project grows (often things liketime_with_zoneornumber_helper).I think that it’s a good way to balance “provides a lot of stuff” and “forces you to use all of its stuff”. I want to complain that the rest of Rails’ sub-projects don’t provide the level of utility when included gradually, but I also appreciate that that is the whole bargain: Rails’ perceived productivity boosts are derived from a tight coupling that pays dividends, you as long as you go with the Rails Way.
Edit/Afterthought: Over the years, some of the better Ruby libs to hit this balance of providing a lot but allowing for gradual or clean interop (with other libs, or with your application code) have been those authored or worked on by Piotr Solnica.
What I do in my Python work is generally be willing to depend on large, tested projects with well established, thriving communities that have a good history of responding to security incidents. These packages usually do not depend on other packages or when they do, on well established ones (e.g. requests library).
I might include other, less known packages, if I intend to use most of their functionality and if it is not easy to replicate on my own. Otherwise I write my own version or copy with attribution and license only the relevant parts.
This is not perfect, but gives me some confidence that things will be fine.
It is also an approach that completely breaks down in my Javascript/Typescript work because of so many packages being pulled in and everything seemingly depending on everything else (yes, I am exaggerating).
The problem already starts with licenses. For many apps for example you’ll have to list the licenses of all used libraries, and the libraries, including all transitive dependencies in the UI.
For my Android app that’s a few dozen overall, and it took only a few hours to do this. For even the smallest Angular project, this is insanity.
So if you want compliance, you’ll need to throw the entire ecosystem out right now. Or you just violate the licenses. Often even projects such as angular have dependencies that have no license.
I just tested this. In the old client project I mentioned earlier,
find node_modules/ -iname "license"gives me 743 results.I stumbled upon this gem the other day: https://github.com/nodejs/node/blob/master/deps/npm/node_modules/osenv/node_modules/os-tmpdir/index.js
It’s a module included in npm, that duplicates part of the ‘os’ api that has been in the standard node library since 0.9.9.
I’ll see your ‘duplicating the standard lib’ and raise you a ‘defining the alphabet’[1] and a ‘defining each ANSI colour individually’[2] :
The first one is ridiculous. JS apparently has no range() method, perhaps that would have been worthwhile writing as a module. But no. Three arrays of characters.
The second one is just ridiculous modularisation. A single
ansi-colours(or-colorsif you like your modules “de-queened”) module could export functions and constants (e.g.:ansi(message, ansi.YELLOW)or similar).This is the same guy who wrote the ‘isEven’/‘isOdd’ I mentioned earlier. He has ~1400 packages on NPM, and based on the ones I’ve looked at in the last few days, just his own packages have the typical nodejs dependency tree that mimics crab grass.
1: https://github.com/jonschlinkert/alphabet/blob/master/index.js
2: https://github.com/jonschlinkert?utf8=✓&tab=repositories&q=ansi&type=&language=
Yeesh.
They’re depending on it via a glob matcher by the same author. Pretty much no one else uses that package. That’s the thing with tiny packages, you can often consider them parts of a bigger package by the same author.
And there’s not that many of these tiny packages.
Every once in a while someone writes another version of the same tired rant about the npm ecosystem that extremely exaggerates the problem and repeats the same “omg horrible code we can’t code 1000 layers deep millions of dependencies aaaaaaaa left pad left pad” mantra. COME ON. It’s not true. The dependency trees in Node projects are not that deep and not that wide, and one line packages are not that common.
I hate how it’s trendy to hate on JavaScript/Node/npm now. They absolutely do not deserve it. PHP actually sucked because of awful design. JS historically was closer to having no design, which does mean more dependencies for common simple things, which does have its problems, but is not a “disaster” or “shit” or anything like that. Actually that has its upsides too. “The standard library is where libraries go to die”, so not having a huge one means less old junk floating around that you’ll be tempted to use instead of a good module since that one is Built In™.
The author of this blog post is the same guy who ripped off James Iry’s classic A brief, and mostly wrong, history of programming languages with only a kind of backhanded attribution at the end of his (much less funny) post. He basically writes garbage blogs about programming, and somehow manages to get people talking about them. This post was probably taken directly from the Reddit discussion of this topic from a few days ago - certainly he hasn’t covered anything that wasn’t in the most upvoted comments of that thread.
PHP still has warts but a lot of improvements have been made to solve actual real world issues (i.e. type hinting, performance, removing unsafe/hard-to-use functionality, etc).
JavaScript still has the core problems it’s had since I was a student, at the beginning of this century. How can I write a javascript method that accepts two integers and.. well, knows they’re integers?
Oh. Right, we’re back to that guys “is-number” package, aren’t we?
List of package for a project I am working on which has ~50 dependencies in package.json and <20 dev dependencies (most of which required by couple of frameworks I use):
My typical Python project usually has less than 20 external dependencies. That’s an order of magnitude difference. Admittedly these are not directly comparable projects, but it is also quite typical that my Javascript/Typescript work that depends on external frameworks quickly accumulates other packages numbered in few hundreds.
I find it difficult to argue that JS developers (this is really not a Node issue) are not more likely to depend on other packages than some other programming communities. It is effectively impossible to keep a current understanding of what packages are included in your project, what all of them do and what their state is so describing it as chaotic does not feel that wrong to me.
Even that is misleading. Try
npm ls --parseable | wc -l.On a random (old) client project I had (which lists 18 deps and 27 dev deps), the count balloons out from 619 using
ls node_modules | wc -lto 1024 usingnpm ls --parseable | wc -land its not even “complete” because it failed to compile something, so who know how many dependencies aren’t actually installed.Right, thanks. Then the number is 1474 and in my case it also isn’t complete.
Could one of the most prominent actors in the Javascript/NodeJS community step-up and build a curated standard-lib module that contains all the utility functions many projects rely on? The Guava of Javascript kind of?
https://lodash.com
For somebody on the sidelines of Node, what’s the latest recommendation to avoid this? I’ve heard NPM supports namespaced modules like Go does, but is that predominant? Can it reference a package checksum/signature? Is there any way to control 3rd and 4th tier deps?
It’s pretty easy…don’t use Node or NOM.
I’m not kidding.
This is pretty much it. Working with NodeJS/NPM even at distance is like having a romantic dinner in a fucking dumpster fire.
I’ve had the displeasure of working (backend/ops) on some projects where they insisted on flavour of the week frontend stuff. So there was node, npm. A dozen other interdependent build tools. I ended up helping the guy running the project track down a bug - basically a downstream library had changed something and it broke stuff. That same module was included somewhere like 11 or 12 times, all slightly different versions. The dependency tree for that project’s JS files (keeping in mind this was frontend only) was like one of those “European royalty” family tress: everything is somehow related, and just looking at it makes you want to scrub your eyes with tobasco.