npm is so problematic. Firstly, npm install is not even deterministic which means everytime you on board someone it’s possible the contents of their node_modules is different feom yours. npm install definitely shouldn’t touch your lock file and there should be an npm unlock command.
The defining feature of any package manager is not speed, it is reliability. Speed happens as a result of good design.
I recommend not using npm if you can avoid it, my understanding is that yarn is significantly better with regards to these mission critical features, unfortunately i haven’t had time to do more than cursory research about switching all of our js/node projects to yarn.
I would be curious how others here have solved/avoided this problem.
We’re using yarn. It’s stayed out of our way, so far.
npm does locally cache packages these days so they only need to be downloaded once, but you might be more interested in looking at the next generation package management tools the npm team is currently working on.
So on the one hand this is indeed an issue but on the other hand it’s a simple solution that just works. When the left-pad debacle happened the place I was working at the time didn’t see any downtime. The reason was we archived the npm modules folder with a hash based on package.json. Since our dependencies hand’t changed our CI and deployment was unaffected because we had everything cached.
The big problem isn’t node_modules, npm, or anything language or tool related.
It’s that too much developers are used to just npm install whatever they need without thinking if they need that big dependency (without checking the size), if there’s a lightweight alternative or if could be better to write the solution itself.
I’ve been recently working on a Node project and we absolutely can do a better job at choosing our dependencies (or writing them).
The node_modules problem is much worse on Windows, which is much slower at handling files than Linux or macOS. I don’t have hard numbers on hand from my own testing - but anecdotally, I asked my friend to install a project of mine with yarn on his Windows 10 Surface Book - it took about 5 minutes. On my Ubuntu 18.04 laptop in a new Docker container (no caches), it takes about 60 seconds.
A cursory googling returns this Quora answer with graphs about filesystem performance: https://www.quora.com/Which-is-more-advanced-in-terms-of-speed-and-performance-Linux-or-MS-Windows