1. 13
  1. 4

    This is an important topic but this article was a struggle to understand and does not clearly and accurately propose a solution. Maybe I never understood lock files?

    NPM can already record those versions, in a lock file. But npm install of a new package does not use the information in that package’s lock file to decide the versions of dependencies: lock files are not transitive.

    Why in the name of everything that is holy are NPM lock files not transitive? What does a non-transitive lock file even mean? Am I misunderstanding the article or NPM - doesn’t a lock file permanently freeze the full dependency graph to fixed versions, or do I only get that with npm shrinkwrap?

    Anyone running modern production systems knows about testing followed by gradual or staged rollouts, in which changes to a running system are deployed gradually over a long period of time, to reduce the possibility of accidentally taking down everything at once.

    Why even mention this? I was teased into maybe seeing a proposal for gradually revealing package updates from consumers, or maybe automatically running automated tests, but neither were proposed. I think the Go versioning proposal is weak because it still leaves you prey to arbitrary version upgrades.

    NPM’s design choice is the exact opposite. The latest version of colors was promoted to use in all its dependents before any of them had a chance to test it and without any kind of gradual rollout. Users can disable this behavior today, by pinning the exact versions of all their dependencies. For example here is the fix to aws-cdk. That’s not a good answer, but at least it’s possible.

    Why is this not a good answer? Later in the article the author argues people should use npm shrinkwrap….which is this answer. “

    Is this the crux of my misunderstanding? Isn’t a lock file equivalent to expressing specific version dependencies for all depenendencies in the full dependency graph?

    Other language package managers should take note too. Marak has done all of us a huge favor by highlighting the problems most package managers create with their policy of automatic adoption of new dependencies without the opportunity for gradual rollout or any kind of testing whatsoever.

    Come on, let’s call a spade a spade here. You need to freeze your dependency graph with a lock file. If npm plays weird semantic games with “oh lock files don’t really lock dependencies” then just Docker the whole thing and wrap it in a lead canister. If you depend on an ecosystem that doesn’t honor freezing dependencies then freeze them yourself.

    1. 9

      Am I misunderstanding the article or NPM - doesn’t a lock file permanently freeze the full dependency graph to fixed versions, or do I only get that with npm shrinkwrap?

      The article. You are correct that your project’s package-lock.json freezes the entire dependency tree for your project. The paragraphs at the end of the article starting “NPM also has an npm shrinkwrap…” are somewhat confusingly written.

      Say I set up a new project and I want to use library A, which has a dependency on library B.

      When I run npm install --save A, for the first time, npm will fetch the latest allowed versions (according to version bounds in the dependencies field of package.json) of A and B. It’ll record in package-lock.json whatever versions of A and B it fetched.

      Next time I run npm install or npm ci in my project, it’ll install exactly the same versions of A and B again that it installed last time. (assuming I haven’t changed dependencies in package.json)

      What the article is complaining about is this: package A may contain its own package-lock.json. npm did not consult A’s package-lock.json to decide which version of B to pick when I ran npm install --save A. Only the package-lock.json at the root of the project (if there is one) was consulted. On initial project setup, I don’t have a package-lock.json yet at the top level, so I’m getting all fresh new versions of everything and I’m going to get the sabotaged version of color since it’s the newest one.

      The article proposes that, when you run npm install --save A, npm should check for a package-lock.json inside the package ‘A’ and, if one is present, use the same version of B that A’s package-lock.json has, because A was probably QA’d with that version of B.

      1. 4

        Isn’t this what package.json is for? Or is the problem that everyone uses overpermissive dependencies?

        1. 3

          No. package.json only specifies dependencies one level deep and only specifies names and version numbers. It’s pretty loose. The lock file has content hashes too and specifies the entire tree.

          1. 3

            But hang on, if you check the entire tree of every dependency, aren’t you pretty much always going to get conflicting information? Isn’t that the whole point of package.json that we specify looser dependencies so that we can, SemVer permitting, actually get two packages to agree on which version of a dependency they can both live with? It seems to me that when you give up on the SemVer notion of non-breaking differences, except for very trivial cases, you pretty much give up on versioning.

            1. 2

              Yes, there is a tradeoff here and I suspect that people who wrote npm did actually think about this very issue.

              FWIW you can install mutually conflicting libraries in npm as sub dependencies. If A depends on C==1.0 and B depends on C==2.0, and you install both A and B, you get a node_modules tree like:

              • node_modules/A
              • node_modules/A/node_modules/C
              • node_modules/B
              • node_modules/B/node_modules/C

              So over constraining dependency versions doesn’t necessarily break everything, even though it’s not often what you want.

        2. 2

          Thank you!

      2. 2

        In case people missed the link in the article for “high-fidelity builds”: https://research.swtch.com/vgo-mvs

        I think this article doesn’t quite make sense until you read Russ Cox’s theory of minimal version selection.