1. 12

  2. 4

    npm audit (and tools like it) has the unenviable job of trying to communicate urgency to update while not really being able to do environment-sensitive risk scoring of vulnerabilities. Vulns may have a CVSS Base score, which is good, but as this article shows sometimes a vuln that scores High for CVSS Base is actually not a vuln at all in your environment.

    This issue pops up for static code analysis tools too, where they give a risk rating for their findings (not yet vulns because they haven’t been validated and false positive rates for static code analysis are generally high) based on the confidence the analyzer has in its own analysis, and the usual expected risk rating for the category of weakness represented. So if the analyzer is confident, and the finding is a null pointer dereference (usually High or Critical depending on the tool), you’ll get a High or Critical risk rating. In that case it’s even more muddy because 1) it’s not a CVSS Base score because the tool doesn’t have the context to answer all the necessary questions for producing that score, 2) it’s not particularized to even the specific finding, let alone the broader environment in which the software under analysis will be used.

    Some potential paths toward solutions:

    • Be clearer about risk ratings, and whether they’re from a CVSS Base score and whether they include Temporal or Environmental scores.
    • Enrich the language around risk ratings to differentiate what I’ll call general risk from your risk.

    Now, in this specific case, a lot of findings here are of the sort found by taint analysis. Some path of execution exists in the code between user-controllable input and insecure usage of the input, without validation / cleaning of the input in between. Insecure regexes leading to potential denial of service are an example of this class of weaknesses. These are a problem if your users are untrusted, but in the context shown in the post, you’re the user and you probably trust yourself. So, one idea would be to enable users to tell npm audit that they consider users to be trusted, and for npm audit to silence reports for vulnerabilities tied to CWE (Common Weakness Enumeration) IDs which fit this mold.

    In that situation, if you tell npm audit “I consider users to be untrusted” or you say nothing, all vulns are reported; but if you say “I consider users to be trusted,” then npm audit doesn’t display vulns like the ones in this blog post.

    I definitely don’t agree that npm audit is “broken by design.” I think it’s trying very hard to improve the current situation in a challenging area, and could improve. Communicating about security with end users and developers is very hard. You want them to take certain actions, without burning out taking those actions, so that they’ll continue to take them in the future.

    1. 4

      The fact that it’s an awful job and they’re trying very hard doesn’t negate any of the results of their actions, that nobody is paying attention to these bogative “errors”.

      1. 1

        It looks like there are two problems here. The first is the same as for things like pkg audit in FreeBSD: For a lot of classes of vulnerability, it’s really only a problem if you have two bugs. For example, if a regular expression library generates a DFA for regex and some fairly short inputs can produce 32 GiB DFA, that’s a bug but it’s not really a vulnerability. If the API doesn’t allow specifying a maximum size for the generated DFAs then that’s a poor API design and a security weakness, but not necessarily a vulnerability. If the consumer of that API does not validate that the regex strings that it passes to the library are safe then that’s a vulnerability (made difficult to fix if the library doesn’t expose an API for checking if a given regex is safe).

        This is particularly problematic for large libraries. For example, I very often have every single Python or PHP program on a system trigger pkg audit warnings because they depend on some massive library and that library has a vulnerability like this. It’s impossible for me to tell if that function is actually reachable in the things I have installed, let alone whether the use of the function is a problem. If it’s a bug in a PHP package and it’s from arcanist then the odds of it actually being exploited are tiny: arcanist only interacts with trusted phabricator installs and so the attacker would have to either compromise phabricator or be able to trigger the bug from the diff. If it’s in something network-facing, it may be more of a problem.

        The second is the reporting severity. It’s bad if an attacker can make a service crash by providing crafted input, but that’s much less of a problem if it’s client side. If a user can crash their own browser or local app, that’s mildly annoying. If they can crash the server, that’s a big problem. A lot of these sound like only the developer can crash the system and there are so many other ways that a developer can crash things that it really doesn’t matter.

      2. 1

        To be honest, I find npm audit’s messages to be a somewhat useful proxy for excessive complexity. If a project has so many dependencies that the developers can’t keep npm audit happy, I probably don’t wanna use the library. This, plus looking at the number of deps in node_modules, has guided me to Svelte rather than React, Preact or Vue, and I couldn’t be happier.