Filtering for applicability is great, and should have been a standard. I’m soooo tired of security tools crying wolf. Just like the desktop anti-virus software, they have lost sight of providing security, and became a security theatre and a numbers meta game.
CVE databases are full of low-effort reports, like flagging of uses of RegEx. That’s a slam-dunk “Possible DoS critical vulnerability”.
CVSS scoring rules are designed to be cover-your-ass for security vendors, not for usability by people who need to respond to these reports. The scoring rules are “we can’t know if you have written vulnerable code, but you may have, so that’s a critical vulnerability in the code we imagine you may have written”.
If npm install reports less than 17 critical vulnerabilities, I double-check if the installation went properly.
Hate to say I agree with this. I work in security, and so many of my peers across now several organizations view the reports from these tools as absolute truth rather than raw data/input to further analysis. A vulnerability in a component does not necessarily translate in to a vulnerability in your application, but it is what scanners you and ultimately your customers run will report.
We’ve seriously lost our way. There needs to be more rigor in how these tools are applied. So much focus in “Supply Chain Security” is instead put on shifting responsibility to open source developers who absolutely did not sign up to perform vulnerability verification services for their commercial users.
CVE databases are full of low-effort reports, like flagging of uses of RegEx. That’s a slam-dunk “Possible DoS critical vulnerability”.
The world of security research is a weird one. There’s an incentive to get as many CVEs at as high a severity ranking as possible to bolster your resume. No one in hiring seems to care about quality :(
Supply chain security is hard because the source data is almost always terrible. The full text of the CVE in question is
A prototype pollution vulnerability was found in lodash <4.17.11 where the functions merge, mergeWith, and defaultsDeep can be tricked into adding or modifying properties of Object.prototype.
You can’t make a reliable tool which greps for strings out of a CVE. You need maintainers and researchers to populate data in a structured way such that SCA teams make use of it. The line
To address these issues, SCA tools have to evolve to work for large-scale security teams.
completely misses the point that SCA teams are given garbage to work with. Vulnerability reporting rather than SCA tools need to evolve.
You can’t make a reliable tool which greps for strings out of a CVE. You need maintainers and researchers to populate data in a structured way such that SCA teams make use of it.
Heya, I worked on launching this product. You’re actually completely right, and this is exactly what we’re doing. We have a team of security researchers with an on-call rotation to write high confidence rules for new advisories as they’re announced.
Rules or just curating a list of functions/objects per advisory?
Full-on Semgrep rules. I just spot-checked the most recent one added last night, and it has 55 lines worth of patterns in its definition. (For the record, this supply chain rule repository is currently available only to paid customers.)
Are those rules essentially encoding the affected code as a string match then? We (github) are doing something similar*, but we’re encoding the affected code in the advisory itself so that it’s not tied to any one specific alerting engine. Happy to collaborate with you if you’d like to open your data up.
Semgrep has a pretty full-featured DSL to express a code search pattern: https://semgrep.dev/docs/writing-rules/rule-syntax/ — then this pattern is evaluated on an engine that does symbolic propagation, taint tracking, etc. So it’s a lot, lot more accurate than string matching.
It’s awesome that you’re working on something similar! We actually have a more technical blog post coming out later which does in fact reference the post you linked. Joining forces on this sounds like it would be very beneficial to everyone. Personally I’m imagining the GHSA database listing authoritative Semgrep rules for identifying vulnerable code. Could you reach out to me at bence@r2c.dev? I’ll connect you with the rest of the team.
The suggestion to filter warnings based on reachability analysis reminds me of similar consideration given to conditional compiler warnings in a recent discussion on changing Go’s for-loop syntax (an incredibly considerate discussion in general).
Filtering for applicability is great, and should have been a standard. I’m soooo tired of security tools crying wolf. Just like the desktop anti-virus software, they have lost sight of providing security, and became a security theatre and a numbers meta game.
CVE databases are full of low-effort reports, like flagging of uses of
RegEx. That’s a slam-dunk “Possible DoS critical vulnerability”.CVSS scoring rules are designed to be cover-your-ass for security vendors, not for usability by people who need to respond to these reports. The scoring rules are “we can’t know if you have written vulnerable code, but you may have, so that’s a critical vulnerability in the code we imagine you may have written”.
If
npm installreports less than 17 critical vulnerabilities, I double-check if the installation went properly.Hate to say I agree with this. I work in security, and so many of my peers across now several organizations view the reports from these tools as absolute truth rather than raw data/input to further analysis. A vulnerability in a component does not necessarily translate in to a vulnerability in your application, but it is what scanners you and ultimately your customers run will report.
We’ve seriously lost our way. There needs to be more rigor in how these tools are applied. So much focus in “Supply Chain Security” is instead put on shifting responsibility to open source developers who absolutely did not sign up to perform vulnerability verification services for their commercial users.
[Comment removed by author]
The world of security research is a weird one. There’s an incentive to get as many CVEs at as high a severity ranking as possible to bolster your resume. No one in hiring seems to care about quality :(
Supply chain security is hard because the source data is almost always terrible. The full text of the CVE in question is
You can’t make a reliable tool which greps for strings out of a CVE. You need maintainers and researchers to populate data in a structured way such that SCA teams make use of it. The line
completely misses the point that SCA teams are given garbage to work with. Vulnerability reporting rather than SCA tools need to evolve.
Heya, I worked on launching this product. You’re actually completely right, and this is exactly what we’re doing. We have a team of security researchers with an on-call rotation to write high confidence rules for new advisories as they’re announced.
Rules or just curating a list of functions/objects per advisory?
Full-on Semgrep rules. I just spot-checked the most recent one added last night, and it has 55 lines worth of patterns in its definition. (For the record, this supply chain rule repository is currently available only to paid customers.)
Are those rules essentially encoding the affected code as a string match then? We (github) are doing something similar*, but we’re encoding the affected code in the advisory itself so that it’s not tied to any one specific alerting engine. Happy to collaborate with you if you’d like to open your data up.
*: https://github.blog/2022-04-14-dependabot-alerts-now-surface-if-code-is-calling-vulnerability/
Semgrep has a pretty full-featured DSL to express a code search pattern: https://semgrep.dev/docs/writing-rules/rule-syntax/ — then this pattern is evaluated on an engine that does symbolic propagation, taint tracking, etc. So it’s a lot, lot more accurate than string matching.
It’s awesome that you’re working on something similar! We actually have a more technical blog post coming out later which does in fact reference the post you linked. Joining forces on this sounds like it would be very beneficial to everyone. Personally I’m imagining the GHSA database listing authoritative Semgrep rules for identifying vulnerable code. Could you reach out to me at bence@r2c.dev? I’ll connect you with the rest of the team.
The suggestion to filter warnings based on reachability analysis reminds me of similar consideration given to conditional compiler warnings in a recent discussion on changing Go’s for-loop syntax (an incredibly considerate discussion in general).
We need to rework the foundations so that it becomes tractable.