Literally all the CVE notifications GitHub spammed me with were complete nonsense that didn’t apply to my code or wasn’t relevant to how my applications are used. 99% of this stuff is brainless busywork. I can’t wait for when they make GPT write spurious vulnerability reports for everyone’s repositories, sounds like an idea managers would love. Stuff like this happens when process is elevated to be more important than results.
There were some discussions of this in another thread about the Go notification system. The root of the general problem (ignoring cases like the article, though this would probably fix that as well) is that a package is totally the wrong granularity for this. Let’s say there’s a vulnerability in Curl. Does it affect my program? That depends on whether I use curl with untrusted URL inputs, whether it’s talking to untrusted servers, and whether I even hit that code path at all. Currently, it’s good as a first step for a human to diagnose but no use for automation. To do it well, you probably need reachability analysis and taint tracking.
Yeah, over a period of 4 years getting deploys blocked on a regular basis by the vulnerability scanner at work, I can’t recall a single time we saw a legitimate vulnerability that applied to our application.
A couple of years ago, I had a job at a startup that delivered the hosted version of its product as a set of helm charts and docker images. The one client that we did this for had a hard requirement that all images had either 0 vulnerabilities, or that said vulnerabilities were “explained”.
It became then my job to go through every single CVE from the docker images and either fix it, by updating the image, or explain it away.
Quickly we realized that updating wasn’t exactly doable. The product used CUDA base images and updating anything on those was very time consuming. Just updating the base image wasn’t worth it either, because it wouldn’t fix all vulnerabilities, and messed up the ML models (which is why we need the CUDA images).
So I was left with just manually reviewing CVEs, and that’s as exciting as you’re imagining. Although I did learn quite a bit in the process, so, that was nice.
The main lesson I took from this was that a lot of CVEs are kind dumb, and you really need to read them and understand them to make an informed decision. Here’s a short list of things I remember finding, with the vulnerability, and the reason why it didn’t matter:
calling certain tar commands in a malicious file could result in denial of service: we never ran any tar commands or even dealt with files, and worse case scenario the container would crash, but no data would be compromised
sshd with certain configurations could lead to remote execution: The containers didn’t run sshd, and the pods and services didn’t expose the ssh port. Even if a malicious payload tricked the container into running sshd, it couldn’t be accessed outside the container
Bug on https handling in python libraries could lead to denial of service: we terminated TLS at the ingress, and even then, denial of service wasn’t a big deal, data integrity was the main concern.
The only real one I can remember was something to do with HTTP or JSON handling that could lead to remote execution. But even that we considered safe, because:
all affected routes required authentication first, via another microservice
it was an internal app, accessible only in the intranet
So an attacker would have to gain access to the intranet and get valid credentials to our system before being able to exploit the vulnerability. But by them, said attacker would have much better targets available.
So, basically, automatic vulnerabilities scans are really pretty stupid, in the sense that they have no context. Following them blind goes as well as you would expect, i.e., badly, lots of wasted effort. And yet, none of the vulnerability scanning solutions we evaluated at the time had anything for managing those vulnerabilities, like marking them as false positives, adding notes, etc. They all solved the easy half of the problem and called it a day.
Maybe there are some better tools nowadays (the job in question was in 2021) for that kinda of process, or at least people are realizing that said tools are needed.
It seems that a big part of your problem was having a load of things in the container that you didn’t need. This is unfortunately common because containers are defined in terms of layers with a single parent, so you can’t have a layer that just contains the CUDA run-time libraries and nothing else that you can then apply to a mostly empty base, you have to start with something like a minimal Ubuntu install, where maybe 60% is stuff you will never use, then apply the CUDA layers, then add the stuff that you actually do want.
Absolutely. I suppose if we were to hand craft our base image, we could get a clean, CUDA only image, but being a 20 people startup, that wasn’t considered a wise investiment of engineering effort.
Which is why, as much as I dislike go as a developer, as a SRE, deploying go stuff is a fucking bliss.
And this is why FROM nixos/nix is my favorite way to start a Dockerfile. It’s not just that we can use e.g. vulnix on the built container, but also that we start with nothing inside the container besides the ability to add new packages via Nix. I would imagine, as a Go user, that you also like FROM scratch since you probably don’t need to add new packages!
The problem is that, at least in my experience, most people are not creating images from scratch themselves, only using available images, or adding small tweaks to them. And it’s kinda how it should be, really.
But the most common base images are basically full blown Linux. Even the ones supposed to be lightweight, like alpine, or debian-slim.
90% of the time people just wanna deploy some code, they don’t want to figure out every single linked library, config file, directory, etc, that their code needs so they can build an image from scratch with just the absolutely necessary. And again, they shouldn’t have to.
The ultimate solution will have to be, eventually, better tools, that create truly minimal images from just the code, and maybe some config.
The vulnerability notifications are junk even working as intended.
GitHub still believes CVSS scores make sense, and therefore it’s a critical vulnerability, 9.8 out of 10, that if you discover there’s an undocumented method called __private_get_type_id__, choose to implement it manually, and implement it incorrectly, and give attacker a way to control it, then you have a bug. They spammed people with notifications over this.
That’s understandable given the incentives. If you give a vulnerability too high a score, some people will patch it unnecessarily and you’ve made some work for other people. If you give it too low a score and an attacker finds an exploit that does more than you said was possible then people who are exploited will complain loudly that you said they didn’t need to.
This is why memory-safety bugs are usually treated as high severity: they allow the attacker to put the program in a state that is outside of the language’s abstract machine. No one wants to enumerate the set of possible states that it could enter (for all possible compilers and compiler flags) and declare that they’re all safe and no one wants to guess and for an attacker to be right.
I have a lot of sympathy with the OpenBSD view that the severity of a vulnerability is a property of the intelligence of the attacker far more than it is a property of the bug.
I think this incentive model applies to traditional, distro-centered security workflow, where there’s OpenSSL-announce mailing list and what not.
But I think recently we observe a raise of qualitatively different workflow, centered around language package managers. Specifically, these days I see “666 critical vulnerabilities” every time I do npm whatever, and then there’s a false dependabot alarm once in a while.
To me it seems that incentives there are fueled by demand for security theater. There’s a vague uncertainty about supply chain attacks, so me must do something about it. Automatically spamming everyone with notifications is quite a lot of something!
In this model, extra noise is exactly the target function we are maximizing.
Yes, but I don’t think the problem in this case was that the memory-unsafety of the bug was more benign than the 9.8/10 CVSS score suggests. The problem is that it happens in a super super niche misuse of that library only, and CVSS has no way to quantify this kind of thing. If this was C or C++ instead of Rust, reporting “if you misuse this hidden, undocumented library API, you can break safety invariants” would probably not even be considered a bug, let alone a 9.8/10 CVE.
Edit: I think what’s missing for me from CVSS is maybe something like an “exposure score” that answers “given somebody uses the affected piece of software, how likely are they to be exposed to this vulnerability?”.
Where “this is exploitable in the default configuration shipped by Debian, Ubuntu and RHEL” or something is a 10/10 and “this is exploitable if you really go out of your way to create a weird configuration, but we don’t know of any instances of anybody doing this in the real world” is a 1/10.
I don’t disagree that the bug was the wrong category, I’m just pointing out that the consequences for giving something too low a rating are significantly worse than giving it too high a rating and, with that set of incentives, the system will eventually tend towards giving everything ‘10/10: maximum severity’ rating.
I got the impression there is some support somewhere to write coccinelle/semgrep style code matches to check if a security advisory needs action by a usage of the buggy software. I.e. then Dependabot would only notify users of a library that actually use the vulnerable part of the library.
I thought I had seen people talk about Github Advisories / Dependabot using such a matcher for some CVEs. Anyone knows if that is implemented or has a pointer to docs or an example? But maybe it was only a rule for https://github.com/returntocorp/semgrep .
Though then there are denial of service bugs like https://github.com/advisories/GHSA-f8vr-r385-rh5r where for some users of the library there is nothing else to deny. Probably not worth it to make that fact computer legible, just to avoid reading up on a few security issues to know you don’t need to act.
These CVEs seems to pop up every once in a while. Remind me of https://unit42.paloaltonetworks.com/jsonwebtoken-vulnerability-cve-2022-23529/ where you did need to execute arbitrary code to create code execution. If anything, CVE spam is a good reason to manage your dependencies and keep them low, because compliance. Otherwise it’s a whole lot time wasted…
Literally all the CVE notifications GitHub spammed me with were complete nonsense that didn’t apply to my code or wasn’t relevant to how my applications are used. 99% of this stuff is brainless busywork. I can’t wait for when they make GPT write spurious vulnerability reports for everyone’s repositories, sounds like an idea managers would love. Stuff like this happens when process is elevated to be more important than results.
There were some discussions of this in another thread about the Go notification system. The root of the general problem (ignoring cases like the article, though this would probably fix that as well) is that a package is totally the wrong granularity for this. Let’s say there’s a vulnerability in Curl. Does it affect my program? That depends on whether I use curl with untrusted URL inputs, whether it’s talking to untrusted servers, and whether I even hit that code path at all. Currently, it’s good as a first step for a human to diagnose but no use for automation. To do it well, you probably need reachability analysis and taint tracking.
Eclipse Steady is a project I’m watching in that regard.
Yeah, over a period of 4 years getting deploys blocked on a regular basis by the vulnerability scanner at work, I can’t recall a single time we saw a legitimate vulnerability that applied to our application.
A couple of years ago, I had a job at a startup that delivered the hosted version of its product as a set of helm charts and docker images. The one client that we did this for had a hard requirement that all images had either 0 vulnerabilities, or that said vulnerabilities were “explained”.
It became then my job to go through every single CVE from the docker images and either fix it, by updating the image, or explain it away.
Quickly we realized that updating wasn’t exactly doable. The product used CUDA base images and updating anything on those was very time consuming. Just updating the base image wasn’t worth it either, because it wouldn’t fix all vulnerabilities, and messed up the ML models (which is why we need the CUDA images).
So I was left with just manually reviewing CVEs, and that’s as exciting as you’re imagining. Although I did learn quite a bit in the process, so, that was nice.
The main lesson I took from this was that a lot of CVEs are kind dumb, and you really need to read them and understand them to make an informed decision. Here’s a short list of things I remember finding, with the vulnerability, and the reason why it didn’t matter:
The only real one I can remember was something to do with HTTP or JSON handling that could lead to remote execution. But even that we considered safe, because:
So an attacker would have to gain access to the intranet and get valid credentials to our system before being able to exploit the vulnerability. But by them, said attacker would have much better targets available.
So, basically, automatic vulnerabilities scans are really pretty stupid, in the sense that they have no context. Following them blind goes as well as you would expect, i.e., badly, lots of wasted effort. And yet, none of the vulnerability scanning solutions we evaluated at the time had anything for managing those vulnerabilities, like marking them as false positives, adding notes, etc. They all solved the easy half of the problem and called it a day.
Maybe there are some better tools nowadays (the job in question was in 2021) for that kinda of process, or at least people are realizing that said tools are needed.
It seems that a big part of your problem was having a load of things in the container that you didn’t need. This is unfortunately common because containers are defined in terms of layers with a single parent, so you can’t have a layer that just contains the CUDA run-time libraries and nothing else that you can then apply to a mostly empty base, you have to start with something like a minimal Ubuntu install, where maybe 60% is stuff you will never use, then apply the CUDA layers, then add the stuff that you actually do want.
Absolutely. I suppose if we were to hand craft our base image, we could get a clean, CUDA only image, but being a 20 people startup, that wasn’t considered a wise investiment of engineering effort.
Which is why, as much as I dislike go as a developer, as a SRE, deploying go stuff is a fucking bliss.
And this is why
FROM nixos/nixis my favorite way to start a Dockerfile. It’s not just that we can use e.g. vulnix on the built container, but also that we start with nothing inside the container besides the ability to add new packages via Nix. I would imagine, as a Go user, that you also likeFROM scratchsince you probably don’t need to add new packages!The problem is that, at least in my experience, most people are not creating images from scratch themselves, only using available images, or adding small tweaks to them. And it’s kinda how it should be, really.
But the most common base images are basically full blown Linux. Even the ones supposed to be lightweight, like alpine, or debian-slim.
90% of the time people just wanna deploy some code, they don’t want to figure out every single linked library, config file, directory, etc, that their code needs so they can build an image from scratch with just the absolutely necessary. And again, they shouldn’t have to.
The ultimate solution will have to be, eventually, better tools, that create truly minimal images from just the code, and maybe some config.
The vulnerability notifications are junk even working as intended.
GitHub still believes CVSS scores make sense, and therefore it’s a critical vulnerability, 9.8 out of 10, that if you discover there’s an undocumented method called
__private_get_type_id__, choose to implement it manually, and implement it incorrectly, and give attacker a way to control it, then you have a bug. They spammed people with notifications over this.That’s understandable given the incentives. If you give a vulnerability too high a score, some people will patch it unnecessarily and you’ve made some work for other people. If you give it too low a score and an attacker finds an exploit that does more than you said was possible then people who are exploited will complain loudly that you said they didn’t need to.
This is why memory-safety bugs are usually treated as high severity: they allow the attacker to put the program in a state that is outside of the language’s abstract machine. No one wants to enumerate the set of possible states that it could enter (for all possible compilers and compiler flags) and declare that they’re all safe and no one wants to guess and for an attacker to be right.
I have a lot of sympathy with the OpenBSD view that the severity of a vulnerability is a property of the intelligence of the attacker far more than it is a property of the bug.
I think this incentive model applies to traditional, distro-centered security workflow, where there’s OpenSSL-announce mailing list and what not.
But I think recently we observe a raise of qualitatively different workflow, centered around language package managers. Specifically, these days I see “666 critical vulnerabilities” every time I do npm whatever, and then there’s a false dependabot alarm once in a while.
To me it seems that incentives there are fueled by demand for security theater. There’s a vague uncertainty about supply chain attacks, so me must do something about it. Automatically spamming everyone with notifications is quite a lot of something!
In this model, extra noise is exactly the target function we are maximizing.
Yes, but I don’t think the problem in this case was that the memory-unsafety of the bug was more benign than the 9.8/10 CVSS score suggests. The problem is that it happens in a super super niche misuse of that library only, and CVSS has no way to quantify this kind of thing. If this was C or C++ instead of Rust, reporting “if you misuse this hidden, undocumented library API, you can break safety invariants” would probably not even be considered a bug, let alone a 9.8/10 CVE.
Edit: I think what’s missing for me from CVSS is maybe something like an “exposure score” that answers “given somebody uses the affected piece of software, how likely are they to be exposed to this vulnerability?”. Where “this is exploitable in the default configuration shipped by Debian, Ubuntu and RHEL” or something is a 10/10 and “this is exploitable if you really go out of your way to create a weird configuration, but we don’t know of any instances of anybody doing this in the real world” is a 1/10.
I don’t disagree that the bug was the wrong category, I’m just pointing out that the consequences for giving something too low a rating are significantly worse than giving it too high a rating and, with that set of incentives, the system will eventually tend towards giving everything ‘10/10: maximum severity’ rating.
But this ass-covering devoid of common sense is my issue with the CVSS. It’s a broken system with perverse incentives. It’s worse than useless.
I got the impression there is some support somewhere to write coccinelle/semgrep style code matches to check if a security advisory needs action by a usage of the buggy software. I.e. then Dependabot would only notify users of a library that actually use the vulnerable part of the library.
I thought I had seen people talk about Github Advisories / Dependabot using such a matcher for some CVEs. Anyone knows if that is implemented or has a pointer to docs or an example? But maybe it was only a rule for https://github.com/returntocorp/semgrep .
Though then there are denial of service bugs like https://github.com/advisories/GHSA-f8vr-r385-rh5r where for some users of the library there is nothing else to deny. Probably not worth it to make that fact computer legible, just to avoid reading up on a few security issues to know you don’t need to act.
These CVEs seems to pop up every once in a while. Remind me of https://unit42.paloaltonetworks.com/jsonwebtoken-vulnerability-cve-2022-23529/ where you did need to execute arbitrary code to create code execution. If anything, CVE spam is a good reason to manage your dependencies and keep them low, because compliance. Otherwise it’s a whole lot time wasted…