Very well written! Sorry you had to go through this. Also, I’m glad you could get this issue resolved.
Here is my own DNS horror story where the domain name I was using was taken away from me without any warning or due process: https://susam.net/blog/sinkholed.html.
The underlying cause of the issue in my story was a false-positive during anti-malware operation too. What really surprised me was that an anti-malware foundation had the authority to remove me as the registrant of the domain name and assign someone else without any form of notification to me or the domain name reseller.
Wow… I got lucky by comparison — at least I didn’t lose control of anything, merely got inaccessible for a part of the user base. Extra-judicial domain seizure is really extreme.
The unfortunate thing with these sort of malware detectors is that they operate with an expected false-positive rate. The industry generally thinks that’s fine, because its better to hit more malware than to miss any.
That only works if the malware decector authors are receptive to feedback, though.
CrowdStrike Falcon on macOS and SentinelOne on Windows have cost me so much wasted time as an employee of companies that uses them. Falcon routinely kills make, autoconf, etc. SentinelOne does the same thing when using msys2 or cygwin on Windows.
At least SentinelOne tells me, Falcon tries its best to leave zero useful information on the the host. When processes start randomly being terminated it takes a bit of effort to find out what the hell is actually happening. Often I realized it was Falcon only after some poor desktop security tech gets assigned a ticket and they reach out to me with a lot of confusion around some crazy complex command lines being sent through the various exec calls.
Because of the high frequency in which I encounter the issue, if something randomly fails in a way I don’t expect I immediately suspect Falcon.
My understanding is that signed binaries don’t help - if a binary is rarely run (because it’s just been released, or it’s just not a mainstream tool) there’s a good chance it’ll be detected as malicious no matter what.
I believe it depends on the kind of signature. Last time I checked, companies can buy more expensive certificates which are privileged insofar that the binaries don’t need to be run on a lot of machines to be considered safe.
I’ve worked at a company where the IT was so terrible that their actions bordered on sabotage. They caused more damage and outages than actual hackers would. The anti-virus would delete our toolchains and fill the disk with ominous files until there was no space left. Luckily, they left our Linux machines alone, so we put everything we cared about on Linux (without GUI) and hoped that their lack of know-how would prevent them from messing with those machines. It worked.
When I was in the business of releasing commercial Windows software many moons ago, I also had the same problems with AV Vendors flagging my executable as false positive. The main problem was that I used an exe-protector (Themida) for anti-cracking purposes. It was a lot better with a custom Themida stub and a code signing certificate - the latter being ridiculously complicated to obtain 15 years ago.
Once flagged as malware by one vendor, your programs signature was shared around to other vendors over time (I am not sure about the politics behind this), and you could expect to be identified as malware by more and more vendors over the next days/weeks. Getting whitelisted was basically impossible - many vendors offered some sort of whitelisting process, but it always only concerned a single executable hash. So your next update started the process again.
It’s sad that today we still face the same general problem as back then, but on a totally different level.
I wonder if you could make a case for libel. If I write an article calling someone’s product dangerous, with no evidence, then they have a very strong case if they take me to court and I am likely to be liable for any loss of business and reputation that I’ve caused. I would expect an antivirus company to have similar liability: they are telling a load of your customers that your product is malware.
I am no lawyer, but I cannot imagine that AV companies don’t have extensive legal moat against an allegation like that. But it is an interesting thought.
It’s worth noting that libel laws are quite different in different countries. In England and Wales, libel is treated as an accusation and so the principle of ‘innocent until proven guilty’ is applied to the target of the libel and not the perpetrator. This means that, if they have made written messages appear on users’ computers saying (or even strongly implying) that your program is malware then they would have to demonstrate that this accusation is true. In the US, there is no special case for libel and the person bringing the suit has to prove that the allegation is false. I suspect that most US-based antivirus firms have not considered being sued in England in their legal threat model. As far as I’m aware, Sophos is the only British antivirus company.
If Quad9 and friends redirected users to a page with a “malicious website blocked” message, users could understand what was going on and try to contact the owners, but those services choose to keep them in the dark by returning a SERVFAIL.
How would this be accomplished? A DNS provider can’t hijack the A record and point to their own error page. At least not without horrible consequences.
Citation needed. Unlike with CAs, there’s no official requirement list for serving DNS. It all boils down to end user trust, and that follows from the intent inferred by users from your actions, and not technicalities. If you advertise malware blocking as a feature, users will likely see the serving of such an A record as expected behavior.
My point is, when they receive responses with A/AAAA records but send NXDOMAIN or SERVFAIL to the requester, that’s already tampering with responses. I can’t see how returning a different address is any more of a hijacking. If they think hijacking for the sake of security is acceptable, at least they should make it clear to the users what’s going on and not pretend that the upstream DNS server has a problem.
It’s so much more of a hijacking to return a different address. If a user is in a browser they might turn around and send a GET request with sensitive information in the request headers. If they’re not in a browser who knows what they are doing with that bad ip.
There’s a world of difference between “blocks bad dns entries completely” and “hijacks dns entries and masquerades as various parts of the internet”
This is making me think that the real problem is that the DNS protocol doesn’t have a way to cleanly indicate something is blocked. All you can do is send SERVFAIL/NXDOMAIN, or forge an A response.
I’m imagining some DNS responses similar to HTTP 451 except not specific to legal reasons.
DNSSEC is the biggest technical hurdle here, but I think we can all agree that tampering with authoritative DNS responses is pretty gross even if it maybe sort of does work most of the time.
I recall that one reason that — formerly? — often was cited for switching away from one’s ISP’s DNS service to Google Public DNS and OpenDNS (I’m not sure Quad9 existed) was to get actual “domain not found” DNS responses rather than the pages of Web search results and ads that ISPs would serve, so I would think it is possible for a DNS provider to give an informative error page if it blocks a domain.
I’m not saying it isn’t technically possible†, I’m saying it is extremely undesirable from a security perspective. And just technically gross.
† In those days we didn’t have ubiquitous HTTPS, and DNSSEC (as /u/strugee pointed out) wasn’t a thing, so it is now actually technically much harder / sometimes impossible to takeover a domain like this. But I’m not objecting to the feasibility so I didn’t originally bring it up.
Dealt with something similar at $WORK, but from the threat hunting side. Some vendors had flagged a CDN’s IP as malicious. After seeing it pop up a few times we just blocked it (admittedly without looking into it properly).
We had a crapload of users start complaining about “threat blocked” notifications from the AV. They were understandably concerned.
I looked into it more, and it turns out this CDN is one that Microsoft uses to distribute Windows+Edge updates. The domain it reverse resolves to can be seen in an MSDN page as well.
What I assume happened is the CDN was used to host something malicious, and this IP got dragged into a bunch of blocklists. The CDN then removed the malicious stuff, and reusing the IP and it happens to host windows updates now.
I had a bug recently where I couldn’t deploy to Netlify apparently because they mistook Go Lambda functions for malware at the WAF level and rejected the upload.
Very well written! Sorry you had to go through this. Also, I’m glad you could get this issue resolved.
Here is my own DNS horror story where the domain name I was using was taken away from me without any warning or due process: https://susam.net/blog/sinkholed.html.
The underlying cause of the issue in my story was a false-positive during anti-malware operation too. What really surprised me was that an anti-malware foundation had the authority to remove me as the registrant of the domain name and assign someone else without any form of notification to me or the domain name reseller.
Here is yet another story where someone’s domain name was lost due to a registrar bug: https://medium.com/thisiscala/the-duct-tape-holding-the-internet-together-12118be60ff1.
Wow… I got lucky by comparison — at least I didn’t lose control of anything, merely got inaccessible for a part of the user base. Extra-judicial domain seizure is really extreme.
The unfortunate thing with these sort of malware detectors is that they operate with an expected false-positive rate. The industry generally thinks that’s fine, because its better to hit more malware than to miss any.
That only works if the malware decector authors are receptive to feedback, though.
Basically every new release of Rust on Windows is detected as malware by one vendor or another.
I guess it probably doesn’t help that the binaries aren’t yet signed.
CrowdStrike Falcon on macOS and SentinelOne on Windows have cost me so much wasted time as an employee of companies that uses them. Falcon routinely kills make, autoconf, etc. SentinelOne does the same thing when using msys2 or cygwin on Windows.
At least SentinelOne tells me, Falcon tries its best to leave zero useful information on the the host. When processes start randomly being terminated it takes a bit of effort to find out what the hell is actually happening. Often I realized it was Falcon only after some poor desktop security tech gets assigned a ticket and they reach out to me with a lot of confusion around some crazy complex command lines being sent through the various exec calls.
Because of the high frequency in which I encounter the issue, if something randomly fails in a way I don’t expect I immediately suspect Falcon.
My understanding is that signed binaries don’t help - if a binary is rarely run (because it’s just been released, or it’s just not a mainstream tool) there’s a good chance it’ll be detected as malicious no matter what.
I believe it depends on the kind of signature. Last time I checked, companies can buy more expensive certificates which are privileged insofar that the binaries don’t need to be run on a lot of machines to be considered safe.
And still miss a lot of malware. ;)
I wonder what would be the best way to find out what the real rate of false positives is.
I’ve worked at a company where the IT was so terrible that their actions bordered on sabotage. They caused more damage and outages than actual hackers would. The anti-virus would delete our toolchains and fill the disk with ominous files until there was no space left. Luckily, they left our Linux machines alone, so we put everything we cared about on Linux (without GUI) and hoped that their lack of know-how would prevent them from messing with those machines. It worked.
When I was in the business of releasing commercial Windows software many moons ago, I also had the same problems with AV Vendors flagging my executable as false positive. The main problem was that I used an exe-protector (Themida) for anti-cracking purposes. It was a lot better with a custom Themida stub and a code signing certificate - the latter being ridiculously complicated to obtain 15 years ago.
Once flagged as malware by one vendor, your programs signature was shared around to other vendors over time (I am not sure about the politics behind this), and you could expect to be identified as malware by more and more vendors over the next days/weeks. Getting whitelisted was basically impossible - many vendors offered some sort of whitelisting process, but it always only concerned a single executable hash. So your next update started the process again.
It’s sad that today we still face the same general problem as back then, but on a totally different level.
I wonder if you could make a case for libel. If I write an article calling someone’s product dangerous, with no evidence, then they have a very strong case if they take me to court and I am likely to be liable for any loss of business and reputation that I’ve caused. I would expect an antivirus company to have similar liability: they are telling a load of your customers that your product is malware.
I am no lawyer, but I cannot imagine that AV companies don’t have extensive legal moat against an allegation like that. But it is an interesting thought.
It’s worth noting that libel laws are quite different in different countries. In England and Wales, libel is treated as an accusation and so the principle of ‘innocent until proven guilty’ is applied to the target of the libel and not the perpetrator. This means that, if they have made written messages appear on users’ computers saying (or even strongly implying) that your program is malware then they would have to demonstrate that this accusation is true. In the US, there is no special case for libel and the person bringing the suit has to prove that the allegation is false. I suspect that most US-based antivirus firms have not considered being sued in England in their legal threat model. As far as I’m aware, Sophos is the only British antivirus company.
Great write-up.
How would this be accomplished? A DNS provider can’t hijack the A record and point to their own error page. At least not without horrible consequences.
Citation needed. Unlike with CAs, there’s no official requirement list for serving DNS. It all boils down to end user trust, and that follows from the intent inferred by users from your actions, and not technicalities. If you advertise malware blocking as a feature, users will likely see the serving of such an A record as expected behavior.
My point is, when they receive responses with A/AAAA records but send NXDOMAIN or SERVFAIL to the requester, that’s already tampering with responses. I can’t see how returning a different address is any more of a hijacking. If they think hijacking for the sake of security is acceptable, at least they should make it clear to the users what’s going on and not pretend that the upstream DNS server has a problem.
It’s so much more of a hijacking to return a different address. If a user is in a browser they might turn around and send a GET request with sensitive information in the request headers. If they’re not in a browser who knows what they are doing with that bad ip.
There’s a world of difference between “blocks bad dns entries completely” and “hijacks dns entries and masquerades as various parts of the internet”
This is making me think that the real problem is that the DNS protocol doesn’t have a way to cleanly indicate something is blocked. All you can do is send SERVFAIL/NXDOMAIN, or forge an A response.
I’m imagining some DNS responses similar to HTTP 451 except not specific to legal reasons.
DNSSEC is the biggest technical hurdle here, but I think we can all agree that tampering with authoritative DNS responses is pretty gross even if it maybe sort of does work most of the time.
I recall that one reason that — formerly? — often was cited for switching away from one’s ISP’s DNS service to Google Public DNS and OpenDNS (I’m not sure Quad9 existed) was to get actual “domain not found” DNS responses rather than the pages of Web search results and ads that ISPs would serve, so I would think it is possible for a DNS provider to give an informative error page if it blocks a domain.
2009: https://superuser.com/questions/50922/isps-hijacking-dns-errors-on-the-web
2020: https://superuser.com/questions/1521827/how-can-i-block-my-isps-search-engine
I’m not saying it isn’t technically possible†, I’m saying it is extremely undesirable from a security perspective. And just technically gross.
† In those days we didn’t have ubiquitous HTTPS, and DNSSEC (as /u/strugee pointed out) wasn’t a thing, so it is now actually technically much harder / sometimes impossible to takeover a domain like this. But I’m not objecting to the feasibility so I didn’t originally bring it up.
Unrelated: Lobsters supports @ mention syntax, though there’s no notification mechanism. @altano
Neat, thanks for the tip
Dealt with something similar at $WORK, but from the threat hunting side. Some vendors had flagged a CDN’s IP as malicious. After seeing it pop up a few times we just blocked it (admittedly without looking into it properly).
We had a crapload of users start complaining about “threat blocked” notifications from the AV. They were understandably concerned.
I looked into it more, and it turns out this CDN is one that Microsoft uses to distribute Windows+Edge updates. The domain it reverse resolves to can be seen in an MSDN page as well.
What I assume happened is the CDN was used to host something malicious, and this IP got dragged into a bunch of blocklists. The CDN then removed the malicious stuff, and reusing the IP and it happens to host windows updates now.
I had a bug recently where I couldn’t deploy to Netlify apparently because they mistook Go Lambda functions for malware at the WAF level and rejected the upload.