There are a whole bunch of malware families that communicate partially and/or completely over DNS.
Some, like SUNBURST, exfiltrate a small amount of encoded data by resolving it as a subdomain to the primary C2, similar as in the linked article. In the case of SUNBURST, the data was info like victim’s domain, presence of security software, and other profiling data. Bits within the corresponding DNS A record indicated what mode the malware should enter.
Other families implement bidirectional data flow over e.g., DNS TXT queries. I’ve even seen threat actors tunnel RDP over DNS to evade network restrictions.
On the brighter side (?), it’s not uncommon for organizations to capture DNS logs, which makes it feasible to scope these compromises and occasionally reconstruct the tunneled traffic. Also, as you can imagine, it’s a noisy technique with many, high-entropy domains being resolved.
Yes, I saw quite a bit of work on sending data “upstream” using DNS, but surprisingly little on pulling data “downstream”. Surely something like this chunking scheme has been done before!
There was a lot of research done at Georgia Tech around detecting C2 channels, particularly through DNS. Some of the researchers spun out and started Damballa Security to commercialize it. They weren’t terribly successful, tho not because it wasn’t a good idea. I believe some of the Damballa folks have evolved the idea and have another startup in the works.
I wonder how many recursive resolvers have assumptions that don’t match this use of DNS, either in the form of the contents of the records or the size of them.
Me too! But so far, so good? (That said, I’ve been having trouble on one of my machines when it’s connected to Tailscale with MagicDNS. It could be cache poisoning from when I was doing testing during development; with DNS it’s always hard to tell. From a quick google, it looks like Tailscale does have TCP-fallback implemented for DNS, but the symptoms I see are consistent both with cache problems and with lack of correct fallback, so I’ll just wait and see if the problem goes away with time!)
Could you make a debug hostname where you can send different responses depending on if the response is sent over UDP or TCP, or does the DNS server not expose that difference?
I just re-ran it now, with tailscale and magicdns on, and
the first run succeeded, with full records returned. Yay!
the second and subsequent runs were given a truncated record, and so did not work. However, logging on the server side shows that both UDP (truncated) and TCP (nontruncated) requests were made.
So some intermediate caching resolver is making a mistake somewhere. Perhaps tailscale’s local resolver? Perhaps 9.9.9.9? Digging @100.100.100.100 yields the broken behavior, so perhaps it’s a bug there. Speculating: when it cached the response from the server on the first run, perhaps it cached the truncated UDP response instead of the nontruncated TCP response, even though it sent the TCP response back on that first run, and perhaps for the second and subsequent runs it is using its empty cached UDP record.
There’s also a chance that some caching resolver isn’t prepared to store 48 KB data per record and instead only are prepared to store e.g. 2-4 KB per record. When you do the first attempt with a cold cache you might be served the response directly, but the second time with a hot cache you instead get it served from the caching layer that may have silently truncated the response to fit with their (broken) expectation of how much data a DNS response can hold.
However, most “modern” configured domains (DNSSec, various email validation systems, etc) have a variety of decent sized TXT records in their configurations - so it’s possible that there’s already a lot of data being stored about various domains, and this wouldn’t inflate the size too much. Hard to say.
This is the kind of thing that always brings me back to lobsters, tonyg! Thanks a lot for this fun write up and experiment. Very nice work. It already has me thinking of other DNS (ab)uses that would be fun to experiment with.
The distribution of chunk sizes is pretty strange though. I don’t know in what sense the “average” actually is an average; the minimum and maximum cut-offs are enforced by the algorithm though.
It’s an expectation. Content-defined chunking is basically about computing a running hash on the data and declaring a chunk boundary whenever the hash meets some condition (like “has 10 trailing zeroes”). If you treat the data as random and the hash as ideal, then the a-priori probability of chunking at any given byte is constant and independent of history.
Supposing that condition is “has 10 trailing zeroes” then the probability of chunking at a given byte is 1/1024, and the distribution of chunk sizes is geometric, with semi-infinite support (meaning every chunk size from 1 to infinity has a nonzero probability), but the expected (average) number of bytes per chunk is simply 1024.
Adding minimum and maximum chunk sizes means ignoring the hash (never chunking) if the chunk size is less than the minimum, and ignoring the hash (always chunking) if the chunk size reaches the maximum. That basically amounts to truncating the distribution. Which changes the expectation, meaning that the average chunk size after applying the min and max is probably not the “average chunk size” you configured, unless they did some fancy math to correct it.
What is the point of adding such a condition though ?
In the case of data deduplication, your only “condition” is wether or not the hash was already found, so you can skip the chunk and move on to the next one. You definitely need a min/max so you can still chunk data when there is no match.
There are two reasons why that’s not how deduplication works, but the simpler one is: a lookup table of every value the hash took on at every location ever would be bigger than the data itself. So instead
We break the data into chunks using the algorithm I described.
We hash each chunk using a different, stronger hash like SHA-256 (the rolling hash is optimized for speed rather than strength to begin with, and we’ve further “compromised” it by restricting it to a limited subset at chunk points).
The SHA is what’s used for deduplication — chunks with the same SHA are only stored once.
So the content-defined chunking the deduplication do its job without having crazy overhead, in a way that fixed chunking wouldn’t. Any hunk of duplicate data large enough to contain at least n chunk points will produce at least n-1 duplicate chunks, regardless of “alignment” (where the duplicate data starts in relationship to the previous chunk point). That makes it much more likely that we will actually find duplicate blocks.
And it means that — suppose you’re taking periodic backups of a directory, and deduplicating them so that you can restore back to any point but you’re only paying for storage for what changed. Suppose that directory contains a ten-million-row CSV file, and in between backup runs someone adds a new row right in the middle of the file, leaving everything else unchanged. What you don’t want is to store the first half of the file as duplicates and the second half as completely new data because the chunk boundaries moved and none of the later chunks match chunks from the previous version of the file. What you do want is to store one new chunk containing the edited portion in the middle of the file, and have dupes for the beginning and the end. Content-defined chunking makes that work.
(Tech note: because of the enforced min and max chunk sizes, it’s not guaranteed that only the minimum number of chunks change. A change at one point can have a cascade effect on later chunk boundaries. But as long as the parameters are chosen sensibly it will be close. It’s another expectation thing. You have, say, a 90% chance of the edit affecting one chunk, a 9% chance of it affecting two, a .9% chance of it affecting three, etc. which means that on average an edit affects 1.111 chunks. Those are numbers I just made up but they’re probably worse than reality.)
There are a whole bunch of malware families that communicate partially and/or completely over DNS.
Some, like SUNBURST, exfiltrate a small amount of encoded data by resolving it as a subdomain to the primary C2, similar as in the linked article. In the case of SUNBURST, the data was info like victim’s domain, presence of security software, and other profiling data. Bits within the corresponding DNS A record indicated what mode the malware should enter.
Other families implement bidirectional data flow over e.g., DNS TXT queries. I’ve even seen threat actors tunnel RDP over DNS to evade network restrictions.
On the brighter side (?), it’s not uncommon for organizations to capture DNS logs, which makes it feasible to scope these compromises and occasionally reconstruct the tunneled traffic. Also, as you can imagine, it’s a noisy technique with many, high-entropy domains being resolved.
Yes, I saw quite a bit of work on sending data “upstream” using DNS, but surprisingly little on pulling data “downstream”. Surely something like this chunking scheme has been done before!
There’s probably novel research to be done in identifying and fingerprinting DNS servers that enable this sort of setup.
There was a lot of research done at Georgia Tech around detecting C2 channels, particularly through DNS. Some of the researchers spun out and started Damballa Security to commercialize it. They weren’t terribly successful, tho not because it wasn’t a good idea. I believe some of the Damballa folks have evolved the idea and have another startup in the works.
I wonder how many recursive resolvers have assumptions that don’t match this use of DNS, either in the form of the contents of the records or the size of them.
Me too! But so far, so good? (That said, I’ve been having trouble on one of my machines when it’s connected to Tailscale with MagicDNS. It could be cache poisoning from when I was doing testing during development; with DNS it’s always hard to tell. From a quick google, it looks like Tailscale does have TCP-fallback implemented for DNS, but the symptoms I see are consistent both with cache problems and with lack of correct fallback, so I’ll just wait and see if the problem goes away with time!)
Could you make a debug hostname where you can send different responses depending on if the response is sent over UDP or TCP, or does the DNS server not expose that difference?
I just re-ran it now, with tailscale and magicdns on, and
So some intermediate caching resolver is making a mistake somewhere. Perhaps tailscale’s local resolver? Perhaps 9.9.9.9? Digging @100.100.100.100 yields the broken behavior, so perhaps it’s a bug there. Speculating: when it cached the response from the server on the first run, perhaps it cached the truncated UDP response instead of the nontruncated TCP response, even though it sent the TCP response back on that first run, and perhaps for the second and subsequent runs it is using its empty cached UDP record.
There’s also a chance that some caching resolver isn’t prepared to store 48 KB data per record and instead only are prepared to store e.g. 2-4 KB per record. When you do the first attempt with a cold cache you might be served the response directly, but the second time with a hot cache you instead get it served from the caching layer that may have silently truncated the response to fit with their (broken) expectation of how much data a DNS response can hold.
As a first order estimate, “About all of them.”
However, most “modern” configured domains (DNSSec, various email validation systems, etc) have a variety of decent sized TXT records in their configurations - so it’s possible that there’s already a lot of data being stored about various domains, and this wouldn’t inflate the size too much. Hard to say.
This is the kind of thing that always brings me back to lobsters, tonyg! Thanks a lot for this fun write up and experiment. Very nice work. It already has me thinking of other DNS (ab)uses that would be fun to experiment with.
It’s an expectation. Content-defined chunking is basically about computing a running hash on the data and declaring a chunk boundary whenever the hash meets some condition (like “has 10 trailing zeroes”). If you treat the data as random and the hash as ideal, then the a-priori probability of chunking at any given byte is constant and independent of history.
Supposing that condition is “has 10 trailing zeroes” then the probability of chunking at a given byte is 1/1024, and the distribution of chunk sizes is geometric, with semi-infinite support (meaning every chunk size from 1 to infinity has a nonzero probability), but the expected (average) number of bytes per chunk is simply 1024.
Adding minimum and maximum chunk sizes means ignoring the hash (never chunking) if the chunk size is less than the minimum, and ignoring the hash (always chunking) if the chunk size reaches the maximum. That basically amounts to truncating the distribution. Which changes the expectation, meaning that the average chunk size after applying the min and max is probably not the “average chunk size” you configured, unless they did some fancy math to correct it.
What is the point of adding such a condition though ?
In the case of data deduplication, your only “condition” is wether or not the hash was already found, so you can skip the chunk and move on to the next one. You definitely need a min/max so you can still chunk data when there is no match.
There are two reasons why that’s not how deduplication works, but the simpler one is: a lookup table of every value the hash took on at every location ever would be bigger than the data itself. So instead
So the content-defined chunking the deduplication do its job without having crazy overhead, in a way that fixed chunking wouldn’t. Any hunk of duplicate data large enough to contain at least n chunk points will produce at least n-1 duplicate chunks, regardless of “alignment” (where the duplicate data starts in relationship to the previous chunk point). That makes it much more likely that we will actually find duplicate blocks.
And it means that — suppose you’re taking periodic backups of a directory, and deduplicating them so that you can restore back to any point but you’re only paying for storage for what changed. Suppose that directory contains a ten-million-row CSV file, and in between backup runs someone adds a new row right in the middle of the file, leaving everything else unchanged. What you don’t want is to store the first half of the file as duplicates and the second half as completely new data because the chunk boundaries moved and none of the later chunks match chunks from the previous version of the file. What you do want is to store one new chunk containing the edited portion in the middle of the file, and have dupes for the beginning and the end. Content-defined chunking makes that work.
(Tech note: because of the enforced min and max chunk sizes, it’s not guaranteed that only the minimum number of chunks change. A change at one point can have a cascade effect on later chunk boundaries. But as long as the parameters are chosen sensibly it will be close. It’s another expectation thing. You have, say, a 90% chance of the edit affecting one chunk, a 9% chance of it affecting two, a .9% chance of it affecting three, etc. which means that on average an edit affects 1.111 chunks. Those are numbers I just made up but they’re probably worse than reality.)