1. 34
  1. 7

    This is a very well written post! I used to work on large scale backup systems – we had customers with thousands or tens of thousands of backup clients, all using the same huge deduplicated storage pools. Deduplication leaking the presence/absence of data was a well known security problem for us and our customers, but it might not be as well known in the security community.

    1. 6

      Deduplication (heh my phone wants to correct to “reduplication”??) in ZFS is kind of a mis-feature that makes it easy to destroy the performance. (I’ve had some painful experiences with it on a small mail server…) Pretty much everyone recommends not enabling it ever. So indeed it’s not a realistic concern, but it is fun to think about.

      It shouldn’t be that hard to add a setting to ZFS that would only show logicalused to untrusted users, not used.

      1. 9

        For folks not familiar with ZFS, just want to expand on what @myfreeweb said: “pretty much everyone” even includes the ZFS folks themselves. The feature has a bunch of warnings all over it about how you really need to be sure you need deduplication, and really you probably don’t need it, and by the way you can’t disable it later so you better well be damn sure. btrfs’ implementation, though, does not AFAIK suffer from the performance problems ZFS’ does because btrfs is willing to rewrite existing data pretty extensively, whereas ZFS is not because this operation (“Block Pointer Rewrite”) would among other problems break a bunch of the really smart algorithms they can use to make stuff like snapshot deletion fast. A btrfs filesystem after offline deduplication is not fundamentally different from the same filesystem before. ZFS deduplication fundamentally changes the filesystem because it adds a layer of indirection.

        logicalused seems like a good idea. It doesn’t fix the timing side channel, though. I think you’d want to keep a rolling average of how long recent I/O requests took to service, plus the standard deviation. Then pick a value from that range somehow (someone better at statistics than me could tell you exactly how) and don’t return from the syscall for that amount of time. Losing the performance gain from a userspace perspective is unavoidable since that’s the whole point, but you can use that time (and more importantly, I/O bus bandwidth) to service other requests to the disk.

        (Side note: my phone also wanted to correct to “reduplication”. Hilarious. Someone should stick that “feature” in a filesystem based on bogosort or something.)

        1. 2

          It shouldn’t be that hard to add a setting to ZFS that would only show logicalused to untrusted users, not used.

          I think that’s harder than you think. The df(1) command will show you free space, I’m not sure you can set a quota that hides whether a file was deduplicated. Also a user can use zpool(8) to see how much space is used in total.

          However, I hardly think this is going to be a problem with ZFS, because, as you say, “Pretty much everyone recommends not enabling it ever”. I have never experienced a use case where deduplication in ZFS would be advantageous for me, on the contrary; ZFS gets slower because it has to look up every write in a deduplication table, and it uses more space because it has to keep a deduplication table. If you enable deduplication on ZFS without thorough research, you will be punished for it with poor performance long before security becomes an issue.

          1. 2

            I mean report logicalused to everywhere like df, hide the zpools..

            The pools would already be hidden if it’s e.g. a FreeBSD jail with a dataset assigned to it.

        2. 3

          I’m not sure if it’s just me but to suggest that disk utilisation statistics can be realistically used as a law enforcement weapon in this way seems farfetched. It’s also worth noting that there are all sorts of reasons why apparent disk utilisation might not be what it appears, including but not limited to overlay mounts, in-memory caching, compression.

          I’m also not sure that inferring that data exists across virtual machine boundaries is practical either. With lightweight containers maybe, but in a “real” virtual machine with emulated or paravirtual disk controllers, on-disk deduplication is invisible to the virtual machine. The virtual machine will have a filesystem of its own and will report the disk space as used regardless, even if it was deduplicated in the real world, because the filesystem descriptors on the virtual hard disk will say that it is used. How is the virtual machine supposed to know otherwise?

          1. 4

            to suggest that disk utilisation statistics can be realistically used as a law enforcement weapon in this way seems farfetched

            Very much so. I worked as a digital forensics analyst for several years, and never once did this kind of technique even remotely appear. I may not go so far as to say that it would never be used, maybe in some crazy high stakes case something like this could be used in a very targeted last ditch effort but that be a major exception. In most forensic cases you’re looking at either raw data blocks where permissions aren’t an issue anyways, or encrypted blobs where a technique like this wouldn’t make sense either.

            So perhaps the author is focusing more on typical malicious software.

            1. 3

              VMs were mentioned in the context of timing, not space utilization. i.e. you could detect (I guess) a suspiciously fast sync write and infer that it didn’t have to write the data to the disk because it’s been detected as being already there.

              1. 4

                Perhaps this can be reproduced in controlled conditions with very specific hardware but it also seems like a stretch otherwise. The moment you move away from a single 5400rpm disk up to a hardware RAID controller or a full SAN appliance, you’re suddenly subject to fabric congestion, flash write caches and possibly even multi-tiered storage. That’s still ignoring the fact that the emulated/paravirtual disk controller still has to handle the writes as if they are really happening, as deduplication won’t be happening until the hypervisor writes virtual blocks out to real disk. I suppose the notable exception here is with raw device mappings or cases where something like iSCSI is used to present a LUN to the virtual machine itself, but you’d still be subject to a whole range of other conditions first.