Was it evil of me that I checked to see if their hex dump had also been sanitized to redact the paths redacted in the text part of “TCP Payload Corruption”?
The thing that jumped out at me was, “How is their environment so heterogeneous?” Multiple kernel versions with multiple versions of Xen with Xen in different modes, I’m surprised they found it.
This might seem odd, however its not unusual. A couple of the reasons that come to mind
Not necessarily valid reasons for the case at hand, but just to give some ideas.
I noticed that too, often the version of Xen is out of your control if you’re running on top of IaaS, but the Linux version? I am very curious as to why you would run that many different kernels.
This is speculation, but I wonder if that helps in some ways. Once you’ve found an environment where it happens, and one where it doesn’t, you can narrow your search to the differences between the two.
This is a seriously deep rabbit hole. Kudos to not only finding all these bugs but having the patience to write them up.