1. 2

    That’s interesting. The main problem we’ve identified with sanitizer builds though is that they are hard to run in any kind of real world setting for most users. They are hungry for memory and slower. We’ve been asking users to do it anyway and promise a bug bounty if it helps finding security problems, but I’m curious how you do it.

    Do you run asan builds in anything other than CI?

    1. 0

      Yeah, not suitable for our larger real-world workloads. It works for our test nodes which have a smaller foot print. The test nodes are used by all other developers on the team for pre-release features and we have a test corpus that we run through them as well as through our CI suite. ASAN performs sufficiently well in that circumstance. Just the simple act of cross-report aggregation and alerting has already caught several real world issues for us (in particular, LSAN output is humongous due to various pointer encoding tricks that we play).

      Q3, we will be doing some production mirroring (highly sampled) to provide more ASAN coverage in the test nodes with the idea that the nodes will be restarted quite often due to memory utilization.