1. 3
  1.  

  2. 5

    The conclusion was neat. The improved caching reduced the load on their Cassandra cluster. The reduced load allowed them to increase the consistency level of their reads from Cassandra. So in the end they get both better performance and higher accuracy results.

    I’d be terrified the whole time while writing that caching and invalidation layer, though. I’d be surprised if there wasn’t at least one bug in which it can return stale data. Still, as long as any condition in which it returns stale data occurs less often than the previous situation in which the previous low consistency Cassandra reads were also returning stable data, then it’s a net win for data accuracy.

    1. 3

      Thank you for the thoughtful response! Once live, the impacts were pretty staggering for our customer experience, and thankfully without any major incident. I attribute that to very thoughtful test design and staging on Manu’s part; he is an incredible engineer to work with.