Threads for rbranson

    1. 1

      I read through the DynamoDb problem and was surprised that they had to add special logging to figure out and disable hot keys like ‘user_id’. If they had setup structured logging, a quick python script over the log would have shown this issue.

      1. 4

        (I debugged this issue) This is roughly what we did. I guess it didn’t come across in the blog that way?

        1. 1

          From the blog

          So we dreamt up a simple hack to give us the data we needed: anytime we were throttled by DynamoDB, we logged the key

          Is there a reason you don’t want to log every single key?

          1. 1

            a) it’s sensitive data belonging to third parties, b) at the scale they are dealing with the logs would impose significant size and time penalties

          2. 1

            Isn’t that unnecessary? You really only care about when there’s a problem, or else your logs will be noisy. If you log every time, it’ll be kind of hard to differentiate which keys are okay and which keys are a problem.