Unlike metric values, prometheus stores timestamps in a “delta of deltas” format. That is, the first timestamp in a series is encoded directly, the second is encoded by subtracting the first, and the third and later ones are encoded by subtracting the previous timestamp and then encoding the difference in the difference. All of this is done as millisecond resolution.
When you have metrics on a nice even cadence, this works out really nicely: say you scrape every 30 seconds and the first timestamp in a block is 1234567890123. The first value will be encoded directly, taking 68 bits. The second value will be 30000, which takes 20 bits. And the subsequent values will all be 0, which is encoded as a single bit. That’s 15 bytes per hour in the long run, which is pretty excellent!
But… millisecond resolution. Of the actual time when prometheus executed the scrape. Which is on a timer that doesn’t always hit its deadline within a millisecond, especially on a big system with lots of targets and lots of GC pressure. So your timestamps won’t be uniformly incrementing by 30,000 each time, and you won’t be encoding a stream of all zeroes, there will be some jitter, and the encoded values in the column will be, let’s say, between -100 and 100, mostly between -10 and 10, but only rarely zero. And while zeroes encode down to one bit, as soon as you go nonzero that jumps up to 16 bits. 16x is a pretty big expansion, and it can end up increasing your total memory and disk usage by more than 50%.
Prometheus has an option, --scrape.timestamp-tolerance to fudge the timestamps to make them compress better, but it defaults to 2ms. Bumping that up to 25 or 50 milliseconds makes a really noticeable difference in resource consumption, for what most people would consider basically zero loss in accuracy.
Wow I knew about delta of delta compression but it never really clicked why it’s useful. I also didn’t know about that Prometheus flag, thanks for the info!
Something related I discovered a week or two ago:
Unlike metric values, prometheus stores timestamps in a “delta of deltas” format. That is, the first timestamp in a series is encoded directly, the second is encoded by subtracting the first, and the third and later ones are encoded by subtracting the previous timestamp and then encoding the difference in the difference. All of this is done as millisecond resolution.
When you have metrics on a nice even cadence, this works out really nicely: say you scrape every 30 seconds and the first timestamp in a block is 1234567890123. The first value will be encoded directly, taking 68 bits. The second value will be 30000, which takes 20 bits. And the subsequent values will all be 0, which is encoded as a single bit. That’s 15 bytes per hour in the long run, which is pretty excellent!
But… millisecond resolution. Of the actual time when prometheus executed the scrape. Which is on a timer that doesn’t always hit its deadline within a millisecond, especially on a big system with lots of targets and lots of GC pressure. So your timestamps won’t be uniformly incrementing by 30,000 each time, and you won’t be encoding a stream of all zeroes, there will be some jitter, and the encoded values in the column will be, let’s say, between -100 and 100, mostly between -10 and 10, but only rarely zero. And while zeroes encode down to one bit, as soon as you go nonzero that jumps up to 16 bits. 16x is a pretty big expansion, and it can end up increasing your total memory and disk usage by more than 50%.
Prometheus has an option,
--scrape.timestamp-toleranceto fudge the timestamps to make them compress better, but it defaults to 2ms. Bumping that up to 25 or 50 milliseconds makes a really noticeable difference in resource consumption, for what most people would consider basically zero loss in accuracy.Wow I knew about delta of delta compression but it never really clicked why it’s useful. I also didn’t know about that Prometheus flag, thanks for the info!