Why your p99 is lying to you
· 9 min
p99 is the standard go-to metric for tail latency, but it's routinely computed in ways that hide exactly the problems it's supposed to catch. A green dashboard doesn't mean the experience of the slowest 1% of users is fine, it means the math behind that number hid what was happening to them.
A percentile of percentiles isn't a percentile
A common mistake is aggregating p99 across multiple instances, or across time windows, by averaging (or even taking a percentile of) percentiles that were already computed per instance. Mathematically this throws away information: one consistently slow instance can get diluted into the average of the others and never show up in the final number, even if one in every hundred users on that specific instance is having a consistently bad experience.
Coordinated omission: the benchmark that lies to itself
A load generator that waits for a response before sending the next request under-samples precisely the slow requests, while the server is stalled processing one of them, fewer requests get sent during that window, so the system's slowest period ends up under-represented in the final sample. The result is a measured percentile that looks much better in the benchmark than what production will actually deliver, where requests keep arriving regardless of what the server is doing.
Sampling distorts the tail too
Collecting metrics by sampling (recording only 1 in N requests) works fine for the median but is statistically unstable for p99 and p999 precisely because those are rare-event statistics, fewer samples in the tail mean much wider confidence intervals, and a smooth p99 line on a dashboard can just be an artifact of too few tail samples, not a real signal of stability.
What actually helps
Real histograms (like the HDR format) let you compute exact percentiles from the raw distribution instead of aggregating already-computed per-instance percentiles, and open-model load testing tools (ones that don't wait for a response before firing the next request) avoid the coordinated omission problem from the point of collection. Neither is more work than what's already being spent staring at a number that is, in practice, lying.