"How NOT to Measure Latency" by Gil Tene on HackerTube: recommended by 9 Hacker News commenters across 9 separate months, 2015–2026.
talk · intermediate · evergreen
Gil Tene of Azul Systems explains common pitfalls in measuring latency and response time, including why averages mislead and the problem of coordinated omission. He introduces open source tools like HdrHistogram for better measurement and reporting.
The canonical recommendation whenever percentiles, tail latency, load testing or averages come up, described by some as life changing.
“This a life changing video by Gil Tene on this topic.”
20 Hacker News comments link this video. Each one links back to its thread.
We use locust at work but I HIGHLY recommend wrk for a very robust yet simple load testing tool. [link] And of course, this talk by Gil Tene is fantastic if you're interested in load testing stats ▶ this video
in “Grasshopper – An Open Source Python Library for Load Testing” · View on HN
This a life changing video by Gil Tene on this topic. Bottom line is because of the number requests to support a single customer "request", the high percentiles are the ones that actually matter. ▶ this video
in “Ask HN: Must Read from ACM Library?” · View on HN
This topic always leads me to think about this great talk from Gil Tene on how NOT to measure latencies (basically, don't use averages!). ▶ this video I'm also a huge fan of how Dormando showed latency distributions in one of his recent Memcached Extstore posts. The default is 95th percentile but you can change the percentile to what matters to you (i.e. 99th percentile if you ask me!). Scroll down to see what he did and play with it. [link]
in “Averages Can Be Misleading: Try a Percentile (2014)” · View on HN
Doesn't this depend on how "tuned" your subcomponents are? If I had a system that's supposedly heavily optimized, I would simply start seeing the worst-case latency on a higher percentile, 99.9th and so forth. Gil Tene has an excellent talk[1] on this. [1] ▶ this video
in “Throughput vs. Latency and Lock-Free vs. Wait-Free” · View on HN
Tried to link to the specific timestamp in the video, where the p99 lie is discussed. Core takeaway for me was that most users of a page will experience the p99 latency of the system: ▶ this video
in “How Not to Measure Latency (The P99 Lie) by GIL Tene” · View on HN
who cares about lower percentage tiers in the world of big data? what does the p30 or 50 even matter at large scale? See Gil Tene's talk on how not to measure latency. ▶ this video What matters is what most of your customers will get - p99, p999, p9999, p100.
in “Kafka Removing Zookeeper Dependency” · View on HN
The "how not to measure latency" talk by Gil Tene is another really good explanation of this topic: ▶ this video
in “Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers” · View on HN
Worth watching is How NOT to Measure Latency: ▶ this video The speaker Gil Tene is also the author of the HdrHistogram which addresses this articles point: [link]
in “We can do better than percentile latencies” · View on HN
Gil Tene has an entire talk about this point called "How NOT to measure latency" ( ▶ this video ). In many circumstances, a web service (either server or client) has large numbers of dependencies, so a user only has to hit 95th-percentile latency for one dependency to have their overall latency significantly hurt. In other words, users may encounter the "rare" bad case on a majority of requests. Thus, even higher percentiles should be tightly controlled in order to truly keep a web service's overall latency under control.
in “Unlearning descriptive statistics” · View on HN
That reminds me of this talk[0] by Gil Tene called "How NOT to Measure Latency" at the Strangeloop conference in 2015 (or read this blog post[1] that contains the most important points). [0] ▶ this video [1] [link]
in “Queueing Requests Queues Your Capacity Problems, Too” · View on HN
Performance (Really) Matters by Emery Berger talks a lot about layout. [link] And the topic of Coordinated Omission, "How NOT to Measure Latency" by Gil Tene ▶ this video Injection slowness into an application to measure the relative impact of that portion of the application has really worked well for me.
in “Four Kinds of Optimisation” · View on HN
I remember seeing a talk years ago about percentiles and how they lie: ▶ this video You should be exposing the maximum metric from your app, computing a percentile from an aggregated histogram is lossy. [edit: Found the link, "How NOT to Measure Latency" by Gil Tene]
in “Your nines are not my nines (2019)” · View on HN
> I found this article hard to read; it felt like bulleted noted after watching a presentation and many of the statements lacked the context to back them up. Because it is. Here's the presentation. ▶ this video
in “Your load generator is probably lying to you” · View on HN
I don't know if it's the correct talk, but those ideas are explored in "How NOT to Measure Latency by Gil Tene: ▶ this video
in “Suspicious Discontinuities” · View on HN
Does K6 deal with the coordinated omission problem? Gil Tene (Azul Systems) has argued convincingly [1] (slides [2]) that monitoring tools get latency measurement wrong because sudden spikes aren't represented correctly in timings because of averaging and the wrong use of percentiles. He argues that percentiles simply aren't useful, because, statistically, most requests will experience >= 99.99-percentile response times. All percentiles lie; the only truly useful and realistic number is, in fact, the 100th percentile. He also argues that the most revealing way to show latency is with a complete histogram. I haven't used K6 yet, but I've been looking for a good load-testing tool, and intend to try it out. [1] ▶ this video [2] [link]
in “K6: Like unit testing, for performance” · View on HN
Some additional examples of people fixing instances of this problem Dissolving the fermi paradox [0], convolves distributions instead of multiplying point estimates of life/no life. How to NOT measure latency , looks at CDFs instead of slow/fast thresholds. [0] [link] [1] ▶ this video
in “Deterministic thinking: a problem in how we think, not just in how we act” · View on HN
You almost never want to use average as your metric when dealing with time. For time, the 95 or 99 percentile latency is going to be measure of how many requests out of a hundred a less than your metric. Esp when the thing you are measuring is a piece of larger system, which is the crux, because of how a resource in turn requests k-more resources. Each request has an equal chance of being over the latency percentile. Even at 10 requests, only 90% of sessions will be under the 99% latency. See also, Gil Tene ▶ this video A nice overview, [link] See also, [link]
in “Sanic – Python 3.5+ web server that's written to go fast” · View on HN
this is like.. average latency of 600ms.. sounds great when you tell your CEO but it still doesn't really tell the true story about how a huge portion of your customers are waiting 1.2s for their goddamn page to load. Max doesn't tell the whole story either, but it's pretty revealing. Show me that 99.9999 percentile baby! 4 point precision then we're talkin.. and none of this Coordinated Omission bullshit. ▶ this video </rant>
in “Tech-Hub Housing Costs” · View on HN
This is a summary of ▶ this video . It looks like a good summary, so I guess we won't replace the URL above, even though HN prefers original sources. We did change the title to that of the talk though.
in “How Not to Measure Latency” · View on HN
I'm sure internally folks are looking at more stuff, but blog post wise, it makes it look like you're intentionally omitting it (see ▶ this video )
in “Firefox Got Faster for Real Users in 2023” · View on HN