Reliability targets Reviewed 29 Oct 2026

Latency percentiles

A percentile says how slow a given share of requests can be: the p99 is the latency that 99% of requests meet. Use percentiles to look at latency, but write latency SLOs as the share of requests at or below a threshold: good and total request counts can be summed across servers and time windows, then divided, and percentiles cannot be combined that way.

Percentiles in one paragraph

Sort the requests in a window by latency. Under the nearest-rank definition used on this page, the p50 (median) is the smallest observed latency with at least half of the requests at or below it, and the p99 the smallest with at least 99% at or below it, so at most 1% are slower. Each extra nine looks at a tail ten times smaller: the slowest 1%, 0.1% and 0.01% of requests. A percentile says nothing about how slow the requests beyond it are: a p99 of 200 ms is the same whether the slowest requests take 300 ms or time out after 30 seconds.

Percentiles need enough requests to mean anything. Under the nearest-rank definition, the p99 of 50 requests is the 50th, which is simply the slowest request. Tools also differ in how they compute percentiles, interpolating between values or estimating from buckets, so the same data can give slightly different numbers in two systems.

Why not the average

Latency is skewed: most requests are fast and a few are much slower. The SRE book shows a service where a typical request takes about 50 ms while 5% of requests are 20 times slower, and notes that alerting on the average would have shown no change over the day while the tail moved a lot. In the example below, the mean latency is 51.6 ms, yet 0.95% of requests take 900 ms or more.

Watch at least the p50 for the typical request and a high percentile such as the p99 for the tail. The SRE book notes that users typically prefer a slightly slower system to one with high variance, which is why some teams track only high percentiles.

You can’t average percentiles

Two instances serve the same endpoint. A is busy and healthy; B is quiet and slow for 5% of its requests. These illustrative counts show what happens when per-instance p99s are combined:

The true p99 across both instances is 60 ms. The plain average says 620 ms and the weighted average 156 ms; neither value comes from the data. For exact nearest-rank percentiles over the same window, the worst instance’s p99 (1,200 ms) is a safe upper bound, because at most 1% of either instance’s requests are slower. It can be far off: here it equals the p99.9 of all requests, 1,200 ms. The bound need not hold for interpolated or estimated percentiles. The same goes for time: the p99 for a day is not the average of each minute’s p99.

The last column does combine. The share of requests at or below 300 ms across both instances is the total of fast requests divided by the total of requests, 99.05%, and the same sum works across minutes, days or regions. To get a percentile across sources, merge the underlying distributions and compute it once. With raw latencies that gives the exact value; with merged histogram buckets it gives an estimate whose accuracy depends on the buckets.

Writing latency SLOs

State the SLI as a ratio of good requests, with the threshold inside the definition:

latency SLI = valid requests at or below threshold ÷ valid requests

• Pick thresholds from user needs, then a target for each. Several thresholds can cover the typical case and the tail, as in the SRE book’s example: 90% of Get calls in under 1 ms, 99% in under 10 ms and 99.9% in under 100 ms.
• Decide what happens to failed requests. A request that fails fast can look like excellent latency. Count latency over successful requests and cover errors with an availability SLI, or count failures as not good.
• Say where latency is measured: at the server, at the load balancer or in the client, and to first or last byte. The client view is closest to the user but hardest to collect.
• Error budgets and burn rates work unchanged. A 99% latency SLO allows 1% of requests to be slower than the threshold, and burn-rate alerts use the share of slow requests in place of the error ratio.

For exact nearest-rank percentiles, a statement such as “p99 at or below 300 ms” means the same as “at least 99% of requests at or below 300 ms”. Measure it as that share rather than by computing p99s and checking them: the share aggregates correctly, and it does not depend on how a tool estimates percentiles.

Count requests, not minutes. A time-based SLI that marks each minute good or bad by checking that minute’s p95 throws away most of the information. Take a 99.99% SLO over 30 days, a budget of 4m 19s, and one minute in which 94% of requests finish within 300 ms, so it fails a “p95 ≤ 300 ms” check. The whole minute is bad, which spends 23.15% of the month’s budget; counted per request, the 6% that were slow spend 1.389%, at steady traffic. At high targets one bad evaluation interval can be a large part of the budget.

Measuring with histograms

Most monitoring systems store latency in histogram buckets and estimate percentiles from them. Prometheus offers three choices:

• Summaries estimate chosen quantiles in each process, with a configurable rank error: configured as 0.95 ± 0.01, the documentation’s example could report anywhere from 270 ms to 330 ms around a true 300 ms. As the Prometheus documentation puts it, averaging their quantiles across instances “yields statistically nonsensical values”.
• Classic histograms count requests in fixed buckets. They aggregate correctly if every instance uses the same bucket boundaries, but a percentile estimate can be anywhere within its bucket. In the documentation’s example, requests that all take about 220 ms, with buckets at 200 and 300 ms, give an estimated 95th percentile of 295 ms.
• Native histograms use many exponential buckets and aggregate correctly. In the same example they estimate 228 ms. The Prometheus documentation recommends them where available.

With classic histograms, use the same bucket boundaries on every instance, including one exactly at each SLO threshold. If an instance lacks the threshold bucket, the ratio below silently leaves its fast requests out of the numerator while counting all its requests in the denominator. With the boundary in place the share needs no interpolation within a bucket, although rate() still extrapolates at the edges of its range. These examples treat 2xx responses as successful; adjust the code matcher to your own definition, keeping it identical in the numerator and denominator.

# Share of successful (2xx) checkout requests served in 300 ms or less over 5 minutes
# (classic histogram with a bucket boundary at 0.3 seconds on every instance).
  sum(rate(http_request_duration_seconds_bucket{job="checkout", code=~"2..", le="0.3"}[5m]))
/
  sum(rate(http_request_duration_seconds_count{job="checkout", code=~"2.."}[5m]))

# Estimated p99 of successful requests across all instances:
# sum the buckets first, then take the quantile.
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout", code=~"2.."}[5m]))
)

Percentiles from sampled data

Tracing and event tools often keep only a sample of requests and record the rate each was kept at: an event kept at a rate of 1 in 9 stands for 9 requests. If every request is kept at the same rate, percentiles of the kept events estimate the real ones without adjustment. If rates vary, for example keeping every error and slow request but 1 in 100 fast successes, each event must be counted as many times as it represents. In the example from Observability Engineering, three events kept at different rates (1 ms at 1 in 5, 3 ms at 1 in 2, 7 ms at 1 in 9) stand for 16 requests. Their plain median is 3 ms; weighted by rate it is 7 ms. Check that your tool weights by sample rate, and keep enough slow requests to estimate the tail you care about.

Tail latency at scale

When one user request fans out to many backend calls in parallel and waits for all of them, the slowest call sets its latency. Dean and Barroso’s example: if each server is slow (one second) for 1% of requests and the servers are slow independently, a request that needs 100 such servers is slow 63.4% of the time. Even at one slow response in 10,000, a request touching 2,000 servers is slow 18.13% of the time.

In a real Google service they measured, the 99th percentile for one leaf request to finish was 10 ms, but for all leaf requests to finish it was 140 ms. A backend’s p99 on its own says little about what users wait for. Remedies work on the tail rather than the median: hedged requests, which send a second copy to another replica after a short delay and use whichever answer arrives first, returning good-enough partial results, and reducing interference from background work. In their benchmark, hedging after 10 ms cut the 99.9th percentile from 1,800 ms to 74 ms while sending 2% more requests. Hedge only operations that are safe to repeat, cancel the redundant copy, and cap the extra load; if the replicas share a bottleneck, a second copy is slow too.

Load tests that hide the tail

Many load generators send a request on each connection only after the previous response arrives. When the server stalls, they stop sending. If real arrivals would have continued during the stall, the test records a handful of slow requests and omits the queued demand behind them. Gil Tene calls this coordinated omission. In the wrk2 README’s example with a 1.4-second pause, the corrected 99th percentile was about 200 times higher than the uncorrected one. Model the arrival process you expect in production and measure latency from when each request should have started, not when it was sent; wrk2 does this for a constant request rate.

Sources: Google, Site Reliability Engineering, 1st ed., ch. 4 “Service Level Objectives”; Site Reliability Engineering, 2nd ed. (2026), ch. 8 “Service Level Objectives” (“Recommendations for Beginners”: latency SLOs as the share of requests under a threshold); Jeffrey Dean and Luiz André Barroso, The Tail at Scale (Communications of the ACM, 2013); Prometheus, Histograms and summaries; Gil Tene, wrk2 README (coordinated omission); Charity Majors, Liz Fong-Jones and George Miranda, Observability Engineering, 2nd ed. (2026), ch. 12 “Acting On and Debugging SLO-Based Alerts” (good minutes versus good requests) and ch. 15 “Cheap and Accurate Enough Sampling” (weighting by sample rate).