Reliability targets Updated 27 Oct 2026

Multiwindow, multi-burn-rate alerting

Alert on how fast the error budget is being spent, not on whether an error rate crossed a line. Page when a large share of the budget is being spent quickly, open a ticket for slow leaks, and require a short window to agree so alerts clear soon after recovery.

Burn rate in one paragraph

burn rate = error rate ÷ (1 − SLO)

With a 99.9% SLO the budget is 0.1%. An error rate of 0.1% is a burn rate of 1: the budget lasts exactly the window. An error rate of 1.44% is a burn rate of 14.4: a 30-day budget lasts 50 hours. To alert when a given share of the budget has been spent within an alert window:

burn-rate threshold = budget share × SLO window ÷ alert window

2% of a 30-day (720-hour) budget in 1 hour gives 0.02 × 720 ÷ 1 = 14.4.

Why simple threshold alerts fail

Google’s SRE Workbook judges alerting strategies on four properties:

• Precision: the share of alerts that were significant.
• Recall: the share of significant events that alerted.
• Detection time: how long a real problem takes to alert.
• Reset time: how long the alert keeps firing after the problem is fixed.

Alerting when the error rate over a short window exceeds the SLO (say 0.1% over 10 minutes) has good recall and detection time, but terrible precision: a 10-minute blip that spends 0.02% of the monthly budget pages someone. Lengthening the window improves precision but makes detection and reset slow. Adding a for: duration delays every alert equally, including the severe ones. A single burn-rate alert fixes precision but either misses slow burns or pages too late for fast ones. Combining several burn rates, each checked over a long and a short window, gets good scores on all four.

Recommended parameters

Starting values from the Workbook for a 30-day SLO window. Each long window is paired with a short window one twelfth its length; both must exceed the threshold.

Burn rates depend on the SLO window. A 28-day window gives 13.44, 5.6 and 0.93; a 7-day window gives 3.36, 1.4 and 0.23. The calculator recomputes the table for any SLO and window.

Detection time

Because each alert fires when a fixed share of the budget has been spent, worse incidents alert faster. For a 99.9% SLO, if the error rate jumps suddenly and stays there:

Detection time = threshold error rate ÷ actual error rate × long window. These are idealised: they assume no errors before the incident, a full window of history and steady traffic. Add scrape interval, rule evaluation interval and notification delay. Even so, a total outage crosses the page threshold in under a minute, while a 2% error rate takes about 43 minutes, by which point 2% of the monthly budget is gone, which is by design.

Why the short window

Without it, a one-hour window keeps the alert firing for up to an hour after a fix, because the bad minutes are still inside the window. Requiring the 5-minute window to agree means the alert stops within minutes of recovery, and it also stops a burst that has already ended from paging someone late.

Prometheus rules

Record the error ratio for each window, then alert on combinations. This example assumes a counter http_requests_total with a code label, for a service with job="checkout" and a 99.9% SLO.

groups:
  - name: slo:checkout:recordings
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout", code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout"}[5m]))
      # Repeat with [30m], [1h], [6h] and [3d], recording
      # ratio_rate30m, ratio_rate1h, ratio_rate6h and ratio_rate3d.

  - name: slo:checkout:alerts
    rules:
      - alert: CheckoutErrorBudgetBurn
        expr: |
          (
            job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
            and
            job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
          )
          or
          (
            job:slo_errors_per_request:ratio_rate6h{job="checkout"} > (6 * 0.001)
            and
            job:slo_errors_per_request:ratio_rate30m{job="checkout"} > (6 * 0.001)
          )
        labels:
          severity: page
        annotations:
          summary: Checkout is burning its 30-day error budget fast

      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          job:slo_errors_per_request:ratio_rate3d{job="checkout"} > 0.001
          and
          job:slo_errors_per_request:ratio_rate6h{job="checkout"} > 0.001
        labels:
          severity: ticket

• No for: clause is needed; the windows already provide the smoothing.
• The Workbook’s own example also opens a ticket at 3× over 24 hours with a 2-hour short window (10% of the budget in a day), alongside the 3-day rule.
• A 3-day rate() over raw series can be expensive. Many setups compute long windows from shorter recorded rates, or use a tool that generates these rules, such as Sloth or Pyrra.
• Latency SLOs work the same way: the error ratio becomes requests slower than the threshold (from histogram buckets) ÷ all requests.

Low traffic and very high SLOs

At low request rates a handful of failures is a large ratio. At 10 requests per minute, an hour holds 600 requests: nine failures in that hour (1.5%), one of them in the last five minutes (2% of 50), satisfy the 14.4× page for a 99.9% SLO. Options:

• Generate synthetic traffic so the denominator is large enough. Successful probes also dilute real failures, so keep probe traffic small relative to real traffic or track it separately.
• Combine small services that share a user journey into one SLO. A busy service can then hide a failing quiet one, so keep per-service dashboards.
• Require a minimum request count in the window, for example add and sum by (job) (increase(http_requests_total{job="checkout"}[1h])) > 100. A complete outage that stops traffic altogether then never alerts, so pair it with a traffic-drop or probe alert.
• Fix the product so a single failure is retried or has less user impact.

Two Prometheus details matter at any traffic level. If no request has ever failed, the error series may not exist, and the ratio returns no data instead of zero; initialise counters to zero for every label combination you alert on. With no traffic at all, the ratio is 0 ÷ 0, which is not a number and never exceeds a threshold, so missing traffic and missing telemetry need their own alerts, such as absent() or a traffic-drop rule.

Very high targets have the opposite problem. At 99.999% a 30-day budget is under 26 seconds of full outage, so even the fastest alert leaves almost no time to respond. Meeting such targets depends on making full outages rare and small, through redundancy, gradual rollouts and limited blast radius, and on automated mitigation; paging still matters for follow-up. Very low targets break the other way: at 90%, a full one-hour outage spends only 1.4% of a 30-day budget, so the 2%-in-one-hour page can never fire. Tune the parameters for such services.

Source: Google, The Site Reliability Workbook, ch. 5 “Alerting on SLOs”.