Operations Reviewed 29 Oct 2026

Alerting principles

A page should mean that users are being hurt, or soon will be, or that another serious harm such as lost money or data is under way, and that a person needs to act now. Alert on symptoms users feel, use causes for diagnosis, and send everything that can wait to a ticket or a dashboard. Noise is not harmless: it trains people to ignore the pager.

Pages, tickets and notifications

Google’s SRE book defines an alert as a notification meant for a human, delivered as a page, a ticket or an email. PagerDuty’s principle is narrower: an alert is something that requires a human to act; anything else is a notification, and notifications should never wake anyone. Its published priorities are a useful starting point:

The SRE book is sceptical of email alerts in particular: they are of very limited value and quickly fill with noise. A dashboard of ongoing, non-urgent problems usually serves better.

What deserves a page

The SRE book asks these questions of every paging rule:

• Does it detect an otherwise undetected condition that is urgent, actionable, and actively or imminently visible to users?
• Will I ever be able to ignore it, knowing it’s benign? If so, when, and how can that case be removed?
• Does it definitely mean users are affected? Filter out cases where they aren’t, such as drained traffic or test deployments.
• Can I act on it, and must the action happen now rather than in the morning? Could it be automated?

And it sets four tests for a page: you should be able to react with urgency, which people can only do a few times a day before fatigue sets in; every page should be actionable; every response should need intelligence, because a page that merits a robotic response should be automated; and pages should be about a novel problem.

Symptoms, not causes

A symptom is what is broken from the user’s point of view; a cause is why. Page on symptoms, and keep causes on dashboards to speed up diagnosis. A symptom alert says that something is wrong, not why, so the responder needs telemetry that can be broken down by endpoint, build, customer or host to find what the failing requests have in common.

Why symptoms:

• Coverage. A symptom alert catches causes of that symptom that nobody predicted. A list of cause alerts only catches the failures someone thought of. You still need a symptom for each way users can be hurt: an error-ratio alert won’t see wrong answers served with a 200, for example.
• Precision. Many causes don’t hurt users. A CPU spike that the service absorbs, or one failed replica behind a healthy load balancer, pages someone for nothing.
• Fewer duplicate pages. A single failure can trip dozens of cause alerts at once, but only the few symptoms users feel.

The Prometheus documentation puts it the same way: alert on high latency and error rates as high up the stack as possible, page on latency at only one point in a stack, and don’t page separately on lower-level errors that will show up as user-visible ones. Note that symptom and cause depend on where you stand: slow database reads are a symptom to the database team and a cause to the front-end team.

Google’s second edition of the SRE book (2026) still favours paging on SLOs, but calls “alert on symptoms” the previous edition’s mantra and builds on it in two ways. A static threshold, such as 0.05% errors for every service, wakes someone when a few requests fail on a quiet service and is too lax for a busy one, so Google learns each service’s normal behaviour with statistics or machine learning and alerts on deviations from it. And it keeps cause alerts that remain useful, such as quota and resource usage, configured centrally for every service so that nobody forgets one. The binomial test in low traffic and very high SLOs is one such statistical threshold. Observability Engineering (2nd ed., 2026) takes the opposite view of learned thresholds and of tools that group or suppress floods of alerts: it treats them as managing noise that should have been removed at the source, by deleting alerts that don’t reliably mean users are affected.

When to alert on a cause

• Definite and imminent. The first edition of the SRE book allows cause alerts only for very definite, very imminent causes, such as a disk that will be full in four hours. Whether that pages or opens a ticket depends on how soon it will hurt.
• Quota and resource exhaustion. The second edition names these as cause alerts that remain useful. Define them once for every service, not service by service.
• Severe but not user-visible. Prometheus suggests paging on failures users don’t see if they still need a human, for example when the business is losing a lot of money.
• Lost redundancy. Reduced redundancy has no user impact yet but brings the next outage closer. The SRE Workbook suggests a ticket when repair can safely wait. Losing the last spare may need a page, depending on how likely another failure is and how long recovery takes; the SRE book counts having no redundancy left as imminent.
• The monitoring itself. Alert when the monitoring or alerting pipeline is down, ideally with an end-to-end test that an alert actually arrives, and add an external black-box probe as a fallback.

Symptoms by system type

Check per request type as well as in total: Prometheus notes that problems in low-traffic requests can be drowned out by high-traffic ones.

What a page should contain

PagerDuty’s guidelines for alert content:

• A descriptive, specific title: “Disk is 80% full on prod-web-lb-1”, not “Something went wrong”.
• The metric or query that triggered it.
• What the problem is and why it matters.
• Steps to fix it, or a link to a runbook. PagerDuty calls alerts with neither useless.

In Prometheus, put these in annotations. This example extends the fast-burn page from the burn-rate guide with a summary, impact, dashboard and runbook link:

groups:
  - name: checkout
    rules:
      - alert: CheckoutErrorBudgetBurn
        expr: |
          job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
        labels:
          severity: page
          team: checkout
        annotations:
          summary: "Checkout failed {{ $value | humanizePercentage }} of requests over the last hour"
          description: >
            Both the 1-hour and 5-minute error ratios exceed the fast-burn threshold
            for the 99.9% SLO (14.4x the budget rate). Customers see errors at payment.
          dashboard: https://grafana.example.com/d/checkout-slo
          runbook_url: https://runbooks.example.com/checkout/error-budget-burn

Fewer pages per problem

A network partition can make hundreds of instances fail the same check at once. Alertmanager has three tools to turn that into far fewer, more useful notifications:

• Grouping combines alerts that share labels, such as the alert name and cluster, into a single notification that still lists every affected instance.
• Inhibition mutes some alerts while another is firing. If a whole cluster is unreachable, there is no point also paging for each service in it that is unreachable for the same reason.
• Silences mute matching alerts for a set time, for example during planned maintenance. Give each one an expiry and a reason, so a silence doesn’t outlive its purpose.

An example routing and inhibition fragment; it omits the receiver definitions, which you must supply:

# Routing and inhibition fragment: define the receivers (pager, ticket queue) yourself.
route:
  receiver: checkout-tickets
  group_by: [alertname, cluster]   # one notification per alert name and cluster, not per instance
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: checkout-pager

inhibit_rules:
  # While a cluster is unreachable and paging, mute only the per-service
  # reachability alerts that it explains, in the same environment and cluster.
  - source_matchers: [alertname="ClusterUnreachable", severity="page", cluster=~".+"]
    target_matchers: [alertname=~"ServiceUnreachable|ProbeFailed", cluster=~".+"]
    equal: [environment, cluster]

Inhibition is easy to get dangerously wrong:

• Mute only what the source explains. Target named dependent alerts, not every page. A rule that mutes all pages in a cluster also mutes unrelated emergencies such as data corruption or a security alert.
• Match on non-empty labels. Alertmanager treats a label missing on both alerts as equal, so equal: [cluster] alone can let an alert with no cluster label mute alerts everywhere. Require the label on both sides, and include the environment if cluster names repeat across staging and production.
• Inhibit only from a page. If the source is a ticket-level alert, the page it mutes is never replaced by anything urgent.
• Silences don’t lift inhibition. A silenced source alert still mutes its targets, so silencing the parent can leave nobody paged. Test that combination.

Keeping noise down

• Set a pager budget. Google SRE targets at most two distinct incidents per on-call shift, counting all the alerts for one problem as a single incident. If a rotation regularly exceeds its budget, treat it as a reliability problem to fix, not a cost of the job.
• Review new alerts like code. The SRE Workbook suggests that the whole team reviews proposed paging alerts, each has a playbook entry, and each is trialled in production without paging, for example by emailing its author, to check false positives and predict its effect on the pager budget.
• Allow for blips. Prometheus advises slack for small blips. Burn-rate windows provide it; a for: duration also works but delays every alert equally, including severe ones.
• For request errors, prefer a ratio to a raw count. “More than 50 errors a minute” fires on busy days and misses outages on quiet ones. The error ratio normalises for volume, though at very low traffic it needs a statistical threshold; see low traffic and very high SLOs. Keep absolute signals where each failure matters on its own, such as failed payments.
• Test alerts. PagerDuty treats an untested alert as no alert. Check the threshold isn’t noisy, that missing data is caught where it matters, and that the alert clears by itself when the metric recovers.
• Follow up every page. The Workbook warns against closing pages as “transient” or “fixed itself”: find the cause, fix the class of problem where you can, and link repeat pages to an existing bug so their cost is visible.
• Watch what incident reviews add. Postmortems often add an alert for the cause just found. Each makes sense alone, but together they fire all at once in the next incident. Prefer fixing the symptom alert’s coverage, and retire alerts as you add them.
• Review what isn’t used. The SRE book says alerting configuration that is rarely exercised (for some Google SRE teams, less than once a quarter) should be up for removal, and signals that no dashboard or alert uses are candidates for removal. Keep rare alerts that guard against serious failures, and exercise them deliberately so you know they still work.
• Automate rote responses. A page that always gets the same scripted fix should become automation; the SRE book calls a team’s reluctance to do this a red flag. Keep a page for when the automation fails or doesn’t restore service.

One caution from the Workbook: for alerts already based on SLOs and symptoms, loosening the threshold is rarely the right answer to being paged. The page means users were hurt; fix the system, or revisit the SLO deliberately.

Common mistakes

• Paging on resource thresholds such as “CPU above 80%” when users are unaffected.
• Paging at every layer of the stack for the same failure.
• Alerts with no runbook, dashboard or owner.
• Sending “informational” alerts to the pager or a busy email list, where they teach people to ignore alerts.
• Leaving permanent silences in place instead of fixing or deleting the alert.
• No alert when telemetry stops, so a dead exporter looks like a healthy service.

Sources: Google, Site Reliability Engineering, 1st ed., ch. 6 “Monitoring Distributed Systems”; The Site Reliability Workbook, ch. 8 “On-Call”; Site Reliability Engineering, 2nd ed. (2026), ch. 9 “Observability and Monitoring” (“From Static Thresholds to Intelligent Alerting”) and ch. 10 “Incident Management and On-Call” (“Optimal Paging Load”); Prometheus, Alerting, Alerting rules and Alertmanager; PagerDuty, Incident Response: Alerting Principles; Charity Majors, Liz Fong-Jones and George Miranda, Observability Engineering, 2nd ed. (2026), ch. 8 “Getting Started with Observability Analysis” (the core analysis loop) and ch. 11 “Using Service Level Objectives for Reliability” (what versus why, alerts added after incidents, AIOps).