On-call cheatsheet

When you are paged

1. Acknowledge so it does not escalate, then open the alert’s runbook.
2. Check user impact on the SLO dashboard: who is affected, how badly, since when.
3. Declare an incident if you need a second team, customers can see it, or it is unsolved after an hour. Open one channel, set a provisional severity, name the incident commander.
4. Post the first update soon after declaring, and say when the next one will come.
5. Look at recent changes: deploys, config, feature flags, traffic.
6. Mitigate first with a known-safe rollback, failover or feature switch-off, or by shedding load. Check for data migrations before rolling back, and the replica’s health before failing over. Find the cause later; keep the evidence.

Incident roles and communication

Severity at a glance

A starting point, not a standard. Classify by impact, not cause; when in doubt, go higher. A risk without impact, such as losing the last spare replica, still needs action now.

Incident severity levels

Escalation and handing over command

• No acknowledgement? The page goes to the secondary, then a manager or the wider team, after a fixed timeout such as 5 minutes.
• Need another team? Declare an incident and page their on-call. Do not wait to be sure.
• Handing over command: brief the next IC, say “You are now the incident commander, OK?”, wait for a yes, and announce it in the channel.
• Long incident? Rotate responders and the IC every few hours; tired people make mistakes.

On-call basics Incident roles

Internal update

Copy template

[SEV?] [Service]: [what users see]. Update [n], [HH:MM] UTC

Impact: [who is affected, how, since when]. [What still works.]
Current state: [what we know, what we are doing].
Next steps: [next action and owner].
Help needed: [none / team or person].
Next update: [HH:MM] UTC or sooner if anything changes.
IC: [name]   Comms: [name]   Channel: #inc-[id]

Send one on schedule even with no news. Say what users see, not component names.

More incident templates

Burn-rate alerts

burn rate = error rate ÷ (1 − SLO)

Error rate shown for a 99.9% SLO over 30 days; “empty in” is how long the budget lasts at that burn. Both windows must be over the threshold for the alert to fire. A 14.4× or 6× page is at least SEV2.

Burn-rate alerting Calculate for your SLO

Downtime allowed

Full outage at steady traffic. A partial outage spends the budget more slowly.

Availability table