Incidents Reviewed 29 Oct 2026

Incident roles and communication

A large incident needs someone coordinating who is not also debugging. Give the incident commander the overall picture, let an operations lead run the fix, and have a communications lead keep everyone else informed on a fixed schedule.

Why roles

The first edition of the SRE book traces unmanaged incidents to three hazards: the responder focuses so hard on the technical problem that nobody sees the bigger picture; communication stops, so stakeholders chase engineers for updates; and people freelance, making uncoordinated changes that can make things worse. Roles address all three. Google’s process, like PagerDuty’s, is adapted from the Incident Command System that US firefighters developed for wildfires, and the SRE Workbook sums up its goals as the three Cs: coordinate the response, communicate with responders, the organisation and the outside world, and keep control of the response.

When to declare an incident

Declare early. Closing an incident that turned out to be small costs little; starting coordination hours into a growing problem costs a lot. The first edition of the SRE book gives one team’s rule of thumb, which works as a starting point: treat it as an incident if any of these is true.

• You need a second team to help fix it.
• Customers can see the problem.
• It is still unsolved after an hour of concentrated work.

Pair this with your severity levels, which decide how large the response is.

The roles

A starting model for a major incident; scale it down for smaller ones.

The names vary. The second edition of Google’s SRE book (2026) names three primary roles: incident commander, communications lead and operations lead. The first edition also had a planning lead, which this guide keeps as an option for long incidents. When customers are affected, Google may split communications into an internal and an external communications lead. PagerDuty adds a deputy, who supports the IC and can take over command, and makes the same split as a customer liaison (usually from support) and an internal liaison. Not every role needs its own person: in a small incident the IC may also communicate and scribe. What matters is that the IC is named, everyone knows it, and leads add people when their load gets too high.

Running the response

• One place to talk. Open one incident channel, plus a call for large incidents, and move the discussion there. Side conversations are where freelancing starts.
• A live incident document. The first edition of the SRE book called keeping it the IC’s most important responsibility; the second edition gives documenting the state of the incident to the communications lead. Either way, name who owns it. Put the summary, roles and next update time at the top; it lets newcomers catch up without interrupting anyone. Keep it somewhere that does not depend on the systems that are failing.
• Short status reports. PagerDuty asks experts to report as condition (is the service healthy?), actions (what needs doing?) and needs (what help do they need?).
• Mitigate first. Prioritise reducing impact over finding the cause. Use a known-safe rollback, failover or feature disable where one fits; otherwise investigate just enough to choose a safe mitigation. A rollback past a data migration, or a failover to an unhealthy replica, can make things worse. Preserve evidence for the review.
• Explicit handoffs. When command changes, brief the new IC, then say it plainly, for example “You are now the incident commander, OK?”, and wait for a clear yes. Announce the change in the channel and update the document.
• Rotate on long incidents. Tired responders make mistakes. During an eight-hour response period in an incident described in the SRE Workbook, PagerDuty rotated its on-call engineers and its IC every four hours.

Communication

Three audiences need different things:

Update on a schedule, and every update says when the next one will come. A promised update with no news (“still working on it, next update 15:10”) is better than silence, which makes stakeholders interrupt responders. The severity levels guide suggests internal updates every 30 minutes for SEV1 and every 60 for SEV2 as a starting point.

For customers, PagerDuty’s published practice is to post a templated first message within 5 minutes of starting the incident call, a message scoping the impact within 5 minutes after that, then updates at least every 20 minutes for the first two hours, and also whenever the impact changes. After two hours, teams may choose a lower frequency for a long-running incident, while still announcing the next update and reporting meaningful changes in impact. The final message confirms recovery and reports any known data loss, corruption or lingering impact.

For external messages:

• Describe the symptoms customers see, in their words, not internal component names.
• Say who is affected and who is not, and what they can do in the meantime.
• Don’t guess at causes or promise a fix time you can’t keep; promise the next update instead.
• Prepare templates in advance and agree who approves messages, so the first post doesn’t wait on drafting.
• Make statements about billing, retries or data integrity only after they are verified. Say what is still being checked instead, and promise a public summary only if your organisation has agreed to publish one.

Templates

Starting points to adapt. Keep them where the on-call engineer can find them in seconds.

Incident document

Copy incident document

# [Service]: [short description]   [YYYY-MM-DD]

Status: active | mitigated | resolved        Severity: SEV[n]
Channel: #inc-[id]    Call: [link]           Next update: [HH:MM UTC]

Roles
  Incident commander:   [name]
  Operations lead:      [name]
  Communications lead:  [name]
  Planning lead:        [name, if used]
  Scribe:               [name]
  Next commander:       [name or "to be decided"]

Summary (comms lead keeps current)
  [Who is affected, how, since when, and what is being done.]

Exit criteria
  [ ] [Condition that means the incident is over, e.g. errors within SLO for 30 minutes]

Actions in progress
  [ ] [Action]  owner: [name]  started: [HH:MM]

Timeline (UTC, newest first)
  [HH:MM] [name] [what happened or was decided]

Internal update

Copy internal update

[SEV2] Checkout: card payments failing for some users. Update 3, 14:40 UTC

Impact: about 30% of card payments failing since 13:52 UTC. Other payment methods work.
Current state: failures follow this morning's release. Rollback started 14:31, about half complete.
Next steps: finish rollback; confirm failure rate returns to normal.
Help needed: none.
Next update: 15:10 UTC or sooner if anything changes.
IC: Sam   Comms: Priya   Channel: #inc-2141

Status page updates

Atlassian Statuspage offers four incident statuses: investigating, identified, monitoring and resolved.

Copy status page update

Investigating
  We are investigating failed card payments at checkout. Some customers may see an error
  when paying by card. We will post an update within 20 minutes.

Identified
  Card payments are failing for some customers in Europe because of a faulty change we made.
  We are reverting it. Other payment methods are working. Next update by 15:10 UTC.

Monitoring
  We have reverted the change and card payments are succeeding again. We are monitoring to
  confirm, and checking whether any payments made during the incident need correcting.
  Next update by 15:40 UTC.

Resolved
  Card payments failed for some customers between 13:52 and 14:58 UTC. Payments are working
  normally again. [Only once verified: what happened to affected payments, whether to retry,
  and any data loss or corruption. Otherwise say what is still being checked and when you
  will report.]

Practise

• Use the process for smaller things. The first edition of the SRE book notes that incident management skills fade quickly without use, and suggests applying the same structure to large planned changes.
• Rehearse. Role-play old incidents, or include incident command in disaster recovery tests.
• Rotate roles. If you were IC last time, take another role next time, so more people can do each one.
• Review the response, not just the bug. Ask in the postmortem whether roles were clear, updates went out on time and the right people were involved.

Sources: Google, Site Reliability Engineering, 1st ed., ch. 14 “Managing Incidents” and appendix C “Example Incident State Document”; Google, The Site Reliability Workbook, ch. 9 “Incident Response”; Site Reliability Engineering, 2nd ed. (2026), ch. 10 “Incident Management and On-Call” (sidebar “Incident Management at Google”); PagerDuty, Incident Response: Different Roles and External Communication Guidelines; Atlassian, Statuspage: Create an incident.