Reference Reviewed
Drill: write the first update
Ten minutes to write the first internal update and a status page message while the facilitator interrupts. About 30 minutes. Built on incident roles and communication.
PDF, two sheets: A4 · US Letter
How to run this drill
- A discussion drill worked from a handout: a lighter take on the SRE book’s live disaster role playing (“Wheel of Misfortune”).
- One facilitator; participants write alone or in pairs. Print the page: give out the first sheet, keep the facilitator notes on the second.
- Read the handout aloud, start a ten-minute timer, and play the two interruptions in the notes. Then read the drafts aloud and compare.
- No scores and no record of who wrote what. Finish by comparing the drafts with your own update template.
Handout
You are Priya, the communications lead. You have ten minutes to write the first internal update, then a two-sentence status page message; the facilitator will interrupt twice. What you know at 14:16 UTC:
- 14:02: card payments at checkout started failing for customers in the EU. About 30% of EU checkout attempts fail. The EU is 40% of checkout traffic.
- 14:09: the burn-rate page fired. 14:12: Sam declared an incident and opened the call and #inc-2207; Sam is incident commander.
- 14:14: Sam asked Ana to prepare a rollback of the deploy and paged the payments team.
- That deploy went out at 13:55. The payment provider’s status page shows no problems.
- Other regions and other payment methods look normal. Nobody knows the cause yet.
The service has a 99.9% availability SLO over 30 days.
Facilitator notes
Interruptions: at minute 4, as a vice-president, ask “When will it be fixed?” At minute 7, as an engineer, say in passing “It’s probably the payment provider again.”
- Size it. Severity follows impact, so check the numbers:30% × 40% = 12% of all checkouts fail, a 120× burn; paged after 7m 12s; the month’s budget would be gone in 6h, and the first hour alone spends 16.7%A core feature down for some users fits SEV2 on this site’s scale, so internal updates every 60 minutes; some teams would call it SEV1 for a payment path.
- Did they hold the line? A good first update gives no time for a fix and no cause that isn’t confirmed. The provider guess has nothing behind it, and the evidence points elsewhere (their status page is clear; a deploy went out 7 minutes before the failures began). It must not reach the update.
- Did it say what users see? “Card payments failing for some EU customers” beats “elevated 5xx on payment-svc”.
- Did it promise the next update? A time, not “soon”.
- Was the customer message late? In the cadence the incident roles guide takes from PagerDuty, a templated status page message is due within 5 minutes of the call starting, by about 14:17, before the internal update can be ready. Then a message scoping the impact by about 14:22, and updates at least every 20 minutes after that. Ask who should have posted it while the internal update was being drafted.
One possible answer, in the update template from the incident roles guide:
[SEV2] Checkout: card payments failing for some EU customers. Update 1, 14:18 UTC
Impact: since about 14:02 UTC around 3 in 10 checkout attempts from EU customers
fail at the payment step. Other regions and other payment methods look normal.
Current state: cause not yet known. A payment service deploy went out at 13:55;
the payment provider reports no problems.
Next steps: Ana is preparing a rollback of the 13:55 deploy.
Help needed: payments team, paged to join the call.
Next update: 14:45 UTC, or sooner if anything changes.
IC: Sam Comms: Priya Channel: #inc-2207Status page: “Some customers in Europe can’t pay by card at checkout. We’re investigating and will post an update within 20 minutes.” Compare drafts: what did each leave out, and what did each say that wasn’t known?
Sources: Google, Site Reliability Engineering, 1st ed., ch. 28 “Accelerating SREs to On-Call and Beyond” (disaster role playing, “Wheel of Misfortune”, in which a game master plays out an outage live while one or two on-callers ask questions and act; this drill adapts that format as a discussion from a handout rather than following it); The Site Reliability Workbook, ch. 5 “Alerting on SLOs”; PagerDuty, External Communication Guidelines (customer message timing). The scenario is invented for practice; severity and update cadence follow this site’s severity levels, a starting point, not a standard.