Incidents Reviewed 29 Oct 2026

Learning from incidents

A postmortem that names one root cause and a fix records what broke. Learning from an incident asks more: what combination of conditions made it possible, how it looked to the people inside it, and what they did that kept it from being worse. Those answers are where the lasting improvements come from.

Why one cause isn’t enough

Richard Cook’s How Complex Systems Fail argues that serious failures need several faults at once, each harmless alone, so there is no single root cause to find. Google’s books still use the term but ask for every contributing cause, and the second edition of the SRE book (2026) adds a method built on the same idea, described below; the postmortem template sets out where they differ. Two of Cook’s other points shape how to run a review:

• Hindsight bias. Knowing the outcome makes the warning signs look obvious. Cook calls it the main obstacle to accident investigation. Reconstruct what people could see at the time, not what the graphs show now.
• People create safety. Systems run with flaws all the time and mostly work because people adapt around them. An incident is a rare look at that adaptation: the workaround, the hunch, the person who knew whom to call.

Watch the language. “Should have”, “failed to” and “human error” describe what someone didn’t do, judged with knowledge they lacked. Write down what they did and why it made sense at the time. The same goes for outcome bias: judge a decision by the reasoning available when it was made, not by how it turned out.

A systems view: CAST

Google’s second edition describes CAST (Causal Analysis based on Systems Theory), a postmortem method from Nancy Leveson’s systems-safety work. It rests on STAMP, a causality model that rejects the idea of a single root cause: incidents happen when the system’s design does not safely constrain what its people and software do, so human and software errors are signs of a design flaw rather than explanations. Instead of asking which step in a chain of events failed, CAST asks what in the whole system let the incident happen. The book’s steps:

1. Assemble the basic information: the losses, the hazards (system states that can lead to a loss), a first timeline, and questions about it, without assigning responsibility.
2. Model the control structure as it was at the time: who or what controls each part of the system, and the feedback each controller gets.
3. Analyse each controller’s contribution: what it did, and why that made sense given the information and pressures it had.
4. Find the flaws in the control structure, including ones that grew slowly as conditions changed.
5. Plan improvements that address every causal factor found, not only the last step before the loss.

In the book’s worked example, a Google Maps incident, CAST produced more recommendations than the original postmortem, aimed at the process and its feedback rather than at individual steps. Its forward-looking twin, STPA, applies the same model to a design before anything has gone wrong. Leveson’s CAST Handbook (2019) covers the method in full.

Ask how, not why

Etsy’s debriefing facilitation guide recommends asking people “how” rather than “why”. “Why” invites a justification and builds the questioner’s own story; “how” gets a description of what happened and what people were thinking. The guide’s aim is to learn how work normally goes right, not only how it went wrong this once.

Before: what did you expect to see? What did "normal" look like?
Noticing: when did something first seem off? What drew your eye to it?
Sense-making: what did you think was happening? What else could it have been?
Options: what could you have done at that point? What made you pick this one?
Pressure: what else was going on? What were you trying not to break?
Help: what made you bring someone in then? How did you know who to call?
Tools: which graphs or logs did you trust, and which did you ignore? Why?
Handoffs: what did the next person need to know? How did they find out?
Surprises: what surprised you? What do you know now that you didn't then?
Normal work: how does this usually go right? What usually stops it going wrong?

Starting prompts, grouped by the part of the incident they open up. Follow the answers rather than the list.

Running a learning review

1. Pick a facilitator who wasn’t involved, ideally with a co-facilitator and a separate note-taker. Google’s second edition describes the same role as an independent postmortem lead, who gathers the information, interviews the people involved and looks at what went right as well as wrong.
2. Build the timeline first from chat logs, alerts, graphs, deploys and short one-to-one conversations with the people involved.
3. Invite everyone who appears in the timeline, and someone from each team that does.
4. Spend most of the meeting on the timeline. Etsy’s guide suggests 35 to 40 minutes of a 60-minute debrief. Ask people to note fixes as they think of them and hold them until the timeline is agreed; fuller context often makes early fixes moot.
5. Then collect learning points as a brainstorm, not tickets.
6. Turn them into actions later. A few people close to the incident review the list and meet again about two working days later to agree specific, owned actions.

What went well, and where you got lucky

Google’s example postmortems have a section for each. They are often the richest sections. What went well shows capacity worth protecting: a manual failover someone knew how to run, a team that joined quickly. Where you got lucky marks the defences that did not exist; the next incident may not be so lucky. John Allspaw suggests looking for the data people distrust or ignore, the tools they improvise mid-incident, and how experts explain to each other what to watch for.

Learning across incidents

• Choose what to study by impact and novelty, not by severity label alone, as the worked example shows. Cook notes that reviews after an accident nearly always find a history of earlier “proto-accidents”, and warns that arguing they should have been recognised beforehand rests on “naïve notions of system performance”. Study near misses when they happen, not in hindsight after the outage.
• Prefer depth to counts. Allspaw calls frequency, durations and severity counts “shallow data”: easy to collect, and averages hide more than they reveal. See MTTR, MTTD and MTBF for why the averages mislead.
• Look for themes: the same dependency, the same kind of change, the same confusing dashboard. The Workbook suggests a weekly report of the past week’s incidents, shared widely.
• Treat repeats as a signal. The Workbook’s questions: are action items too slow to close, is feature work winning over reliability fixes, were the right actions captured, is the service overdue for redesign, are fixes covering a deeper problem?
• Replay incidents as training. The Workbook uses past postmortems in Wheel of Misfortune exercises, with the original incident commander attending.

Is it working?

Etsy’s guide sets two simple tests for a debriefing: did at least one person learn something that will change how they work, and would at least half the attendees come to another? The guide adds that none of this works if people fear retribution, even the subtle social kind.

Sources: John Allspaw, Morgan Evans and Daniel Schauenberg, Debriefing Facilitation Guide (Etsy, 2016); Richard I. Cook, How Complex Systems Fail; Google, The Site Reliability Workbook, ch. 10 “Postmortem Culture: Learning from Failure”; Site Reliability Engineering, 2nd ed. (2026), ch. 11 “Learning from Incidents” (“Evolving the Learning from Incident Practices”) and ch. 12 “Safety Engineering for SRE with STPA and CAST” (“STAMP”, “CAST: A New Method for Postmortems”); John Allspaw, Moving Past Shallow Incident Data (Adaptive Capacity Labs, 2018).