Field reference Updated
The numbers and practices SREs look up every week
A site reliability engineering (SRE) reference: downtime per nine, error budgets, burn-rate alerts, incident templates and one-page printable cheatsheets. Numbers are calculated, and every guide cites its sources.
- 43m 12sdowntime 99.9% allows in 30 days
- 4m 19sdowntime 99.99% allows in 30 days
- 14.4×Workbook starting threshold for a page: 2% of a 30-day budget in an hour, over both 1h and 5m windows
- 50hto spend a full 30-day budget at that constant burn rate, starting from none spent
Tools
Reliability targets
- SLI vs SLO vs SLAWhat each term means, how to choose indicators and targets, and example SLOs by service type.
- Error budget policyWhat happens when the budget runs out, with a policy template you can adapt.
- RTO vs RPORecovery time, recovery point and restore window objectives, with worked recovery examples and DR patterns.
- Multiwindow, multi-burn-rate alertingWhy burn-rate alerts beat threshold alerts, the Workbook’s starting thresholds, PromQL for SLIs and Prometheus rules.
- Latency percentilesp50 to p99.9, why averaged percentiles mislead, latency SLOs that add up, and tail latency at scale.
Incidents
- Incident severity levelsA SEV1–SEV4 matrix with criteria, response expectations and common pitfalls.
- Incident roles and communicationIncident commander, ops and comms leads, handoffs, update cadence, and templates for incident updates.
- Blameless postmortem templateA copyable incident review template and guidance on writing one without blame.
- Runbook templateA copyable Markdown runbook for on-call alerts, with a filled example and upkeep advice.
- Learning from incidentsBeyond root cause: hindsight bias, facilitated learning reviews, questions to ask and patterns across incidents.
- MTTR, MTTD and MTBFWhat each incident metric measures, how MTBF and MTTR give availability, and why the averages mislead.
Operations
- On-call basicsRotation sizing, shift load, escalation, handoffs and keeping on-call sustainable.
- Alerting principlesWhat deserves a page, symptoms versus causes, alert content, grouping, and cutting alert fatigue.
- Toil: definition and how to reduce itThe six properties of toil, how to measure it, and what to automate first.
- Production readiness review checklistWhat a PRR covers, how to run one, checking an SLO against its dependencies, and a copyable checklist.
- Safe rollouts and canaryingCanary versus control, stage sizes and bake times, automatic rollback, rollback-safe data changes and a rollout plan template.
- Timeouts, retries and backoffChoosing timeouts, propagating deadlines, backoff with jitter, retry budgets and stopping retry storms.
- Overload and load sheddingGoodput versus throughput, shedding by criticality, bounded queues, client-side throttling and an overload runbook.
- Golden signals, RED and USEThree monitoring frameworks compared, with what to measure for services and resources.
Reference
- On-call cheatsheetOne page for when you are paged: first steps, severity, downtime, burn-rate thresholds and an update template.
- SLO design cheatsheetOne page for writing or reviewing an SLO: a spec template, SLI types, targets, budgets per target, burn-rate alerts and policy steps.
- Drill: should this page someone?A printable tabletop discussion drill: sort five alerts into page, ticket or nothing, with facilitator notes.
- Drill: budget 80% spent, 10 days leftA printable tabletop discussion drill: forecast the month and decide what to do, with facilitator notes.
- Drill: write the first updateA printable tabletop discussion drill: write the first incident update under interruptions, with facilitator notes.
- SRE glossaryConcise definitions of reliability, incident and operations terms.