Operations Updated
SRE on-call basics
Being on-call means being reachable and able to respond to production problems within an agreed time. Done well, it is a manageable part of the job. Done badly, it burns people out and makes them worse at it. The levers are rotation size, alert quality, tooling, and how follow-up work is handled.
On this page
Rotation sizing
Google’s SRE book caps on-call at 25% of an SRE’s time, with at least half of all time going to engineering work. From that rule:
- Single-site 24/7 rotation: a bare minimum of eight engineers, assuming a primary and a secondary at all times and week-long shifts. Each person is then on-call (primary or secondary) about one week in four.
- Two sites, follow-the-sun: a bare minimum of five engineers per site. With sites far enough apart, twelve-hour shifts during each site’s day largely avoid night pages.
- Plan for absence: the SRE Workbook adds one engineer per site to cover holidays, illness and departures, giving nine for a single site and six per site for two sites.
Smaller teams can still run on-call, but should know they are above that load. Options include business-hours only coverage for lower-severity alerts, sharing a rotation across teams that own related services, or reducing what pages at all.
Load per shift
The SRE book observes that handling an incident end to end, including root-cause analysis, remediation and postmortem follow-up, takes about six hours on average. That implies at most about two incidents per 12-hour shift. If on-call consistently sees more, the problem is the alerting or the system, not the engineer. Too few pages is a risk too: people lose familiarity with production and tooling, so practise with game days and drills.
Track, per shift or week: pages, pages outside working hours, time to acknowledge, incidents, and how many pages were actionable.
Primary, secondary and escalation
- Primary responds to pages first.
- Secondary is the fallback if the primary does not acknowledge in time, and helps during large incidents. Some teams give the secondary non-urgent operational work instead.
- Escalation policy: primary, then secondary, then a manager or the wider team, with explicit timeouts (for example, 5 minutes to acknowledge before escalating).
- Subject-matter escalation: on-call engineers should know how to pull in owners of other services without hesitation. Declaring an incident is usually the cleanest way.
What deserves a page
Every page should be:
- Urgent: it needs action now, not tomorrow morning.
- Actionable: a human can do something useful about it.
- User-relevant: it reflects real or imminent user impact, ideally via SLO burn-rate alerts rather than resource thresholds.
- Novel: if the response is always the same, automate it.
Everything else becomes a ticket, a dashboard or nothing. Review every page in a weekly on-call handoff and delete or downgrade alerts that did not meet the bar.
Handoffs
A short written handoff at the end of each shift should cover:
- Open incidents and their state.
- Pages received, which were noise, and alerts to fix.
- Risky changes in flight (migrations, launches, freezes).
- Anything the next person must watch or do.
Tooling and preparation
- Runbooks linked from every alert: what it means, how to check impact, first mitigation steps.
- Access that works when you need it: VPN, credentials, break-glass procedures tested in advance.
- A laptop and reliable connection on hand, and agreed response times for each severity.
- Shadow shifts before a new engineer goes primary, and a secondary while they gain confidence.
Keeping it humane
- Compensate on-call, with pay or time off. Unpaid availability is still work.
- Protect rest: after a night of pages, the engineer should be able to start late or hand over.
- Make swaps easy, and plan around holidays well ahead.
- Feed back: alert noise and recurring incidents should become prioritised engineering work, not a permanent tax; see toil.
Sources: Google, Site Reliability Engineering, ch. 11 “Being On-Call”; The Site Reliability Workbook, ch. 8 “On-Call”.