Operations Reviewed 1 Nov 2026

Safe rollouts and canarying

The SRE Workbook defines canarying as a partial, time-limited deployment of a change and its evaluation: the part that gets the change is the canary, the rest is the control, and comparing them decides whether to continue. It matters because 68% of outage triggers in Google’s postmortem sample were binary (37%) or configuration (31%) pushes, so rolling out in stages and rolling back automatically when the two diverge costs a bad change a small slice of the error budget instead of all of it.

Canary and control

The SRE Workbook defines canarying as a partial, time-limited deployment of a change and its evaluation: the part that gets the change is the canary, the rest is the control, and comparing them decides whether to continue. It needs three things: a way to deploy to a subset of the service, an evaluation that judges the canary good or bad, and a release process that acts on the result.

• Compare at the same time, not before and after. Traffic and behaviour change through the day and week, so a before/after comparison mixes the change with everything else. Blue/green switches are before/after comparisons unless traffic is split between the two.
• Break every metric down by version. A canary on 5% of traffic barely moves service-wide graphs; in the worked example below the fast page never fires.
• Judge it from its callers too. A backend canary can return success while its callers fail to handle the new response; the backend looks healthy and the callers’ errors aren’t tied to any change of theirs. Google now has callers report errors and latency labelled with the backend version each request reached, and supervises the rollout on those. If your clients can’t do that yet, at least watch the immediate callers’ SLIs at each stage.
• Use a few metrics that indicate user harm. Start from your SLIs: errors and latency. The Workbook suggests no more than about a dozen; noisy metrics such as host CPU make canaries flaky, and flaky canaries get ignored.
• Exclude errors users cause, such as 404s from a shared broken link, and keep absolute checks against the SLO too: a bad canary can hurt the control it shares backends with.
• Match metric windows to the canary. An errors-per-hour metric cannot judge a 30-minute canary; the hour includes errors from before it started.
• Run one canary at a time. Overlapping canaries contaminate each other’s signals and are hard to reason about mid-incident.

Stages, size and bake time

Google’s SRE book says non-emergency rollouts must proceed in stages, of both binaries and configuration, with stages in different geographies to expose differences in daily traffic patterns and traffic mix. AWS’s published pipeline is a concrete version:

• Pre-production first, with a production-like stage that has the same deployment configuration, alarms and synthetic tests as production.
• One box: one instance or container, typically serving at most 10% of a zone or region, so a mixed fleet runs side by side. AWS rolls back on alarms scoped to that box alone.
• Rolling batches of at most 33% of a region’s instances, so at least two thirds of capacity keeps serving. AWS can do this because all its services are scaled to withstand losing an Availability Zone; size your own batches to the spare capacity you actually have. Some AWS teams use 5%.
• Waves of increasing size: first a low-traffic region, one zone or cell at a time, then more regions in parallel, never several zones of one region at once.
• Bake time after each stage to catch slow-burning problems: AWS’s typical pipeline waits at least an hour after one-box stages, 12 hours after the first regional wave and two to four hours after later ones, plus a minimum number of requests, so the new code has actually run.

Size and duration trade against release speed. The Workbook notes that a team releasing many times a day cannot run week-long canaries, that performance defects often appear only under peak load, and that the higher the request rate, the sooner a canary has a representative sample. Use the record of past canaries to tune stages rather than worst-case guesses.

Automatic rollback

• Roll back first, diagnose afterwards. The SRE book recommends it to shorten recovery; a rollback that needs a human decision waits for one.
• Alarm on what users feel. AWS rolls back on a high-severity alarm combining fault rate, latency and health checks, across the service and its neighbours, as well as on the one-box metrics.
• Block bad moments. Don’t start a stage while an alarm is firing or a wider incident is under way; AWS’s pipelines also allow starts only in working hours, excluding weekends, holidays and often Fridays and late afternoons, so someone can respond if rollback is not enough. An error budget policy can add a freeze when the budget is spent.

Changes that can’t simply roll back

A rollback is only safe if the old version can run alongside the new one and read anything the new one wrote. Data formats, protocols and schemas break this. AWS’s answer is a two-phase deployment:

1. Prepare: deploy a version that can read the new format but still writes the old one. Confirm every instance has it.
2. Bake for long enough, typically a few days, that you will not need to roll back past it.
3. Activate: deploy the version that writes the new format. Rolling it back is safe because the prepared version reads both.
4. Backfill old data before removing support for the old format.

Readers go before writers when rolling forward, writers before readers when rolling back. Test it: deploy to about half the fleet, then everywhere, then roll back the same way, checking for errors at each step.

Configuration and feature flags

Configuration pushes cause almost as many outages as code, so they need the same stages. The SRE book’s advice is to fail sanely: validate new configuration, alert on implausible input such as a file much smaller than the last one, and keep running on the previous configuration rather than accept it. Its examples include an empty file that made Google’s DNS answer NXDOMAIN for six minutes, and a single “/” that marked the whole web as malware.

Feature flags separate deploying code from turning features on, so one feature can be switched off without rolling back the binary. Roll flags out in stages like anything else, give them owners, and remove them when the launch is finished. Record the flag states each request ran with, so a problem that appears only with a particular combination of flags can be found.

Rollout plan template

For risky changes, or as the default for a pipeline. The stage values come from the AWS pipeline above; they are starting points, not standards.

Copy template

Change:        [what, ticket, owner]   Type: binary / config / flag / schema
Rollback:      [command or flag]; tested on [date]; safe after data written? [yes/how]
Compatibility: old and new versions can run side by side and read each
               other's data [yes / two-phase: prepare deployed on ...]
Signals:       [SLIs] broken down by version: canary vs control
Abort if:      canary error ratio > [x] or latency SLI below [y], or any page
               fires; roll back first, diagnose afterwards
Stages:        1 instance   bake ≥ 1h  and ≥ [n] requests
               [5%]         bake ≥ [h]
               one zone / region / cell at a time, [33%] per batch
               rest of fleet in waves of increasing size
Blockers:      no start during an incident, a change freeze or a spent error
               budget; inside [hours], not before weekends or holidays
Watch:         [person or automated alarm]; on-call told: [yes]

Sources: Google, The Site Reliability Workbook, ch. 16 “Canarying Releases” and appendix C “Results of Postmortem Analysis”; Site Reliability Engineering, 1st ed., appendix B “A Collection of Best Practices for Production Services”; Site Reliability Engineering, 2nd ed. (2026), ch. 14 “Testing” (“Client-Focused Change Supervision”) and ch. 17 “Designing for Resilience” (“Change Supervision”); Clare Liguori, Automating safe, hands-off deployments and Sandeep Pokkunuri, Ensuring rollback safety during deployments (Amazon Builders’ Library); Charity Majors, Liz Fong-Jones and George Miranda, Observability Engineering, 2nd ed. (2026), ch. 1 “What Is Observability?” (comparing by flag), ch. 9 “Observability-Driven Development” (rollbacks, single changes), ch. 24 “Systems Thinking for Software Delivery” and ch. 28 “The Organizational Shift” (deploy-day rules).