Reliability targets Updated
RTO vs RPO
The recovery time objective (RTO) is the longest a service may stay down after a disruption. The recovery point objective (RPO) is how much data, measured in time, you can afford to lose. RTO looks forward from the failure; RPO looks back from it. Set both per service and per data store, then prove them with tests.
On this page
last recovery point ←— RPO —→ disruption ←— RTO —→ service restored
The two objectives
| Aspect | RTO | RPO |
|---|---|---|
| Question | How long until the service works again? | How much recent data can we lose? |
| Measured | From the interruption of service to its restoration | Back from the interruption to the last point you can recover to |
| Driven by | Detection, the decision to recover, failover or restore time, and verification | Backup frequency, replication lag, and how long corruption goes unnoticed |
| Reduced by | Standby capacity, automation, rehearsed runbooks | More frequent backups, continuous replication, point-in-time recovery |
| Near zero needs | A full-capacity standby that takes traffic within moments (hot standby), or serving from more than one site at once (active/active) | Synchronous or low-lag replication, plus backups for corruption |
Both are objectives, set before anything fails. What you actually achieve in a test or an incident is a measurement to compare against them. RTO is also not MTTR: MTTR is an average over past incidents, while an RTO is a limit for each disruption it covers.
How they relate to SLOs
Google Cloud’s disaster recovery guide says RTO and RPO are measurable and should be treated as SLOs. They answer a different question from an availability SLO. The SLO limits total unreliability across a window; the RTO limits a single disruption. Check that the two are consistent.
A 99.9% availability SLO allows 43m 12s of full outage per 30 days and 8h 45m 58s per year (see the availability table). One outage that lasts a 4-hour RTO spends 5.56 times the 30-day budget, so with a 30-day window a single such disaster breaks the SLO. Against a yearly window it uses 45.6% of the budget, and the SLO holds only if all other outages that year fit in the rest. If you accept that rare disasters will break a 30-day SLO, say how your error budget policy handles them. Otherwise, the recovery has to be much faster.
Recovery examples
Each example uses illustrative inputs. Replace them with figures measured on your own systems.
Bad migration deletes rows in the main database
The database takes a nightly full backup and archives its write-ahead log to object storage continuously, so it supports point-in-time recovery. A migration deletes rows at 14:00 and is noticed at 14:20.
- RPO: restore on a separate instance to just before the migration’s first destructive statement, then merge the deleted rows back with a checked script. Rows that production has changed since need a decision, not a blind copy. Restoring over production instead would lose every write made after the recovery point.
- RTO: restoring the full backup is often the longest step. At a measured restore rate of 200 MB/s, a 2 TB database takes 2h 46m 40s before replaying the log since the last full backup. The service may stay up during this time with the affected feature disabled, which shortens the user-visible outage.
- Lesson: restore time scales with data size. Measure it regularly; it grows as the data does.
A region is lost
The database replicates asynchronously to a second region, where a scaled-down copy of the application runs.
| Step | Time |
|---|---|
| Detect: the burn-rate page fires and is acknowledged | 5m |
| Decide: confirm the region is lost and authorise failover | 10m |
| Promote the replica and scale up the standby application tier | 10m |
| Shift traffic: DNS change, waiting out a 5-minute TTL | 5m |
| Recovery time | 30m |
- RPO: the replication lag at the moment of failure, often seconds. Monitor the lag and alert when it approaches the RPO; a lag that grows unnoticed quietly breaks it.
- RTO: detection and the decision to fail over take 50% of the time here. Automation shortens the mechanical steps, but someone still has to decide, so the runbook should state who can and on what evidence.
- Lesson: plan the way back as well. Failing back to the original region is another risky change.
Corruption or deletion copied to every replica
A bug or an attacker corrupts records. Replication copies the bad writes to every replica within seconds, so the replicas are no help. Recovery has to come from a backup taken before the corruption began.
- RPO: set by the last clean recoverable point before the corruption started, not by the latest backup. With snapshots only, that is the last clean snapshot, and later changes must be repaired or re-created. With a base backup plus retained write-ahead logs, you can recover to just before the corruption, if the logs reach back that far.
- Retention: if backups are kept for 7 days and the corruption goes unnoticed for 9, no clean backup exists. Retention must exceed the time it takes to notice. Google’s SRE book notes that most of its account-hijacking and data-integrity problems are reported or detected within 60 days.
- Lesson: replication protects against losing a site, not against bad data. Keep point-in-time backups that the production system and its credentials cannot delete.
Stateless services and derived data
- A stateless web tier has no data to lose, so the RPO does not apply. Its RTO is the time to deploy it elsewhere from infrastructure as code and images, then move traffic. Configuration and secrets are state, though; back them up.
- A pipeline’s output can often be recomputed. Its RPO then depends on how long the inputs are kept, for example the retention of the message queue or raw logs. Its RTO is the time to reprocess them.
Disaster recovery patterns
AWS describes four broad patterns. Lower RTO and RPO cost more to run and are more complex to operate.
| Pattern | What runs in the recovery site | RTO | RPO |
|---|---|---|---|
| Backup and restore | Backups only; infrastructure is redeployed during recovery | Longest: redeploy, then restore data | Time since the last backup |
| Pilot light | Replicated data stores; application servers are defined but switched off | Shorter: start and scale the application | Replication lag |
| Warm standby | A scaled-down but working copy of production | Shorter still: scale up and shift traffic | Replication lag |
| Multi-site active/active | Full production, serving traffic | Near zero for the loss of a site | Near zero with low-lag replication |
Every pattern still needs point-in-time backups. Replication alone does not protect against corruption or deletion, and for those even active/active has a non-zero recovery time and recovery point.
Setting and proving targets
- Set targets per data store and service tier. Historical compliance data can accept a long RTO; the payments database cannot. A single company-wide RTO fits nothing well.
- Weigh cost against loss. AWS advises not to put a recovery strategy in place if it costs more than the failure it addresses, unless something else, such as regulation, requires it.
- Test restores, not backups. Google’s SRE book puts it as “No one really wants to make backups; what people really want are restores.” Automate restore tests, run them continuously, and record how long they take.
- Monitor the recovery point. Alert when the newest backup is older than the backup interval plus a margin, and when replication lag approaches the RPO.
- Check recovery dependencies. Credentials, DNS, deploy tooling and the runbook itself must still work when the primary site is gone.
- Write the procedure down. Put each recovery in a runbook and rehearse it in game days.
Sources: Google, Site Reliability Engineering, ch. 26 “Data Integrity: What You Read Is What You Wrote”; Google Cloud, Disaster recovery planning guide; PostgreSQL, Continuous Archiving and Point-in-Time Recovery; AWS, Disaster Recovery of Workloads on AWS (recovery objectives and disaster recovery options).