Operations Updated
Toil: what it is and how to reduce it
Toil is the operational work that keeps a service running but does not make it better: manual, repetitive, automatable work that grows with the service. Some toil is unavoidable. Too much crowds out the engineering work that would remove it.
On this page
The six properties of toil
Google’s SRE book defines toil as work tied to running a production service that tends to be:
| Property | Meaning |
|---|---|
| Manual | A human runs it, even if that means running a script by hand. |
| Repetitive | It is done again and again, not solved once. |
| Automatable | A machine could do it as well, or the need could be designed away. |
| Tactical | It is interrupt-driven and reactive rather than planned. |
| No enduring value | The service is in the same state afterwards; nothing improved permanently. |
| Scales linearly | It grows in proportion to traffic, users or service size. |
Work does not need all six to be toil; the more it has, the more clearly it is toil.
Examples, and what is not toil
| Toil | Not toil |
|---|---|
| Manually restarting a service that leaks memory | Fixing the memory leak |
| Hand-approving routine access requests | Building self-service access with audit logging |
| Running the same release checklist for every deploy | Automating the pipeline that runs it |
| Resizing disks when an alert fires | Autoscaling storage or capacity planning |
| Acknowledging pages that need no action | Deleting or fixing those alerts |
| Copying data between systems for each new customer | Designing the onboarding flow so it is not needed |
Overhead such as meetings, planning, reviews and training is not toil either; it is necessary work that is not directly operational. Handling a genuinely novel incident is not toil, but the second time the same incident is handled by hand, it is heading that way.
The 50% cap
Google’s SRE organisation aims to keep operational work, including toil, below 50% of each SRE’s time, so that at least half goes to engineering that reduces future toil or adds features. The exact number matters less than having a cap, measuring against it, and acting when a team exceeds it, for example by handing work back to the development team or prioritising automation.
Measuring toil
- Tag tickets and pages by type and count time spent; categories reveal the biggest sources.
- Survey regularly: ask engineers what share of their week was toil and which tasks. Imprecise, but fast and repeatable.
- Review on-call handoffs: repeated manual steps show up there first; see on-call basics.
- Watch growth: toil that scales with customers or services will eventually outgrow the team.
Choosing what to automate
time saved over horizon ≈ frequency × minutes per occurrence × horizon − build time − upkeep
Start with tasks that are frequent, slow, error-prone or interrupt-driven at night. Count the cost of mistakes, not just the minutes: a quick task that goes wrong one time in fifty and causes an incident may cost more in incidents than in minutes. Weigh that expected cost against the cost of building and maintaining the automation, and remember that automation fails too, at machine speed, so give it safeguards and a limited blast radius. Also consider eliminating the task entirely by changing the design, which is often cheaper than automating it.
Ways to reduce toil, roughly in order of preference:
- Eliminate the need: fix the underlying bug, change the design, remove the feature.
- Self-service: let the requester do it safely without an operator.
- Automate the response, with safeguards and logging.
- Reduce frequency: batch it, or raise thresholds so it is needed less often.
- Document it well, as a stepping stone to automation, if none of the above is possible yet.
Sources: Google, Site Reliability Engineering, ch. 5 “Eliminating Toil”; The Site Reliability Workbook, ch. 6 “Eliminating Toil”.