Operations Reviewed 29 Oct 2026

Overload and load shedding

A service is overloaded when it is asked for more work than it can finish before its clients give up. Left alone, it slows down for everyone, then fails for everyone. Load shedding keeps goodput flat instead: serve what fits quickly, reject the rest early and cheaply, and make the rejections visible.

Throughput and goodput

Throughput is every request sent to the server. Goodput is the part answered without error and soon enough for the client to use. Past its limit a server does not keep its throughput: it spends more time on contention, context switching and garbage collection, latency climbs past client timeouts, and work finishes after the caller has gone. AWS’s test of a good design is that goodput rises to a plateau and stays flat however much more load arrives. A design that needs fixing shows goodput falling towards zero.

Overload feeds itself. Requests that time out waste the work already done on them, clients retry and add load, and crashing servers push their share onto the rest; see timeouts, retries and backoff. Recovery is harder than prevention: the SRE book notes that a service which started crashing just above its capacity will not recover when load drops back to just below it, because only the few tasks that are up have to absorb all of it.

Shedding at the server

Each server should protect itself, because load balancers and quotas react late and imperfectly. Fail early and cheaply:

• Cap work in progress. The simplest form is to return 503 once more than a set number of requests are in flight, which the SRE book calls “one effective approach”. AWS’s David Yanacek found a fixed maximum-connections setting “too imprecise”, since the right value moves as workloads and dependencies change, and uses proxy connection limits only “as a last resort”, behind shedding that uses the server’s own information. Google’s servers shed on a utilisation signal, such as smoothed CPU or active threads against the cores available.
• Shed the least important first. Google tags every request with one of four criticality levels, propagates it to downstream calls, and rejects lower levels sooner. AWS ranks load balancer health checks above everything: a server that misses them is taken out of service, which shrinks the fleet just when it is short.
• Finish what was started. Prefer requests that complete existing work (the second page, the commit) over ones that start new work, and drop requests whose deadline has already passed.
• Make work bounded. Paginate large results so the cost of admitting a request is predictable.
• Make rejection cheap. A stray log line or slow socket setting can make each rejection costly. Load test far past the breaking point to measure it.

Queues

A queue in front of a worker pool turns overload into latency: a request waits behind everything ahead of it. With 40 workers taking 100 ms each, a full queue of 400 adds 1,000 ms before work starts. Against a 1-second client timeout, every request at the back is wasted.

• Keep queues short for steady traffic: the SRE book suggests no more than half the number of worker threads, 20 here, which adds at most 50 ms. Bursty traffic may need more.
• Bound time in queue, not only length. Record how long each request waited and drop it if it is too old.
• Prefer fresh requests under pressure. A LIFO queue serves the newest first, and CoDel drops requests once queueing delay stays high; both stop the server spending its time on requests whose users have already given up.
• Find the hidden queues: socket buffers, framework executors and load balancer surge queues. AWS prefers load balancers that reject excess traffic over ones that queue it.

Rejections aren’t free

Rejecting a request still costs parsing, a response and a connection. If rejecting costs 10% of serving, a server facing 10 times its capacity spends everything on rejections. Two remedies from Google’s SRE book:

• Per-customer quotas: when the service as a whole is overloaded, only the customers over their agreed share get errors.
• Client-side adaptive throttling: each client counts, over the last two minutes, the requests it attempted and the ones the backend accepted. Once requests exceed K × accepts it rejects new requests locally, before they reach the network, with probability:

reject locally = max(0, (requests − K × accepts) ÷ (requests + 1))

With Google’s preferred K = 2, an overloaded backend ends up rejecting about one request for each one it serves. A lower K wastes less backend capacity on rejections but notices recovery more slowly. It works poorly for clients that send requests only occasionally, because each has too little recent history.

Graceful degradation

Instead of rejecting a request, do less work for it: search a cached subset, skip pictures or recommendations, return slightly stale data. It must be designed in, and it fails in the ways rarely used code fails:

• Keep it simple and give it a fast off switch; complex degradation can trigger when it shouldn’t.
• Exercise it, for example by running a few servers near overload, so the path still works when needed.
• Alert when many servers are degraded; it should happen only after a capacity planning miss or a sudden shift in load.
• Measure it: a degraded response is a success to the load balancer but may not be one to users. A quality SLI can count it.

Shedding, autoscaling and load balancing together

The three are usually configured separately, by different people, and can work against each other. The Workbook and AWS describe the main traps:

• Scale before you shed. If shedding triggers at the same CPU level as autoscaling, CPU never climbs high enough to add instances.
• Errors look like spare capacity. In one Workbook case study a load balancer sent ever more traffic to a region whose servers were shedding, because fast errors made them look efficient. Make the load balancer count rejections as errors.
• Bound the autoscaler. Set a minimum and a maximum, keep enough quota to reach the maximum, and give on-call a documented way to turn it off. A stuck dependency or a CPU-burning bug can otherwise scale a service up until quota runs out, piling more load on the dependency.
• Plan headroom for losing a zone, and remember a fleet that sheds may be closer to its limit than CPU graphs suggest.

Make shedding visible

• Count shed and throttled requests by client and operation, and alert when the volume is significant.
• Keep rejected requests out of latency metrics. Fast rejections make a service’s median latency look excellent while it sheds most of its traffic.
• Graph offered load next to goodput, and queue time next to processing time, so a dashboard shows overload directly.
• Load test to the breaking point and beyond, with both a gradual ramp and a sudden jump, and check that the service recovers on its own when load returns to normal.

Overload runbook

Steps drawn from the SRE book; the order is this guide’s. A starting point to adapt into a runbook:

Copy runbook

1. Confirm it is overload: offered load above tested capacity, latency near the
   client timeout, saturation (CPU, threads, queue age) high on most tasks.
2. Add capacity if you have idle resources and tasks are not crash-looping.
3. Stop the feedback: turn off retries you control, pause batch and background
   load (index builds, copies, reports), block known bad or abusive traffic.
4. Shed or degrade on purpose: lowest criticality first, then degraded responses.
5. If tasks crash as soon as they become healthy: fix the trigger, cut traffic
   hard (for example to 1%), let tasks recover and caches warm, then ramp up.
6. Check health checks are not killing tasks that are only busy; restart only
   tasks that are wedged, gradually, after finding why.
7. Afterwards: load test past the breaking point and record where goodput fell.

Sources: Google, Site Reliability Engineering, 1st ed., ch. 21 “Handling Overload” and ch. 22 “Addressing Cascading Failures”; The Site Reliability Workbook, ch. 11 “Managing Load”; David Yanacek, Using load shedding to avoid overload (Amazon Builders’ Library).