Operations Reviewed 29 Oct 2026

Timeouts, retries and backoff

Every remote call needs a timeout, a deadline it inherits from its caller, and a retry policy that cannot multiply load when a dependency is struggling. Retry at one layer, a few times, with jittered exponential backoff and a budget that stops retries when they stop helping.

Timeouts

Set a timeout on every call to another process, covering both connecting and the request itself. A call without one holds a thread, a connection and memory for as long as the dependency stays silent. The timeout value also caps how much work piles up: requests in flight = arrival rate × time each is held. A dependency that hangs turns a long timeout into thousands of stuck requests.

AWS’s approach is to choose an acceptable rate of false timeouts, the share of calls that would have succeeded but are cut off, and set the timeout at the matching percentile of the dependency’s latency: 0.1% false timeouts means the p99.9. Adjust it in three cases:

• Clients far away, such as over the internet: add a realistic worst-case network delay.
• Tight latency, where p99.9 is close to p50: add padding, or a small slowdown times out a large share of calls.
• Hidden steps: check that the timer covers DNS, the TLS handshake and connection setup, or that it deliberately excludes them. A new connection can be slower than every request that reuses it.

The SRE book warns against timeouts several orders of magnitude above the normal latency: if a small share of requests stop completing, they hold threads for the whole timeout and starve the requests that would have succeeded. Failing fast, with an immediate error when a backend is known to be down, avoids the wait altogether.

Deadlines and cancellation

A per-hop timeout chosen in isolation can outlive the request it serves. With deadline propagation, the frontend sets one absolute deadline and every downstream call gets whatever time is left, less a little for the network and for processing the reply. A server that receives a request whose deadline has passed should drop it: the caller has given up, so the work earns nothing. For multi-stage requests, check the remaining time before each stage.

Cancellation propagation does the same for requests that can no longer succeed: when a call fails permanently or a hedged copy wins, tell the other branches of the call tree to stop. Some gRPC implementations propagate deadlines automatically (Java and Go by default, C++ when enabled), sending each hop the time remaining rather than a clock time. Even then, the server code must stop the work it started when a call is cancelled; check what your own framework does.

When to retry

Most failures in distributed systems are partial or brief, so a second attempt often succeeds. AWS calls retries “selfish”: each one spends more of the server’s capacity to raise one client’s chance of success. That is a good trade when failures are rare and a bad one when the server is failing because it is overloaded.

• Retry only what is safe to repeat. A timeout does not mean the call did nothing. Reads are usually safe; writes need an idempotency key so a repeat has no extra effect.
• Retry only errors that can change. Don’t retry malformed requests or permanent errors; they fail the same way every time. Eventual consistency blurs the line: a “not found” can become a success once state propagates.
• Respect “don’t retry”. The SRE book recommends a distinct overload response that tells callers not to retry, so one overloaded layer does not trigger retries above it.
• Retry at one layer. If every layer retries, attempts multiply: 64× with 4 attempts at 3 layers. AWS’s five-layer example says “three retries at each layer” and multiplies by 3 at each layer, for 243×. The SRE book retries only at the layer immediately above the one that rejected the request; for low-cost operations AWS retries at a single point in the stack.
• Limit attempts per request: Google’s overload handling allows three in total. Stop sooner if the deadline would pass anyway.

Retry budgets

A per-request limit is not enough. If a whole cluster rejects most requests, three attempts each still nearly triples the load on it. A retry budget limits retries across all requests from a client or process:

• Retry ratio (Google): retry only while retries are below 10% of the client’s requests. The SRE book says this cuts the growth to about 1.1× rather than nearly 3×.
• Token bucket (AWS): retries spend tokens from a local bucket; once it is empty, retries continue only at the bucket’s fixed refill rate. AWS SDKs have done this since 2016.
• Per-process cap: the SRE book’s example allows 60 retries a minute in a process and fails requests once it is spent.

All three let retries cover scattered, transient errors and switch them off when errors are widespread, which is exactly when retries make things worse.

Backoff and jitter

Retrying immediately and repeatedly hits a struggling server at the worst moment. Capped exponential backoff doubles the wait before each retry up to a maximum. On its own it still synchronises clients: everything that failed together retries together. Jitter spreads them out. In AWS’s simulation, “full jitter”, a uniformly random wait between zero and the capped backoff, did the least work of the methods tested; “decorrelated jitter” finished slightly sooner for slightly more calls, and un-jittered backoff was the worst on both.

wait before retry n = random(0, min(cap, base × 2^(n − 1)))

Base 25 ms and cap 1 s, as in the policy below; with full jitter the mean wait is half the longest. Pick the cap so the attempts you allow still fit inside the caller’s deadline.

Add jitter to periodic work too: cron jobs, cache refreshes and clients that poll once a minute line up on the first seconds of the minute or the hour. For scheduled work AWS picks jitter that stays the same on each host rather than drawing it at random, so any overload it causes repeats in a pattern that people can spot.

A retry policy to start from

These values are starting points from the sources above, not standards. Record what you choose next to the client code.

# Starting point for one client calling one dependency. Tune it from your own data.
timeout:   the dependency's p99.9 latency, for about 0.1% false timeouts;
           add padding if p99.9 is close to p50; cover connect, TLS and DNS
deadline:  pass the caller's remaining deadline down; never wait longer than it
attempts:  3 per request (1 + 2 retries), at one layer only: the one
           directly above the dependency that failed
retry:     connection failures, timeouts and transient server errors, and only
           for idempotent requests or ones that carry an idempotency key
no retry:  malformed or unauthorised requests, permanent errors, and any
           "overloaded, don't retry" signal; honour Retry-After when sent
backoff:   full jitter: wait random(0, min(cap, base × 2^(n − 1))) before retry n,
           base 25 ms, cap 1 s
budget:    retries at most 10% of requests per client (or a token bucket);
           when it is spent, fail fast instead of retrying

Hedged requests

A hedged request is a retry sent before the first attempt fails: after a short delay, send a second copy to another replica and use whichever answers first. Dean and Barroso suggest waiting until the first request has been outstanding longer than the 95th percentile for its kind, which limits the extra load to about 5%. The same rules apply as for retries: idempotent operations only, cancel the loser, and cap the extra load. See latency percentiles for the measured effect on the tail.

Retry storms and cascading failure

A cascading failure grows through positive feedback: an overloaded server fails, its load moves to the others, and they fail in turn. Retries feed it. Requests that time out are sent again, the server spends its capacity on work whose callers have already given up, and load stays high after the original trigger has gone. Signs during an incident:

• Requests reaching the dependency rise while user traffic is flat; the gap is retries.
• Success rate stays low after the trigger is fixed or rolled back.
• Latency sits at the timeout value: requests are waiting the full time, not failing fast.

The SRE book notes that retry graphs can be mistaken for a symptom rather than a cause. Fixing retry behaviour usually needs a code push, so the immediate options are to cut load sharply, or stop traffic and ramp it back gradually; see overload and load shedding. Graph retries separately from first attempts so the gap is visible.

Sources: Google, Site Reliability Engineering, 1st ed., ch. 22 “Addressing Cascading Failures” and ch. 21 “Handling Overload”; Marc Brooker, Timeouts, retries, and backoff with jitter (Amazon Builders’ Library) and Exponential Backoff And Jitter (AWS Architecture Blog, 2015); Jeffrey Dean and Luiz André Barroso, The Tail at Scale (Communications of the ACM, 2013); gRPC, Deadlines.