Retry strategies, backoff and jitter

What should and should not be retried

Why exponential backoff alone is not enough

If a dependency fails for every caller at once, every caller computes the same backoff schedule and they all retry at the same instant. The service recovers, is immediately flattened by a synchronised wave, fails again, and the cycle repeats. Backoff spread the load in time but kept it perfectly aligned.

Jitter is the fix, and the usual recommendation is full jitter: sleep a random duration between zero and the computed backoff, rather than the backoff exactly. It costs nothing, desynchronises the herd, and in published analyses reduces both contention and total completion time compared with plain exponential backoff.

Retry budgets and amplification

Consider a chain of four services, each retrying three times. A single request at the edge can become 81 requests at the leaf. During an outage, retries do not help - they multiply load against something already failing. Two controls prevent this:

• Retry budget: allow retries only while they are a small fraction of total requests - a few percent is typical. Above that, fail fast. This caps amplification regardless of how many layers retry.
• Retry at one layer, not every layer. Pick the layer closest to the caller that can make a sensible decision and have the others fail fast.
• Circuit breaker: after a failure threshold, stop calling entirely for a cooldown, then probe with a single request before reopening.
• Deadline propagation: pass the remaining time budget down the chain, so a leaf does not start a 30-second retry sequence for a caller that gave up 10 seconds ago.

Parameters that are defensible

• Base delay in the tens to low hundreds of milliseconds for interactive paths; seconds for background work.
• Cap the per-attempt delay, typically tens of seconds, so backoff does not grow unbounded.
• Three attempts for interactive requests; more only for asynchronous work where latency does not matter.
• Always bound total elapsed time, not just attempt count - a user-facing request should fail inside the user's patience, not inside your retry policy.
• Send anything that exhausts retries to a dead-letter queue with enough context to replay it.

The surrounding reliability patterns are in microservices interview questions, and the delivery-guarantee half in exactly-once delivery in practice.

Where CoPilot Interview fits

Disclosure: we build an interview assistant, so treat this as a disclosed pitch. CoPilot Interview is a native Windows and macOS app that puts suggested answers on your own screen in about two seconds, across coding, system design and behavioral rounds, with a permanent free tier that needs no credit card.

Frequently asked questions

What is jitter in retry logic and why does it matter?

Jitter randomises the wait before a retry instead of using the exact computed backoff. Without it, every client that failed at the same moment retries at the same moment, so a recovering service is immediately flattened by a synchronised wave. Full jitter - a random wait between zero and the backoff - is the common recommendation.

Which HTTP errors should you retry?

Retry connection timeouts and resets, 429 and 503 while respecting Retry-After, and 500s on writes only when you have an idempotency key. Do not retry 400, 404 or 422, which are deterministic, or 401 and 403, where refreshing credentials is the correct action.

What is a retry budget?

A cap that permits retries only while they remain a small fraction of total traffic, typically a few percent, and fails fast beyond that. It prevents retry amplification, where several layers each retrying three times turn one edge request into dozens at the leaf during an outage.

How many times should you retry a request?

Around three attempts for interactive requests, more only for background work. More important than the count is bounding total elapsed time, so a user-facing request fails within the user's patience, and propagating deadlines so a downstream service does not keep retrying for a caller that has already given up.