Handling traffic spikes

Why autoscaling is not the answer on its own

Autoscaling reacts on a loop measured in tens of seconds to minutes: metrics have to be scraped, a threshold crossed, instances provisioned, containers pulled, applications started, health checks passed. A spike that arrives in ten seconds is over, or has already taken the service down, before capacity arrives. Autoscaling handles growth and diurnal load well; it does not handle spikes.

• Pre-warm when the spike is predictable - a launch, a sale, a scheduled broadcast.
• Keep headroom rather than running at the efficiency frontier, because headroom is what absorbs the first thirty seconds.
• Scale on queue depth or latency, not CPU, when the work is queue-driven - CPU lags the thing you actually care about.

Queueing versus shedding

A queue converts a load problem into a latency problem - which is right for work that can be done later, and wrong for work nobody will want by then. If requests arrive faster than you can serve them, an unbounded queue grows until every response is past the client's timeout: you then do the full amount of work and nobody receives any of it. This is the failure mode worth naming explicitly.

• Bound every queue, and decide what happens when it is full.
• Drop oldest, not newest, for latency-sensitive work: the oldest item is most likely already abandoned.
• Check the deadline before processing - if a request's budget has expired while it sat in the queue, discard it instead of spending capacity on a response nobody will read.

How to shed well

• Reject early, at the edge, before the request consumes a connection, a worker or a database handle.
• Return 429 with Retry-After so well-behaved clients back off rather than hammering - and expect that not all of them will.
• Shed by priority. Health checks and payments before avatars and recommendations. This requires classifying traffic in advance, which is the actual work.
• Shed per tenant, so one customer's spike does not consume the shared budget.

Degrading on purpose

The best spike responses reduce the cost of a request rather than refusing it. Serve a cached or slightly stale response instead of a fresh one. Drop personalisation and serve the generic page. Disable the expensive recommendation module and keep checkout. Each of these is a decision made calmly in advance and toggled under load - which is why a feature flag for the expensive path is worth more during an incident than another replica.

The caching mechanics are in Redis interview questions, and the rate-limiting primitive in design a rate limiter.

Where CoPilot Interview fits

Disclosure: we build an interview assistant, so treat this as a disclosed pitch. CoPilot Interview is a native Windows and macOS app that puts suggested answers on your own screen in about two seconds, across coding, system design and behavioral rounds, with a permanent free tier that needs no credit card.

Frequently asked questions

Why is autoscaling not enough for traffic spikes?

Its control loop takes tens of seconds to minutes - scrape metrics, cross a threshold, provision, pull images, start, pass health checks. A spike arriving in ten seconds is over or has already caused an outage before capacity appears. Autoscaling handles growth and daily cycles; headroom and shedding handle spikes.

Should you queue or shed excess traffic?

Queue work that is still valuable later and bound the queue; shed work that will be stale by the time you serve it. An unbounded queue is the worst outcome - you do all the work and every response arrives after the client's timeout, so nobody is served.

How do you shed load correctly?

Reject at the edge before the request consumes a worker or database connection, return 429 with Retry-After, prioritise so critical paths survive, and shed per tenant so one customer cannot exhaust the shared budget. Rejection must be far cheaper than service, or shedding becomes its own overload.

What does graceful degradation look like under load?

Reducing the cost of a request rather than refusing it: serving cached or slightly stale responses, dropping personalisation, and disabling expensive non-essential modules while preserving critical paths such as checkout. These are pre-planned toggles, which is why a feature flag is often worth more mid-incident than another replica.