Idempotency and deduplication

Why duplicates are unavoidable

A sender that gets no acknowledgement cannot tell whether the message was lost on the way out, processed and lost on the way back, or is still in flight. Its only options are to retry (risking a duplicate) or not retry (risking a loss). Any system that chooses not to lose data therefore chooses duplicates, which is why at-least-once is the default guarantee almost everywhere.

What makes an operation idempotent

• Naturally idempotent: setting a value, deleting by ID, or any write where applying it twice leaves the same state. `SET balance = 100` is safe; `balance = balance + 100` is not.
• Made idempotent with a key: the caller generates a unique identifier per logical operation and sends it with every attempt, including retries. The server records the key with its result.
• The key must come from the caller, not the server, and must be stable across retries. A key generated per HTTP attempt defeats the entire mechanism - this is the most common implementation bug.

How an idempotency key is actually stored

• On first receipt, insert the key with status `in-progress` using a unique constraint. The constraint, not an application-level check, is what makes this race-safe.
• If the insert conflicts, another attempt is already handling it: either wait and return its result, or return a conflict and let the client retry.
• On completion, store the response body and status against the key, so a later retry returns the identical response rather than re-executing.
• Keep the key write and the business write in one transaction. Split them and a crash in between leaves you with the duplicate you were trying to prevent.

Deduplication windows and why they expire

Storing every key forever is unbounded growth, so real systems keep a window - commonly 24 hours to a few days - matched to the maximum plausible retry horizon of the clients and brokers involved. A duplicate that arrives after the window is treated as new, which is a deliberate, bounded risk rather than a bug. If duplicates genuinely can arrive weeks later, the window is the wrong tool and you need a durable natural key instead.

Where exactly-once actually lives

"Exactly-once delivery" across a network boundary is not achievable; the two-generals problem is not an engineering oversight. What is achievable is exactly-once *effect*: at-least-once delivery plus idempotent processing. Systems that advertise exactly-once - Kafka transactions, for example - deliver it within their own boundary by coordinating offsets and writes atomically, and the guarantee ends where your external side effects begin.

The practical consequence: if your consumer writes to a third-party API, no amount of broker configuration makes that write exactly-once. It has to be idempotent on the other side, or guarded by a key you control. The broker-side mechanics are in Kafka interview questions.

Checklist for a design discussion

• Who generates the key, and is it stable across retries?
• Is the key insert protected by a unique constraint rather than a read-then-write?
• Are the key and the side effect committed together?
• What is the retention window, and does it exceed the longest client retry?
• What happens to a duplicate that arrives after the window?
• Which side effects are outside your transaction boundary, and are those idempotent?

Where CoPilot Interview fits

Disclosure: we build an interview assistant, so treat this as a disclosed pitch. CoPilot Interview is a native Windows and macOS app that puts suggested answers on your own screen in about two seconds, across coding, system design and behavioral rounds, with a permanent free tier that needs no credit card.

Frequently asked questions

How do you make an API operation idempotent?

Have the caller generate a stable key per logical operation and send it with every attempt including retries. Insert that key under a unique constraint, commit it in the same transaction as the business write, and store the response against it so a retry returns the original result rather than re-executing.

Is exactly-once delivery possible?

Not across a network boundary - a sender that receives no acknowledgement cannot distinguish a lost request from a lost response. What is achievable is exactly-once effect: at-least-once delivery combined with idempotent processing. Systems advertising exactly-once provide it only within their own boundary.

How long should you keep idempotency keys?

Long enough to cover the longest plausible retry from any client or broker in the path - commonly 24 hours to a few days. Keys are then expired to bound storage, which means a duplicate arriving later is treated as new. If duplicates can genuinely arrive weeks later, use a durable natural key instead of a window.

What is the most common idempotency bug?

Generating the key per HTTP attempt rather than per logical operation, so every retry carries a new key and the mechanism does nothing. A close second is checking for the key with a read before writing it, which races under concurrency - the unique constraint is what makes it safe.