Designing webhook delivery

Why webhooks are harder than they look

You are making an outbound HTTP call to an endpoint you do not control, operated by someone whose uptime, latency and correctness are unknown to you. A single slow consumer can occupy your workers; a single broken consumer can accumulate an unbounded backlog. Every serious design decision follows from not letting one consumer affect another.

Delivery and retries

• Enqueue, never deliver inline. The event that triggers a webhook should commit to a durable queue and return. Delivering inside the request path couples your latency to the consumer's.
• Decaying retry schedule. Commonly something like seconds, then minutes, then hours, over a window of a day or several - the aim is to survive a consumer's deploy or short outage without hammering them.
• Per-endpoint concurrency limits, so one slow consumer cannot starve the pool. This is the bulkhead pattern applied to egress.
• Disable after sustained failure, with notification. An endpoint dead for a week should stop consuming resources.
• A dead-letter store plus manual replay, because consumers will ask you to resend.

Security: signing, not secrecy

• Sign the raw body with a shared secret - HMAC-SHA256 is the convention - and send the signature in a header. The consumer recomputes it over the exact bytes received.
• Include a timestamp in the signed payload and have consumers reject anything older than a few minutes, which prevents replay of a captured request.
• Compare signatures in constant time. A naive string comparison leaks timing information.
• Support two active secrets so consumers can rotate without downtime.
• Never put sensitive data in the payload if a thin event plus an authenticated fetch would do - it limits the blast radius of a misconfigured endpoint.

Ordering, and why you should not promise it

Retries alone break ordering: event A fails and is retried while event B succeeds immediately, so B arrives first. Guaranteeing order means serialising per consumer and blocking the queue behind any failure, which trades a small convenience for a large availability problem. The better contract is to be explicit that order is not guaranteed and give consumers the tools to cope:

• A monotonic sequence number or version per resource, so a consumer can discard anything older than what it has already applied.
• A stable event ID, so consumers can deduplicate - the consumer side of idempotency.
• State in the payload rather than deltas, where practical, so a missed event is self-correcting on the next one.

What consumers need from your docs

Respond 2xx immediately and process asynchronously; do not do work inside the webhook handler. Expect duplicates and out-of-order delivery. Verify the signature before parsing. Those three sentences prevent most integration incidents, and stating them plainly is part of the design.

Where CoPilot Interview fits

Disclosure: we build an interview assistant, so treat this as a disclosed pitch. CoPilot Interview is a native Windows and macOS app that puts suggested answers on your own screen in about two seconds, across coding, system design and behavioral rounds, with a permanent free tier that needs no credit card.

Frequently asked questions

How should webhook retries be scheduled?

On a decaying schedule spanning a day or more - seconds, then minutes, then hours - so a consumer can survive a deploy or short outage without being hammered. Pair it with per-endpoint concurrency limits, automatic disabling after sustained failure, and a dead-letter store that supports manual replay.

How do you secure a webhook?

Sign the raw request body with HMAC-SHA256 using a shared secret and send the signature in a header, include a signed timestamp so consumers can reject stale requests, compare signatures in constant time, and support two active secrets so keys can be rotated without downtime.

Can webhooks guarantee ordering?

Not in any design that retries, because a retried event arrives after a later one that succeeded first. Guaranteeing order requires serialising per consumer and blocking behind failures, which costs availability. The better approach is to document that order is not guaranteed and provide sequence numbers and stable event IDs.

Should webhook payloads contain sensitive data?

Prefer a thin event plus an authenticated fetch when the data is sensitive. A webhook goes to an endpoint you do not control, so a misconfigured or compromised consumer URL exposes whatever you put in the body - keeping payloads minimal bounds that blast radius.