Design a Distributed Message Queue: System Design Walkthrough

To design a distributed message queue, model each topic as an append-only log split into partitions, spread those partitions across brokers, and replicate every partition to a leader and followers so an acknowledged write survives a broker failure. Producers append to partition leaders, consumers pull from offsets they track themselves, and consumer groups divide partitions among workers. Most of the interview is about trade-offs: ordering versus parallelism, durability versus latency, and at-least-once versus exactly-once delivery.

This walkthrough follows Apache Kafka, the design most interviewers have in mind. Where a number helps, it cites defaults from the current Kafka documentation (version 4.3); defaults change between releases, so treat them as talking points.

1. Clarify the requirements

First ask the question that changes the whole design: a classic work queue or a replayable log?

Most interviewers want the log, so design that and add queue-style behavior on top. Then pin down:

• Functional: publish to topics, subscribe, retain and replay messages, keep same-key messages in order, let several consumer groups read independently, at-least-once delivery with exactly-once as an option.
• Non-functional: high throughput, low latency, no loss of acknowledged writes when a broker dies, horizontal scaling.

Back-of-envelope, with stated assumptions. Say 100,000 messages per second at 1 KB each: about 100 MB/s in, roughly 8.6 TB per day, about 60 TB for seven days of retention, and around 180 TB on disk with a replication factor of 3. Reads scale with consumer groups, so three groups read about 300 MB/s. These numbers justify the partition count, broker count and storage tiering you propose later.

2. The core idea: a partitioned, append-only log

A topic is split into partitions. Each partition is an ordered, immutable sequence of messages, and each message gets a sequential offset. Producers only append; consumers read forward from an offset.

Why a log? Kafka's design documentation argues that sequential disk I/O is far faster than random I/O and that appends and offset reads are constant-time, unlike seek-heavy B-tree operations. It also leans on the operating system's page cache, and because consumers mostly read near the tail, hot data is usually served from memory.

• Segments. Store each partition as segment files and roll to a new one at a size limit (Kafka's default log.segment.bytes is 1 GiB). Retention deletes whole old segments, which is cheap.
• Index. A small per-segment index maps offsets to file positions, so a fetch is one lookup plus a sequential read.
• Retention. By age or size; Kafka's broker default is log.retention.hours=168, or seven days.
• Compaction. For changelog topics, keep at least the latest value per key instead of deleting by age; a null value acts as a tombstone.

3. High-level architecture

Producers --produce(topic, key, value)--> Partition leader (broker A)
                                               |  followers fetch
                                               v
                                    Follower replicas (brokers B, C)

Consumer group --fetch(partition, offset)--> Partition leader
       +--commit(group, partition, offset)--> Group coordinator

Controller quorum: live brokers, partition leaders, ISR, topic config

• Brokers store partitions and serve reads and writes. Each partition has one leader; the other replicas follow it.
• The controller quorum tracks which brokers are alive, which replica leads each partition and which replicas are in sync. Kafka 4.0 was the first major release to run entirely without ZooKeeper; metadata lives in a Raft-based quorum called KRaft.
• The group coordinator assigns partitions to consumers and stores their committed offsets.

Write path:

1. The producer fetches metadata to find each partition's leader.
2. It picks a partition (hash of the key, or spread when there is no key), batches and compresses records.
3. It sends the batch straight to the leader. There is no routing tier.
4. The leader appends to its log; followers fetch from the leader, much like consumers.
5. Once every in-sync replica has the batch, the message is committed, the leader advances its high watermark and, with acks=all, acknowledges the producer.

Read path: a consumer sends fetch requests with an offset. If nothing new exists, the broker holds the request open (long polling) instead of making the consumer spin. Consumers only see committed messages, and Kafka uses zero-copy sendfile from page cache to socket, except when TLS is on.

4. Partitioning, ordering and hot keys

Partitioning is usually the first deep dive, because it decides ordering, parallelism and load balance at once.

• Ordering is per partition only. Same-key messages hash to the same partition, so per-key order holds, but there is no global order. If the interviewer insists on total order, the honest answer is one partition, which caps throughput at what one leader can handle.
• Partition count caps parallelism. Within a group, each partition goes to exactly one consumer, so 12 partitions keep at most 12 consumers busy.
• Adding partitions later remaps keys. With hash-mod-N placement, changing N moves keys and breaks per-key ordering during the transition, so over-provision keyed topics, within reason: every partition adds files, replication traffic and metadata.

Hot keys. One very active user or viral ad can overload a partition. Options: a compound key (ad ID plus region), a random suffix on the hottest keys with a merge downstream (losing per-key order for them), or a dedicated topic for the heaviest keys. Say which you would pick and what it costs.

5. Replication, the ISR and the acks dial

Each partition has a replication factor, typically 3 (the value Kafka's own docs use in their example). The leader takes writes and followers pull from it. The in-sync replica set (ISR) is the replicas that are alive and caught up; in Kafka, a follower that lags longer than replica.lag.time.max.ms (default 30 seconds) is dropped from it. A message is committed only when every ISR member has it.

Current Kafka producers default to acks=all. The catch: if the ISR shrinks to just the leader, acks=all effectively becomes acks=1. That is what min.insync.replicas fixes. The documented pattern is replication factor 3, min.insync.replicas=2 and acks=all: with fewer than two replicas in sync, writes are rejected, a deliberate choice of durability over availability. Both min.insync.replicas and default.replication.factor default to 1, so set them explicitly.

Why an ISR instead of a majority quorum? Majority voting needs 2f+1 replicas to tolerate f failures. Kafka's ISR tolerates f failures with f+1 replicas, because writes wait for all in-sync replicas and a slow one is dropped instead of blocking writes.

If every in-sync replica is gone, wait for one to return (consistent, temporarily unavailable) or elect an out-of-sync replica (available, but committed messages can be lost). Kafka's unclean.leader.election.enable defaults to false, favoring consistency.

6. Consumers: pull, offsets, groups and rebalancing

Pull, not push. Kafka's docs give the reasoning: push has to guess each consumer's speed and can overwhelm a slow one, while pull lets a consumer fall behind, catch up at its own pace and fetch in large batches. That makes pull your first back-pressure mechanism.

Offsets. A consumer's position is one integer per partition, so it can rewind and reprocess after a bug fix. Commit after processing; committing first turns a crash into lost messages.

Consumer groups. Within a group each partition has one consumer, and extra consumers sit idle. Separate groups read the whole topic independently, which is how one stream can feed a notification system and an analytics pipeline without either slowing the other.

Rebalancing. When a consumer joins, leaves or stops heartbeating, its partitions move. A consumer that exceeds max.poll.interval.ms (default 5 minutes) between polls is treated as failed, so keep heavy work off the poll loop. Static membership (group.instance.id) avoids rebalances on quick restarts, and Kafka 4.0 made a new consumer group protocol (KIP-848) generally available to cut rebalance downtime.

Queue semantics on a log. For many workers sharing one partition, mention share groups (KIP-932, early access in Kafka 4.0, production-ready since 4.2): records are locked for a time, acknowledged individually, and delivery attempts are counted so a message that keeps failing stops being redelivered.

7. Delivery semantics: at-least-once vs exactly-once

• Idempotent producer (since Kafka 0.11). The broker discards retried duplicates using a producer ID and per-partition sequence numbers. Current clients enable it by default when no settings conflict; it requires acks=all, retries above zero and at most five in-flight requests per connection.
• Transactions (also 0.11). A producer writes to several partitions atomically and commits consumer offsets in the same transaction, closing the read-process-write loop. Consumers must set isolation.level=read_committed; the default also returns records from aborted transactions.

The point interviewers listen for: exactly-once covers data that stays inside Kafka. Once a consumer charges a card or writes to another database, you are back to at-least-once, and the fix is an idempotent consumer: store a message ID under a unique constraint, or make the operation naturally idempotent. The same reasoning comes up when you design a job scheduler or an ad click aggregator.

8. Back-pressure, retries and scaling

• Watch consumer lag, the gap between the log-end offset and the committed offset. Growing lag means adding consumers (up to the partition count), faster processing or more partitions.
• Producer back-pressure. When records arrive faster than they can be sent, the producer's buffer fills and send() blocks for up to max.block.ms, then fails. The application must slow down, shed load or spill to disk.
• Broker protection. Quotas cap bytes per second and request-handling time per user or client ID; clients over quota get delayed responses, so one noisy tenant cannot starve the rest.
• Retries and dead letters. Classic consumer groups have no built-in per-message retry, so a common pattern is retry topics with growing delays plus a dead-letter topic, so one bad message does not block its partition.
• Throughput. Batching (Kafka 4.0 raised the default linger.ms from 0 to 5 ms), end-to-end compression (gzip, Snappy, LZ4, zstd) and zero-copy reads.
• Storage. Add brokers and reassign partitions. For long retention, tiered storage moves completed segments to external storage such as S3 or HDFS while recent data stays local. It is off by default, needs a remote-storage plugin and does not support compacted topics.

9. Trade-offs to name, and how to pace the round

Strong answers name each dial and pick a setting for the stated requirements: ordering vs parallelism (more partitions, only per-key order), durability vs latency (acks=all adds a replication round trip), consistency vs availability (rejecting writes when the ISR is too small), retention vs cost (tiered storage), and exactly-once vs simplicity (idempotent consumers often reach the same result).

For a typical 45-minute round (length and format vary by company and level):

Prepare one sentence each for common follow-ups. Delayed messages: a topic per delay tier, or a scheduler that republishes when due. Priorities: a topic per priority, drained high-first. Replay from a point in time: look up the offset for a timestamp and seek to it.

The fastest way to find weak spots is to say the whole design aloud on a timer. A free AI mock interview in CoPilot Interview lets you include system design in the question mix, reads each question aloud and scores your spoken answer with what worked and what to improve.

FAQ

What is the difference between a message queue and a distributed log like Kafka?

A classic queue removes a message once a consumer acknowledges it, so each message is handled by one worker. A log like Kafka keeps messages until retention expires and each consumer group tracks its own offset, so many independent readers can consume and replay the same data.

How does Kafka guarantee message ordering?

Kafka only guarantees order within a single partition. Messages with the same key hash to the same partition, so per-key order holds, but there is no global order across the partitions of a topic.

What is the ISR in Kafka?

The in-sync replica set is the group of replicas that are alive and caught up with the partition leader. A message counts as committed once every ISR member has it, and with acks=all the producer is acknowledged only at that point.

Does Kafka support exactly-once delivery?

Kafka supports exactly-once processing for read-process-write pipelines that stay inside Kafka, using the idempotent producer, transactions and read_committed consumers. Side effects in external systems, such as a database write or an email, still need idempotent handling in the consumer.

How many partitions should a topic have?

Enough to cover your target throughput and the largest number of consumers you want in one group, since each partition is read by only one consumer per group. Leave headroom, because adding partitions to a keyed topic later changes which partition each key maps to.