Design ChatGPT: A System Design Interview Walkthrough

To design ChatGPT in a system design interview, you are really designing two systems: an ordinary chat product (accounts, conversations, history, a streamed reply) on top of a GPU inference tier that generates text one token at a time. Most of the difficulty is in the second half: how tokens reach the browser as they are generated, how you batch requests onto scarce GPUs, why the KV cache limits concurrency, and why rate limits are counted in tokens.

This is a generic design built from published serving research and common practice, not a description of OpenAI's internal systems. Every number below is an assumption; say so out loud. For more prompts and practice formats, see our system design interview help page.

1. Design ChatGPT: requirements and scale estimates

Functional requirements

• Send a message in a conversation and see the reply appear token by token.
• List, reopen, rename and delete past conversations.
• Stop generating, regenerate a reply, or edit an earlier message (which creates a branch).
• Several models, with free and paid tiers getting different models and limits.
• Out of scope unless asked: file uploads, images, web search and tool calls. Name them, then park them.

Non-functional requirements

• Time to first token (TTFT) under about a second for a typical prompt.
• Inter-token latency faster than people read, for example 20 or more tokens per second per stream.
• Durable history: losing an occasional reply is tolerable, losing a conversation is not.
• Graceful degradation when GPUs run short: queue, fall back to a smaller model or limit the free tier.

Estimates. Use round assumptions and show the arithmetic. OpenAI's help center gives a rule of thumb of roughly four characters of English text per token.

The takeaways matter more than the numbers: GPU throughput per dollar is the main cost lever, so batching is the core of the design, and hundreds of thousands of long-lived connections belong on their own cheap tier, not on the GPU servers.

2. API design and token streaming (SSE vs WebSockets)

Sending a message is a normal POST whose response is a stream:

• POST /v1/conversations/{id}/messages with { content, model, parent_message_id } → 200 text/event-stream
• POST /v1/messages/{message_id}/cancel to stop generating
• GET /v1/messages/{message_id}/stream?after_seq=N to resume a dropped stream
• GET /v1/conversations?cursor=… and GET /v1/conversations/{id}/messages for paginated history

The stream carries message_start with the new message ID, many delta events with a few characters each, then message_end with the finish reason and token usage, or an error. Give every delta a sequence number so a reconnecting client can ask for everything after the last one it saw.

SSE is the default answer for text, and it is what OpenAI's own API uses: its streaming guide describes HTTP streaming over server-sent events. Two details show depth. The browser's built-in EventSource only sends GET requests, so web clients usually read the POST response body with fetch() and parse the SSE format themselves. And every proxy in the path must flush instead of buffering, with idle timeouts longer than the longest generation; a comment line such as : keep-alive every 15 seconds or so keeps quiet connections open.

3. High-level architecture and request flow

Walk one message through the system:

1. API gateway: TLS, authentication, request-size limits and a first rate-limit check.
2. Chat service: saves the user's message and creates an empty assistant message with status streaming, so the conversation stays consistent if the reply fails.
3. Context builder: loads the active branch (usually from cache), adds the system prompt, tokenizes, and trims history to fit the context window while reserving room for the reply.
4. Input safety check: a fast classifier runs in parallel with routing; a flagged request is cancelled before it reaches a GPU.
5. Inference router: picks a model pool and replica from the model, the user's tier, queue depth, free KV-cache memory and cache affinity, then enqueues with a priority.
6. Model server: runs prefill, then decodes token by token, publishing small groups of tokens to a per-message stream (a Redis Stream or pub/sub topic keyed by message ID).
7. Streaming gateway: holds the client's SSE connection, subscribes to that stream and forwards deltas. Output moderation reads the same stream.
8. Completion: the full reply and its token counts are saved, a usage event goes onto a queue for metering and billing, and async workers write a title and update search.

Why the stream buffer in the middle? Model servers are your most expensive machines and should not hold 700,000 idle connections. The buffer also makes reconnects cheap: a dropped client reconnects to any gateway, which replays from the last sequence number. Cancellation travels the other way as a control message checked between decode steps, so a stopped reply frees its GPU slot at once. History and queues are standard pieces; see our chat app walkthrough and message queue walkthrough.

4. Model serving: GPU scheduling, batching and the KV cache

This is the deep dive most interviewers want.

Prefill and decode behave differently

Prefill processes every prompt token in parallel; it is compute-bound and sets the time to first token. Decode produces one token per forward pass for each sequence, and each pass reads the model weights plus that sequence's cached keys and values; it is memory-bandwidth-bound and sets inter-token latency. A decode step for one sequence leaves most of the GPU's compute idle, so adding sequences to the same step is nearly free at first. Throughput rises with batch size until compute or memory runs out, while each user's token rate slowly drops. Choosing that operating point per model is the central tuning decision.

Continuous batching

Static batching waits for the whole batch to finish, so one long reply holds every slot. Continuous batching schedules per decode iteration: finished sequences leave and queued ones join at every step. It was published as iteration-level scheduling in the Orca paper (OSDI '22) and is now standard in serving engines. Chunked prefill splits long prompts into pieces interleaved with decode steps, so one huge prompt can't stall everyone's next token; some systems also run prefill and decode on separate GPU pools.

The KV cache sets your concurrency

To avoid recomputing attention over the history at every step, the server caches each token's keys and values for every layer: 2 × layers × KV heads × head dimension × bytes per value per token. For a hypothetical 8-billion-parameter model with 32 layers, 8 KV heads (grouped-query attention) of dimension 128 and 16-bit values, that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token. An 8,000-token conversation needs about 1 GiB. With 40 GiB left for the cache after the weights, a GPU holds about 327,000 tokens, roughly 40 such conversations at once. Memory, not compute, often caps the batch.

Reserving a contiguous buffer for each sequence's maximum length wastes much of that memory. PagedAttention, the technique behind vLLM, allocates the cache in small fixed-size blocks, like operating-system pages, which cuts fragmentation and lets sequences share blocks. When memory still runs out, the scheduler preempts a sequence by swapping its cache to CPU memory or dropping it to recompute later. Grouped-query attention and 8-bit KV-cache quantization shrink the per-token cost further.

Prefix caching, routing and autoscaling

Each turn resends the system prompt and the conversation, so without caching, total prefill work grows with the square of the number of turns. Prefix caching keeps KV blocks for recent prefixes and skips recomputing them, but only if the next turn lands on the replica holding them. So the router uses affinity: hash the conversation ID to a preferred replica and fall back to the least-loaded one when it is busy.

A replica is a group of GPUs running one copy of the model, split with tensor parallelism inside a node and pipeline parallelism across nodes for the largest models. Autoscale on queue depth and KV-cache utilization, not CPU. Loading a large model's weights can take minutes, so keep headroom and plan degradation: priority queues (paid and interactive traffic first, batch jobs last), a smaller fallback model for the free tier, and a clear "high demand" message instead of timeouts. Also name weight quantization to 8 or 4 bits, and speculative decoding, where a small draft model proposes several tokens that the large model verifies in one pass without changing the output distribution.

5. Conversation storage and context windows

• conversations: conversation_id, user_id, title, model, created_at, updated_at, deleted_at, indexed on (user_id, updated_at) for the sidebar.
• messages: partitioned by conversation_id, sorted by a time-ordered message_id, with parent_id, role, content, model, token_count, status.

A wide-column or key-value store such as Cassandra or DynamoDB fits, because both hot reads ("messages in one conversation", "recent conversations for one user") are single-partition queries. Redis caches active conversations so the context builder doesn't hit the database every turn; uploads go to object storage. Time-ordered IDs can come from a Snowflake-style generator, as in our unique ID generator walkthrough.

Branches. Regenerating or editing creates a sibling message with the same parent_id, so a conversation is a tree, not a list. The client tracks the active leaf, and the context builder walks parent pointers back to the root.

Fitting the context window. Prompt plus maximum reply length must fit the window, so reserve the output budget first, then fill the rest:

• Sliding window: system prompt plus the most recent turns. Simple; old details are lost.
• Summarization: a running summary replaces older turns, at the cost of an extra model call.
• Retrieval: embed older turns and pull back only relevant ones. Best recall, more moving parts.

Deletion must reach everything derived from a conversation: caches, search indexes, summaries, embeddings and, per your retention policy, backups.

6. Rate limits, safety filters and multi-region

Rate limit by tokens

Cost scales with tokens, so request counts alone are the wrong unit; OpenAI's API, for example, documents limits in requests and tokens per minute and per day. Combine per-tier message quotas with per-key token buckets in Redis. Output length is unknown at admission, so reserve tokens from max_tokens or an estimate and refund the difference when the reply ends. Keep fairness limits separate from capacity-based load shedding, which protects the GPUs regardless of quota. The bucket mechanics are in our rate limiter walkthrough.

Safety filters

Run an input classifier before generation. Output moderation is harder because the user reads as tokens arrive: hold back a small buffer before release (safer, slower), or check chunks as they stream and halt and replace the message on a flag (faster, but text may be retracted after the user saw it). Account-level abuse detection runs asynchronously on usage events, and any tool or web content is untrusted input that can carry prompt injection.

Multi-region

Run the stateless tiers in every region, close to users. GPUs are scarce and rarely spread evenly, so let the router send inference to a region with spare capacity; the extra round trip is small next to a 20-second generation. Data-residency rules may pin some users to a region, so region choice is a policy input. Give each user a home region for conversation data, with asynchronous replication for disaster recovery. If a model server dies mid-reply, restart on another replica with the original prompt plus the tokens already sent, so the reply continues where it stopped.

7. Trade-offs the interviewer probes

FAQ

How do you stream ChatGPT-style responses to the browser?

Use Server-Sent Events on a normal HTTP POST response: a start event, many small text deltas with sequence numbers, and an end event with token usage. Web clients usually read the stream with fetch() because the browser's EventSource only sends GET requests. Use WebSockets when you need two-way real-time traffic such as voice.

What is the KV cache and why does it matter in an LLM system design?

It stores the attention keys and values of earlier tokens so each new token doesn't recompute the whole history. In this guide's example it is 128 KiB per token, about 1 GiB for an 8,000-token conversation, which is why GPU memory often limits how many conversations a replica serves at once.

What is continuous batching?

The server re-forms the batch at every decode step: finished sequences leave and waiting ones join right away, instead of the batch waiting for its longest reply. It raises GPU utilization and cuts queueing delay. It was published as iteration-level scheduling in the Orca paper (OSDI 2022).

How should you rate-limit an LLM chat service?

Count tokens, not only requests. Combine per-tier message quotas with per-key token buckets, reserve an estimated output budget at admission and refund the unused part when the reply ends. Keep capacity-based load shedding separate from per-user fairness limits.

How do you handle conversations longer than the context window?

Reserve room for the reply, then fill the rest with the system prompt and the most recent turns. For long chats, add a running summary of older turns or retrieve relevant ones with embeddings.

Is this how OpenAI actually builds ChatGPT?

No. This is a generic interview design based on published serving research and common practice, not a description of OpenAI's internal systems, and every number in it is an assumption rather than an OpenAI figure.