Design an Online Judge (LeetCode-Style) System

An online judge takes a user's code, compiles and runs it against hidden test cases inside an isolated sandbox, and returns a verdict such as Accepted, Wrong Answer or Time Limit Exceeded. The interview-ready design is an API that records each submission and puts a job on a queue, language-specific workers that run the code in locked-down containers or microVMs with hard CPU, memory, process and network limits, and a results endpoint the client polls. The depth interviewers probe is sandbox security, test data delivery, and surviving a contest where tens of thousands of people submit at once.

Clarify the requirements first

• Functional: browse problems, view a statement with a code stub per language, run code against sample or custom input, submit for a verdict with runtime and memory, and join timed contests with a live leaderboard.
• Security first: untrusted code must never reach the host, the network or another user's data.
• Fast feedback: results in a few seconds for a typical submission.
• Burst tolerance: contests start at a fixed time, so load arrives as a step, not a curve.
• Availability over strict consistency for problem lists and leaderboards, but a submission must never be lost.

Published walkthroughs pick different numbers; Hello Interview's breakdown of this question, for example, targets results within 5 seconds and a contest with 100,000 users. Ask rather than assume: how many languages, whether “Run code” is in scope, whether contests give partial credit per test, and whether plagiarism detection is in scope (usually not; mention it as a later batch job).

Quick capacity math

Suppose 10,000 submissions land in the same minute, each with about 100 test cases at a pessimistic 100 ms per test:

10,000 submissions × 100 tests × 0.1 s = 100,000 CPU-seconds
Finish within 60 s  →  100,000 / 60 ≈ 1,667 cores busy for that minute

Real numbers are lower, since many tests finish in tens of milliseconds and a wrong answer can stop at its first failing test, but no single machine covers that order of magnitude. A queue and a horizontally scaled worker fleet are not optional.

Core entities and API

POST /problems/{id}/submissions  {language, code}         → 202 {submissionId}
GET  /submissions/{id}                                     → {status, verdict, runtimeMs, memoryKb}
POST /problems/{id}/run          {language, code, stdin}  → 202 {runId}
GET  /contests/{id}/leaderboard?page=1

Take the user ID from the session token, never the request body, and stamp the submission time on the server. In a contest, a client-supplied timestamp is an invitation to game the ranking.

The submission pipeline, end to end

1. Accept. Authenticate, apply a per-user rate limit (see Design a Rate Limiter), reject oversized code, and check the language and contest window.
2. Record and enqueue. Write a submission row with status QUEUED and the server timestamp, publish {submissionId, language, problemId, testSetVersion} to that language's queue, and return 202 with the ID.
3. Claim. A worker in the matching pool pulls the message, and no other worker can claim it until it is acknowledged or its lease expires. A crashed worker means a retry, not a lost submission.
4. Compile and run. Compile inside a sandbox with its own limits, then run each test case in a fresh sandbox and compare output as you go.
5. Report. Write the verdict, runtime, memory and first failing test, emit a “judged” event for the leaderboard, and acknowledge the message.
6. Deliver. The client polls GET /submissions/{id} about once a second until the status is final, or gets a push over WebSocket or server-sent events.

Cover two failure cases before you are asked. If the row is written but the enqueue fails, the submission sits in QUEUED forever; use a transactional outbox or a sweeper that re-enqueues stale rows. If a worker dies mid-judging, the job runs again, so judging must be idempotent, with the leaderboard update keyed on the submission ID. The lease, retry and dead-letter mechanics match Design a Distributed Job Scheduler. And keep judge failures separate from user failures: a missing test file or sandbox setup error is an internal error to retry, never a Runtime Error shown to the user.

Sandboxed execution: the deep dive

You are running arbitrary code from strangers, and some of it will try to read your files, open network connections, fork until the machine falls over, or loop forever. Four common isolation levels:

A defensible answer: per-language container pools for speed, each run inside a fresh, tightly limited sandbox, with gVisor or Firecracker as the upgrade when the threat model demands more than a shared kernel. Name full VMs (too slow to start) and serverless functions (cold starts, execution caps) as rejected alternatives, with the reason.

Limits to enforce on every run

• CPU time plus a separate wall-clock limit. CPU time catches infinite loops; wall-clock catches programs that sleep or block. isolate exposes these as --time and --wall-time.
• Memory through cgroups, so it covers every process the program spawns. Crossing the cap kills the run, reported as Memory Limit Exceeded.
• Process count, to stop fork bombs. isolate allows a single process by default; Docker has --pids-limit.
• Output and file size, so nobody fills the disk or streams gigabytes to stdout.
• No network: a private network namespace with only loopback, or --network none.
• Least privilege: a seccomp syscall filter, read-only root filesystem, non-root user, dropped capabilities, and a scratch directory wiped after every run.

Keep a warm pool of containers per language so no submission waits on an image pull or runtime boot, but create a fresh sandbox (or reset the scratch filesystem) for every run so one user's state never leaks into the next.

Test case storage and verdicts

• Version test sets and keep them immutable in object storage, under a path like problems/123/v4/. Each submission records the version that judged it, so rejudges are reproducible.
• Cache them on workers by problem and version, and pre-load a contest's test sets before it starts.
• Write tests once, not per language. LeetCode-style problems call a function, so store one language-neutral serialization of each input (a binary tree as a level-order array, say) and give each language a small harness that deserializes it, calls the user's function and serializes the result.
• Pick a comparison mode per problem: exact, whitespace-tolerant, floating-point tolerance, or a custom checker when several outputs are valid. Run checkers with limits too.
• Never ship hidden tests to the client. Whether to reveal the first failing case is a product decision.

For fair timing, run judges on one consistent machine type, do not run more concurrent jobs than physical cores, and set limits with headroom over the reference solution. Whether slower languages get a time multiplier varies by platform, so state your choice.

Contests at peak load

A contest is a scheduled stampede: everyone opens the same problems in the same second, submits in waves, and rushes again at the end. Because the start time is known, prepare instead of react:

• Pre-scale on a schedule. Queue-depth autoscaling reacts in minutes; the spike arrives in seconds. Warm the containers and test caches before the start.
• Prioritize. Contest submissions get their own queue or higher priority, practice submissions come next, and “Run code” requests go last.
• Absorb, do not drop. The queue turns a spike into a short wait. Show a “Judging” status so people do not resubmit out of impatience.
• Rate limit and de-duplicate in-flight submissions per user, and reject identical code resubmitted within seconds.
• Rank by the server's receive time, not judging time, so a backed-up queue never costs a contestant penalty.
• Protect the read path: serve statements from a cache or CDN, unlocked at the start time, and cache the top leaderboard page for a second or two.

The same everyone-at-once pattern drives Design Ticketmaster, where a ticket drop plays the role of the contest start.

Results delivery and live leaderboards

For individual results, short polling is simple and usually enough; Hello Interview's breakdown says LeetCode itself polls for submission results. WebSockets or server-sent events cut wasted requests but add connection state, so present them as an upgrade.

For the leaderboard, the submissions table stays the source of truth and a Redis sorted set per contest serves reads. A common rule ranks by problems solved, then by lower penalty time (time to each accepted solution, sometimes plus a charge per rejected attempt, depending on the rules), so pack both into one score:

score = solved × 10,000,000 − penalty_seconds

on first Accepted for (user, problem):
  ZADD contest:{id}:board {score} {user_id}

top 100:  ZRANGE contest:{id}:board 0 99 REV WITHSCORES
my rank:  ZREVRANK contest:{id}:board {user_id}

Award a problem only on the first accepted submission, rebuild the set from the database if it is lost, and if test data proves wrong mid-contest, rejudge affected submissions at low priority and recompute. Design a Leaderboard covers the ranking structure in more depth.

Trade-offs to say out loud

Pacing for 45 minutes: about 5 minutes on requirements, 5 on entities, API and capacity math, 10 on the end-to-end pipeline, 15 on deep dives (sandbox first, then contests or the leaderboard), and the rest on failure modes and trade-offs.

Hello Interview's breakdown spells out level expectations: mid-level candidates should land the API, data model and the need for isolation, while senior candidates move quickly through the high-level design, compare containers, VMs and serverless functions, and explain how test cases actually run against user code. The sandbox comparison is easy to know and harder to say crisply against a clock, so rehearse it aloud; a free AI mock interview in CoPilot Interview can include system design questions and scores each spoken answer.

FAQ

What is an online judge system?

An online judge is a service that accepts code submissions, compiles and runs them against hidden test cases inside an isolated sandbox, and returns a verdict such as Accepted, Wrong Answer or Time Limit Exceeded. LeetCode-style practice sites and programming contests are built around one.

Should an online judge run code in containers or virtual machines?

Hardened containers with strict CPU, memory, process and network limits are the usual interview answer because they start fast and are cheap to pool per language. If the threat model needs a stronger boundary than a shared kernel, add a user-space kernel such as gVisor or use microVMs such as Firecracker.

How does an online judge stop infinite loops and fork bombs?

Each run gets a CPU-time limit and a separate wall-clock limit, so busy loops and sleeping programs are both killed. A process limit, such as allowing a single process or setting a Docker pids limit, stops fork bombs before they exhaust the machine.

Why put a queue between the API and the code execution workers?

A queue turns a contest spike into a short wait instead of errors, lets each language pool scale independently, and keeps a submission safe if a worker crashes, because the message is redelivered when its lease expires. The client then polls for the result or receives a push once judging finishes.

How do you build a live leaderboard for a coding contest?

Keep submissions in the database as the source of truth and maintain a Redis sorted set per contest, with a score that packs problems solved and penalty time into one number. Update it idempotently on each first accepted submission and serve the top pages from a short-lived cache.