ML System Design Interview: A Framework, 3 Worked Prompts and a Rubric

An ML system design interview asks you to design an end-to-end machine learning system, such as ranking a feed, flagging fraudulent payments or ordering search results, in a single interview slot. The reliable way through it is a fixed framework: problem, then metrics, data, features, model, training, serving and monitoring. Interviewers mostly grade whether you turn a vague product goal into a well-posed prediction task, choose metrics that match the goal, deal honestly with labels and data, start from a simple baseline, and explain how the model is served and watched in production. Naming the newest architecture earns far less than getting those steps right.

This guide gives you the framework with a time budget, three worked prompts at outline depth, the rubric interviewers tend to use, and the mistakes that sink otherwise strong candidates. For the classic recommendation walkthrough, see design a recommendation system; this page is the general method.

How ML system design differs from regular system design

A classic system design round is about moving and storing data reliably at scale. An ML round keeps the infrastructure questions but adds a second layer: what the model should predict, how you know it works, and how it degrades.

Expect infrastructure follow-ups either way, so keep your classic system design skills sharp: caching, queues, storage choices and latency budgets all come up when you discuss serving.

The 8-step framework

Use the same eight steps for every prompt. The time budget below assumes a 45-minute round; scale it to your slot and spend extra time wherever the interviewer pushes.

A few habits make each step land:

• Problem: restate the goal as a prediction. “Show posts the user will value” becomes “predict the probability of several engagement events for each candidate post, then combine them into one score.” Asking whether a heuristic would do is a strength, not a dodge; the first rule in Google’s Rules of Machine Learning is “Don’t be afraid to launch a product without machine learning.”
• Metrics: pair an offline metric you can compute on a held-out set with an online metric you would A/B test, and name at least one guardrail that must not get worse.
• Data: say where every label comes from and what is wrong with it. Interviewers listen hardest here.
• Model: start simple (logistic regression or gradient-boosted trees), then justify anything more complex by what it fixes.
• Serving: the same Google guide warns about training-serving skew and recommends saving the features used at serving time so you can train on them. Mention a feature store or logged features and you have covered it.

Worked prompt 1: Design feed ranking

Prompt: “Design the ranking system for a social media home feed.”

• Problem. Rank posts from followed accounts (and possibly recommended ones) for each feed load. Clarify scale, whether recommended content is in scope, and whether the goal is short-term engagement or long-term retention.
• Metrics. Offline: log loss or AUC for each predicted event, plus a ranking metric such as NDCG on logged sessions. Online: sessions per user, time spent, retention. Guardrails: hides, reports, unfollows and integrity violations.
• Data. Impression logs joined with clicks, likes, comments, shares, dwell time and hides. Watch for position bias (top slots get more clicks regardless of quality), so log the position and correct for it, for example by training with position as a feature and fixing it at serving time.
• Features. User (interests, activity level), author (follower relationship, past interaction rate), post (type, age, text and image embeddings) and context (device, time of day).
• Model. Two or three stages. Candidate generation pulls a few thousand posts from the follow graph plus embedding retrieval. A ranker, typically a multi-task neural network, predicts several events per candidate, and a weighted sum of those predictions gives the score. A final re-ranking pass enforces diversity, freshness and integrity rules.
• Training. Retrain frequently on recent logs, use time-based validation splits, and calibrate each prediction head so the weighted sum is meaningful.
• Serving. Fit retrieval, feature lookup and ranking inside the feed’s latency budget; precompute embeddings and cache user features.
• Monitoring. Score distributions, feature freshness, and online metrics per user segment. Watch for feedback loops, where the model only learns from what it already chose to show.

The news feed infrastructure side (fan-out, storage, caching) is covered in design a news feed.

Worked prompt 2: Design fraud detection for payments

Prompt: “Design a system that detects fraudulent card transactions.”

• Problem. Score each transaction at authorization time and choose an action: approve, decline, or send to manual review. Clarify transaction volume, latency limits in the payment path, and the size of the review team.
• Metrics. Fraud is rare, so accuracy is meaningless. Use precision and recall at your chosen threshold and precision-recall AUC offline. Translate to business terms online: fraud losses prevented against the cost of falsely declining good customers, and review-queue load.
• Data. Labels come mostly from chargebacks and disputes, which can arrive weeks after the transaction, plus manual review outcomes. That delay shapes everything: recent data is mislabeled as legitimate until the chargebacks arrive. There is also selection bias: transactions you decline never get a true label, so keep a small, controlled way to measure what the model blocks.
• Features. Velocity counts over sliding windows (transactions per card, device or IP in the last minute, hour and day), amount compared with the user’s history, merchant risk, device and location changes, and shared-entity links such as one device used by many cards.
• Model. Rules for known patterns plus gradient-boosted trees as the main model; they handle mixed tabular features well and are easy to explain to risk teams. Graph or sequence models are a later upgrade for organized fraud rings.
• Training. Split by time, never randomly, to avoid leakage; wait out the label delay before evaluating a period; downsample legitimate transactions and reweight. Retrain often, because fraudsters adapt.
• Serving. Real-time scoring in the authorization path with streaming feature computation, a strict timeout, and a rules-only fallback if the model is unavailable.
• Monitoring. Score drift, decline rate by segment, review-queue volume and new attack patterns; recalibrate thresholds when the mix changes.

Worked prompt 3: Design search ranking

Prompt: “Design the ranking for product search on an e-commerce site.”

• Problem. Given a query, return the most relevant products, ordered to maximize purchases without hurting relevance. Clarify catalog size, query mix (head versus long-tail), and whether results are personalized.
• Metrics. Offline: recall at k for retrieval, NDCG at k and mean reciprocal rank for ranking, measured on human relevance judgments and logged sessions. Online: click-through and conversion rate, zero-result rate and query reformulation rate. Guardrail: relevance as judged by raters.
• Data. Click and purchase logs, which carry position bias, plus a smaller set of human-graded query-product pairs as a cleaner signal. Purchases and add-to-cart are stronger labels than clicks.
• Features. Query-product text match (BM25 scores on title and description), embedding similarity, product quality and popularity, price, availability, query category intent, and user history for personalization.
• Model. Two stages. Retrieval combines an inverted index with embedding-based nearest-neighbor search to get a few hundred candidates. Ranking uses learning to rank; a gradient-boosted model trained with a pairwise or listwise objective, such as LambdaMART, is a strong, well-understood choice. Business rules and diversity apply last.
• Training. Correct for position bias, mine hard negatives (products shown but skipped), and evaluate new rankers on judged queries before any A/B test.
• Serving. Index sharding and replication for retrieval, cached results for popular queries, and a ranking stage that fits the remaining latency budget.
• Monitoring. Zero-result and reformulation rates by query type, index freshness for new products, and metric drops after catalog changes.

The typeahead box that sits in front of search is its own classic problem: design search autocomplete.

The rubric interviewers use

Rubrics vary by company and level, but most ML design scorecards cover the same dimensions. Use this table to grade your own practice runs.

Senior candidates are also expected to drive the conversation, raise trade-offs before being asked and talk about how the system evolves after launch. The paper Hidden Technical Debt in Machine Learning Systems (Sculley et al., NIPS 2015) is a good primer for that part: it lists ML-specific risks such as entanglement, hidden feedback loops and undeclared consumers.

Common mistakes and how to practice

• Starting with the model. Spend the first five minutes on the problem and metrics, even if you know the architecture you want.
• Metrics that do not match the goal. Offline accuracy on an imbalanced fraud set, or click-through rate alone for a feed, invites hard follow-ups.
• Ignoring the label problem. Delayed, biased or missing labels are the most common deep-dive topic.
• Random train-test splits on time-ordered data. This leaks the future into training.
• No baseline. Without one, you cannot show the complex model is worth it.
• Skipping serving and monitoring. Running out of time before step 7 is common; watch the clock.
• Buzzword stacking. Every component you name invites a question about why.

A practice plan that works in about two weeks:

1. Pick six prompts: the three above plus, for example, ads click prediction, notification ranking and content moderation.
2. For each, write a one-page outline using the eight steps, then present it out loud in 45 minutes with a timer.
3. Grade yourself with the rubric table and redo the weakest step the next day.
4. Practice with an interviewer. A free AI mock interview in CoPilot Interview builds questions from your résumé and the job description; choose a mix that includes system design, add an optional focus such as ML system design, and each spoken answer is scored out of 10 with what to fix and a model answer. The browser version also has a data science and ML track.

Keep the system design cheat sheet nearby for the infrastructure follow-ups, and see machine learning interview help for the other rounds in an ML engineer loop.

FAQ

How is an ML system design interview different from a regular system design interview?

A regular round focuses on how data and requests flow at scale. An ML round adds what the model predicts, which metrics prove it works, where the labels come from, and how the model is trained, served and monitored, while still expecting sound infrastructure choices.

What framework should I use for ML system design?

Work through eight steps in order: problem, metrics, data, features, model, training and evaluation, serving, and monitoring. Budget your time so you reach serving and monitoring, which many candidates run out of time for.

Do I need to know deep learning architectures in detail?

You should know when a neural model beats a simpler one and be able to describe common patterns such as two-tower retrieval and multi-task ranking. Interviewers usually value a justified baseline and clear trade-offs over architecture detail.

Which metrics should I mention?

Name an offline metric that fits the task, such as precision-recall AUC for rare-event detection or NDCG for ranking, an online metric you would A/B test, and at least one guardrail that must not get worse.

What is training-serving skew?

It is a difference between the data or feature logic used in training and what the model sees in production. Common fixes are logging the features used at serving time and training on them, sharing feature code between pipelines, and monitoring the gap.

How long does it take to prepare for an ML system design interview?

If you already know ML basics, about two weeks of practice on six or so prompts, each presented out loud in a timed session and graded with a rubric, is a reasonable plan. Add more time if serving and infrastructure are new to you.