AI Engineer Interview Questions: 40 Questions With Answer Outlines

The most common AI engineer interview questions test whether you can build reliable products on top of large language models: when to use retrieval instead of fine-tuning, how to evaluate output that has no single right answer, how to cut latency and cost, and how to defend against prompt injection. Compared with an ML research loop, expect fewer derivations and more practical design, debugging and trade-off questions.

Below is the typical loop, then 40 questions in seven groups, each with a short outline of what a strong answer covers. The title "AI engineer" varies; this guide targets applied LLM roles. For classic model-training loops, see our machine learning interview help page.

What the AI engineer interview loop looks like

Companies differ and recruiters will tell you the exact rounds, so ask. A typical applied AI loop includes some or all of these:

For data-heavy roles, our data science interview prep by round covers SQL and statistics screens. AI labs add their own emphasis; our guides to the OpenAI interview process and Anthropic coding interviews cover those loops.

AI engineer interview questions: LLM basics and RAG

LLM basics

1. How does an LLM generate text? Autoregressive next-token prediction: tokenize the input, run a decoder-only transformer, sample a token from the output distribution, append it, repeat. Explain why the same prompt can give different answers.
2. Why do tokens matter for cost and limits? Prices and context limits are counted in tokens, not words. OpenAI's help center gives a rule of thumb of about four characters per English token; code and many other languages use more tokens per word.
3. When would you change temperature or top-p? Lower for extraction and classification, higher for brainstorming. Note that even at temperature 0, hosted APIs don't always guarantee identical outputs.
4. What happens when input exceeds the context window? The request fails or gets truncated. Options: sliding window, summarization, retrieval. Longer prompts also cost more and add latency.
5. What are embeddings and how do you compare them? Dense vectors where similar meaning lands nearby; compare with cosine similarity, and embed queries and documents with the same model.
6. Prompting, RAG or fine-tuning: how do you choose? Start with prompting. Add retrieval for knowledge that changes or needs citations. Fine-tune for consistent format, tone or behavior, or to make a smaller, cheaper model good enough.
7. How do you get reliable JSON out of a model? Use the provider's structured-output or function-calling mode with a schema, validate every response, and retry with the validation error on failure.

Retrieval-augmented generation (RAG)

8. Walk me through a RAG pipeline. Parse, chunk, embed, index with metadata, retrieve, rerank, build a prompt with citations, generate, evaluate. The idea comes from Lewis et al. (2020), which paired a generator with a dense vector index. In a design round, wrap it in the usual arc from our system design cheat sheet.
9. How do you choose chunk size? Small chunks retrieve precisely but lose context; large ones dilute relevance. Split on document structure, add modest overlap, and pick the size that wins on your eval set.
10. Retrieval returns the wrong passages. How do you debug it? Measure retrieval alone with recall@k on labeled queries, then read failures: vocabulary mismatch, bad chunking, missing metadata filters or a stale index. Fixes include hybrid search, a reranker and query rewriting.
11. Dense, keyword or hybrid search? Dense captures meaning; BM25 catches exact terms, product codes and rare words. Hybrid with reciprocal rank fusion is a strong default.
12. How do you reduce hallucinations in RAG answers? Better retrieval first, then instructions to answer only from the context, citations, an explicit "I don't know" path, and a groundedness check before showing the answer.
13. How do you keep the index fresh and permission-safe? Incremental re-embedding on change, deletes that actually remove vectors, and access-control filters applied at retrieval time per user. Never rely on the prompt to keep restricted text from being shown.
14. Does a long context window replace RAG? For small, static corpora it can. Large or changing corpora, per-user permissions, citations and per-query cost still favor retrieval.

Evaluation and fine-tuning questions

Evaluation

15. How would you evaluate a feature with no single right answer? Agree on success criteria with product, build a golden set from real inputs, score with task-specific checks plus a rubric, validate any automated judge against human labels, then confirm online with user and business metrics.
16. What are the pitfalls of LLM-as-a-judge? Zheng et al. (2023) document position, verbosity and self-enhancement bias. Mitigate with swapped-order pairwise comparisons, explicit rubrics and spot checks against humans.
17. How do you evaluate retrieval and generation separately? Retrieval: recall@k, MRR or nDCG on labeled queries. Generation: faithfulness to the retrieved context, answer relevance and correctness.
18. How do you catch regressions when you change a prompt or model? Version prompts like code, run the eval suite in CI, compare results by slice, and block a release on meaningful drops.
19. Offline or online evaluation? Both. Offline suites gate releases; online A/B tests, shadow traffic and user feedback catch what the suite misses.
20. How would you evaluate a code-generation feature? Run generated code against unit tests in a sandbox and report pass@k, plus latency and how often users accept or edit the suggestion.

Fine-tuning

21. When is fine-tuning worth it? Consistent format or style, domain jargon, or distilling a large model into a smaller one for cost and latency. It is a poor way to add fast-changing facts.
22. What is LoRA and why is it popular? LoRA freezes the pretrained weights and trains small low-rank matrices injected into each layer, so you train far fewer parameters and can store many cheap adapters. QLoRA applies the same idea on a 4-bit quantized base model.
23. How do you build the training data? Curated examples that look like production inputs, deduplicated, with a held-out eval split and no leakage between them. Quality beats quantity.
24. SFT, RLHF or DPO? Supervised fine-tuning imitates demonstrations. RLHF fits a reward model on human preferences, then optimizes the policy with reinforcement learning. DPO learns directly from preference pairs with a classification-style loss, without a separate reward model.
25. How do you prove fine-tuning helped? Beat a well-prompted baseline on the same held-out set, and check for regressions on general tasks the model used to handle.

Serving, latency and cost questions

26. The feature feels slow. Where do you look first? Split time to first token from total time. Stream output, trim the prompt, cap output length, cache, route easy steps to a smaller model and run tool calls in parallel.
27. How would you cut model cost in half? Fewer input tokens, shorter outputs, routing to a cheaper model with escalation, response and prompt caching, batch APIs for offline work, and distillation for high-volume tasks.
28. API or self-hosted open-weights model? APIs give strong models with no GPU operations. Self-hosting gives control and data residency, and can be cheaper at high, steady utilization, at the cost of running GPUs.
29. What are the KV cache and continuous batching? The KV cache stores attention keys and values so each new token doesn't recompute the history. Continuous batching adds and removes requests at every decode step to keep GPUs busy.
30. How do you handle provider rate limits and outages? Exponential backoff with jitter, idempotent requests, a queue for non-urgent work, and a fallback model or provider.
31. What can go wrong with a semantic cache? A similar but different question gets a wrong cached answer. Scope the cache by user and permissions, set a strict similarity threshold and expire entries.

Safety and product sense questions

Safety and reliability

32. What is prompt injection and how do you defend against it? Untrusted content (a web page, an email, a document) carries instructions the model follows. The OWASP Top 10 for LLM Applications lists it first. Defenses: least-privilege tools, confirmation before side effects, output validation and no secrets in prompts. Say plainly that there is no complete fix.
33. How do you keep personal data safe? Redact before sending, check the provider's retention terms, apply access controls at retrieval, and limit what logs store.
34. How do you make a tool-using agent reliable? Few, well-typed tools, step and time limits, idempotent actions, human approval for irreversible ones, full traces, and evals on whole trajectories, not just final answers.
35. How do you tune guardrails? Moderate inputs and outputs, and measure both missed harmful content and false refusals; over-blocking is a product failure too.
36. What do you monitor in production? Latency, cost per request, error and refusal rates, user feedback, shifts in input topics, and sampled human review of traces under privacy controls.

Product sense

37. Should this feature use an LLM at all? Compare with rules, search or a classic model on cost per request, error tolerance and maintenance.
38. How do you design UX around an uncertain model? Show sources, make output editable, make undo easy and offer a human handoff for high-stakes cases.
39. Which metric would you launch on? Task success or resolution rate, or time saved, with guardrail metrics for complaints, cost per task and latency.
40. A stakeholder wants 100% accuracy. What do you say? Reframe around an acceptable error rate by risk level, add human review where errors are costly, and show eval data.
41. How do you decide what to improve next? Error analysis: label failures from evals and logs into categories and fix the biggest bucket first.

How to practice for an AI engineering loop

• Week 1: build one small RAG app on documents you know, with 50 labeled questions and a retrieval metric. It gives you a real project to discuss and real numbers.
• Week 2: change one thing at a time (chunk size, hybrid search, a reranker) and record the eval results. That turns into a strong deep-dive story.
• Every day: answer five questions from this list out loud in under a minute each. Speaking exposes gaps that reading hides.
• Before the onsite: run a full mock with design, fundamentals and behavioral questions, and prepare STAR stories with our free STAR story builder and these STAR examples.

FAQ

What questions are asked in an AI engineer interview?

Expect LLM fundamentals (tokens, embeddings, context windows), RAG design and debugging, evaluation, fine-tuning trade-offs, latency and cost, safety topics such as prompt injection, and product sense, plus practical coding and behavioral rounds. The exact mix varies by company, so ask your recruiter.

How is an AI engineer interview different from a machine learning engineer interview?

AI engineer loops for applied LLM roles focus on building products on foundation models: retrieval, prompting, evals, cost and safety. ML engineer loops more often include training models, feature engineering, classic ML theory and ML system design for ranking or prediction.

Do AI engineer interviews include LeetCode-style coding?

Some do, but many applied AI loops favor practical Python: calling an API, parsing structured output, handling errors or building a small pipeline. Prepare for both unless the recruiter tells you otherwise.

What is the most common AI engineer system design question?

Designing a RAG assistant, such as a support bot or a document question-answering tool, is a frequent choice. Cover ingestion and chunking, retrieval and reranking, prompting with citations, evaluation, latency, cost, permissions and prompt-injection defenses.

How should I prepare for an AI engineer interview in two weeks?

Build one small RAG app with a labeled eval set, iterate on it while recording results, answer five questions a day out loud, and finish with a full mock interview covering design, fundamentals and behavioral questions.

Do I need to know how to train models from scratch?

For most applied AI engineer roles, no. You should understand how models work at a conceptual level and know fine-tuning methods such as LoRA, but interviews focus more on building, evaluating and operating LLM features than on pretraining.