Most LLM interview questions check whether you understand what happens between a prompt and a reply: how text becomes tokens, how attention mixes information across them, why context windows have limits, how sampling picks the next token, and why models hallucinate. Interviewers want a correct one-minute answer first, then the trade-off or failure mode behind it. This page is a concept bank of 28 questions with answers at that depth.
Use it for ML engineer, applied scientist and generative AI roles. Each answer is a starting point to say out loud, not a script. For the round structure of ML loops, see our machine learning interview help page; for how concept questions fit into full loops, see our OpenAI interview process guide and our data science interview prep by round.
LLM interview questions on tokenization and embeddings
1. What is tokenization, and why do LLMs use subword tokens?
A tokenizer splits text into units from a fixed vocabulary. Byte-pair encoding (BPE) starts from characters or bytes and repeatedly merges the most frequent adjacent pair into a new token until it reaches the target vocabulary size. Subwords keep the vocabulary manageable while still spelling any rare word from pieces; byte-level BPE never hits an unknown token. WordPiece and SentencePiece are related schemes.
2. Why do LLMs struggle to count letters or do digit arithmetic?
The model sees token IDs, not characters. A word may be one token or three, and long numbers are chunked into multi-digit tokens that don't line up with place value. Character-level tasks therefore require the model to have learned spellings indirectly.
3. What is an embedding?
A learned dense vector. Inside the model, each token ID maps to an input embedding. For retrieval, a separate embedding model maps a whole passage to one vector so that similar meanings land close together, which makes semantic search possible.
4. Cosine similarity, dot product or Euclidean distance?
Cosine similarity ignores vector length and compares direction only. If vectors are normalized to unit length, all three give the same ranking, since squared Euclidean distance equals 2 minus 2 times the cosine. Use whatever the embedding model was trained for.
Attention and transformer architecture questions
5. Explain self-attention.
Each token is projected into a query, a key and a value. A token's query is compared with every key; the scores go through a softmax and weight a sum of the values, so each output mixes information from the tokens most relevant to it. In the notation of Attention Is All You Need: softmax(QKᵀ / √d_k) V. Some ML loops ask you to write algorithms from scratch in NumPy, as our Atlassian ML coding interview guide describes, so code attention by hand at least once.
6. Why divide by the square root of d_k?
If query and key components have mean 0 and variance 1, their dot product has variance d_k. Large dot products push the softmax toward a near one-hot output where gradients are tiny. Scaling by 1/√d_k keeps the variance near 1, as the original paper explains.
7. What is multi-head attention, and what are MQA and GQA?
Multi-head attention runs several attention operations in parallel with separate projections, so different heads can track different relationships, then concatenates and projects the results. Multi-query attention shares one key and value head across all query heads; grouped-query attention shares each key-value head across a group. Both shrink the KV cache with little quality loss.
8. What is causal masking?
Decoder-only models set attention scores to future positions to negative infinity before the softmax, so token t only sees tokens up to t. That lets training predict every next token in a sequence in one parallel pass while matching how generation works.
9. Why do transformers need positional information?
Attention by itself treats its input as a set, so shuffling tokens shuffles outputs the same way. The original transformer added sinusoidal positional encodings; others use learned positions, RoPE (which rotates queries and keys by position-dependent angles so their dot product depends on relative distance) or ALiBi (a distance-based penalty on attention scores).
10. Encoder-only, decoder-only or encoder–decoder?
Encoder-only models such as BERT use bidirectional attention and masked-token training, which suits classification and embeddings. Decoder-only models generate left to right and dominate chat and general generation. Encoder–decoder models such as T5 read the full input bidirectionally, then generate, which suits translation and summarization.
Context window and inference questions
11. What limits the context window?
Standard attention compares every token with every other, so compute grows quadratically with sequence length. FlashAttention-style kernels avoid storing the full attention matrix, but the compute is still quadratic. The KV cache grows linearly per token, and positional schemes trained at one length need adjustment, such as RoPE scaling, to work at longer ones.
12. Does a long context mean the model uses all of it well?
Not necessarily. Lost in the Middle (Liu et al., 2023) found performance is often highest when relevant information sits at the beginning or end of the context and drops when it is in the middle. Put the most important material first or last, and keep retrieval precise.
13. What is the KV cache?
During generation, the keys and values of earlier tokens don't change, so the server stores them instead of recomputing them for each new token. Its size per token is 2 × layers × KV heads × head dimension × bytes per value, and on long conversations it often limits how many requests fit on a GPU.
14. What are prefill and decode?
Prefill processes the whole prompt in parallel; it is compute-heavy and determines time to first token. Decode generates one token per step, reading the weights and the KV cache each time; it is limited by memory bandwidth and determines the time per output token.
Sampling and decoding questions
| Method | How it picks the next token | Typical use |
|---|---|---|
| Greedy | Always the most likely token | Short, deterministic outputs; prone to repetition on long text |
| Beam search | Keeps the B most likely partial sequences | Translation and other tasks with one best answer |
| Temperature | Divides logits by T before the softmax | T below 1 for focused answers, above 1 for variety |
| Top-k | Samples from the k most likely tokens | Simple cap on unlikely tokens |
| Top-p (nucleus) | Samples from the smallest set with cumulative probability of at least p | Open-ended generation; the set adapts to the model's confidence |
15. Why not always use the most likely token?
Maximizing likelihood on open-ended text tends to produce bland, repetitive output. Holtzman et al. proposed nucleus sampling to keep diversity while cutting off the unreliable low-probability tail.
16. What does temperature do mathematically?
It divides the logits by T before the softmax. T below 1 sharpens the distribution toward the top token; T above 1 flattens it. As T approaches 0, sampling approaches greedy decoding.
17. Top-k or top-p?
Top-k uses a fixed number of candidates whether the model is sure or unsure. Top-p uses as many as needed to reach probability mass p, so it narrows when the model is confident and widens when it isn't. Many systems combine either one with temperature.
18. What is speculative decoding?
A small draft model proposes several tokens and the large model verifies them in one forward pass, accepting the matching prefix. With the standard acceptance rule the output distribution is the same as the large model's alone, while latency drops when drafts are often accepted.
RAG, hallucination and evaluation questions
19. What is retrieval-augmented generation, and when does it beat fine-tuning?
RAG retrieves relevant passages at query time and puts them in the prompt, so answers can use fresh, private or citable knowledge. It wins when facts change, need sources or depend on user permissions. Fine-tuning wins for behavior, format and style, and the two are often combined.
20. Why do LLMs hallucinate?
They are trained to produce likely continuations, not verified facts. Gaps or outdated knowledge in training data, prompts that pressure the model to answer anyway and sampling randomness all contribute. Mitigations: ground answers in retrieved sources, allow "I don't know", require citations, lower temperature for factual tasks and verify claims before display.
21. What is perplexity, and what are its limits?
Perplexity is the exponential of the average negative log-likelihood per token on held-out text; lower means the model predicts the text better. It is only comparable between models with the same tokenizer, and it says little about helpfulness or factual accuracy.
22. How do you evaluate an LLM?
Match the method to the task: accuracy or exact match for closed answers; unit tests for code; BLEU or ROUGE for n-gram overlap with a reference, which is weak for open-ended text; rubric-based LLM-as-a-judge checked against human labels; and human review for high-stakes output. Report results by slice, not just one average.
23. What is pass@k?
The probability that at least one of k sampled solutions passes the unit tests, as used with the HumanEval benchmark (Chen et al., 2021). It is estimated by drawing n ≥ k samples per problem, counting c correct ones and averaging 1 − C(n−c, k) / C(n, k) over problems, which is unbiased.
24. What is benchmark contamination?
When test questions appear in training data, scores overstate real ability. Guard against it with held-out or private test sets, newer questions and overlap checks between test items and training data.
Training, fine-tuning and quantization questions
25. What are pretraining, supervised fine-tuning and preference tuning?
Pretraining learns next-token prediction on a very large corpus. Supervised fine-tuning teaches the model to follow instructions from demonstration examples. Preference tuning, through RLHF (a reward model plus reinforcement learning) or DPO (learning directly from preference pairs), shapes which answers the model prefers.
26. How does LoRA work?
LoRA freezes the pretrained weight matrix W and learns a low-rank update ΔW = BA, where the rank r is much smaller than W's dimensions. Only A and B train, so trainable parameters and optimizer memory drop sharply, and the update can be merged into W for no added inference latency.
27. What is quantization?
Storing weights, and sometimes activations or the KV cache, in fewer bits. A 7-billion-parameter model needs about 14 GB for weights at 16 bits and about 3.5 GB at 4 bits, before overhead. Decode is memory-bound, so smaller weights also speed it up. Post-training methods such as GPTQ and AWQ need no retraining; accuracy loss grows as bits drop, so measure on your own eval set.
28. What is a mixture-of-experts model?
Each token is routed to a few expert feed-forward networks out of many. Total parameters are large but active parameters per token are small, so compute per token is lower than a dense model of the same total size, though all experts must still fit in memory. Training adds a load-balancing objective so the router doesn't overuse a few experts.
Say these answers out loud before the interview
Concept questions are easy to recognize and hard to explain crisply on the spot. The free AI mock interview includes a data science and ML track in the browser, and the desktop Mock Interview scores each spoken answer out of 10 with what to fix and a model answer.
Try a free AI mock interviewFAQ
What are the most common LLM interview questions?
Expect questions on tokenization, embeddings, self-attention and its scaling factor, positional encoding, context-window limits, the KV cache, sampling methods such as temperature and top-p, RAG, hallucination, evaluation, fine-tuning methods such as LoRA, and quantization.
How deep do LLM interview answers need to be?
Start with a correct one-minute answer, then add the trade-off or failure mode behind it. Research roles may ask you to derive or implement attention; applied roles care more about when each technique helps and how you would measure it.
Do I need to know the transformer math for an LLM interview?
Know scaled dot-product attention, why it is scaled by the square root of the key dimension, causal masking, and how the KV cache size is computed. Many interviews also ask you to implement attention or a sampling function in Python.
What is the difference between top-k and top-p sampling?
Top-k always samples from a fixed number of the most likely tokens. Top-p samples from the smallest set of tokens whose probabilities add up to at least p, so the candidate set shrinks when the model is confident and grows when it is not.
How do I explain hallucination in an interview?
Say that models are trained to produce likely text rather than verified facts, name the causes (knowledge gaps, pressure to answer, sampling), then give mitigations: retrieval grounding, citations, allowing the model to abstain and checking claims before display.
How should I practice LLM interview questions?
Answer a few questions a day out loud in under a minute each, then check what you missed. Implement attention and nucleus sampling once from scratch, and finish with a mock interview that mixes concept questions with a design problem.