RAG interview questions: retrieval-augmented generation explained

Twelve common RAG interview questions

1. Walk me through a RAG pipeline end to end.
2. How do you choose chunk size and overlap?
3. Dense, sparse or hybrid retrieval — which and why?
4. What does a reranker add?
5. How do you evaluate retrieval separately from generation?
6. What is faithfulness or groundedness, and how do you measure it?
7. The answer is wrong — how do you debug it?
8. How do you handle document updates and deletions?
9. How do you use metadata filters?
10. How do you prevent prompt injection from retrieved documents?
11. When is RAG the wrong approach?
12. RAG versus fine-tuning versus long context?

The pipeline in one answer

Ingest documents, clean them and split them into chunks; embed each chunk and store vectors with metadata in an index. At query time, optionally rewrite the query, retrieve candidates with vector search (often combined with keyword search), rerank the top candidates with a cross-encoder, and assemble the best chunks into a prompt with instructions to answer only from them and cite sources. Then generate, and optionally verify the answer against the retrieved text.

Design choices interviewers probe

Structure-aware chunking — by heading, section or function — usually beats fixed-size splitting, and storing a parent section ID lets you retrieve small chunks but feed their larger parent to the model.

Evaluation: separate the two halves

• Retrieval: build a labelled set of questions with known relevant chunks; measure recall@k and mean reciprocal rank.
• Generation: measure faithfulness (is every claim supported by the retrieved text?) and answer relevance (does it answer the question?).
• End to end: correctness against reference answers, plus human review of a sample.
• Online: user feedback, citation clicks, escalation or follow-up rates.

Debugging a wrong answer

First check whether the right chunk was retrieved. If not, it's a retrieval problem: chunking split the answer, the query wording differs from the document's, an exact identifier needs keyword search, or a filter excluded it. If the right chunk was retrieved but the answer is still wrong, it's a generation problem: too many distracting chunks, the relevant one buried in the middle of a long prompt, or instructions that don't force grounding. Fixes differ completely, which is why interviewers love this question.

Security and freshness

Retrieved documents are untrusted input; a document can contain text that tries to override your instructions. Keep instructions separate from retrieved content, tell the model to treat it as data, restrict what tools the model can call, and enforce access control at retrieval time so users only retrieve documents they're allowed to see. For freshness, re-index changed documents, delete removed ones from the index, and re-embed everything when you change embedding models. See vector database interview questions.

When RAG isn't the answer

If the knowledge is small and stable, putting it directly in the prompt may be simpler. If the problem is style or behaviour rather than knowledge, fine-tuning fits better. If the task needs aggregation across thousands of records — “what's our average order value?” — query a database instead of retrieving text. Long-context models reduce but don't remove the need for retrieval, because cost, latency and accuracy all degrade with very long prompts.

Practising

The fastest way to find gaps is to be asked. Run a spoken mock round on RAG topics — a free AI interview assistant such as CoPilot Interview builds one from your résumé and the job description and scores every answer — then study whatever you stumbled on. Build a tiny RAG system over a few documents and break it deliberately — that experience makes every answer more convincing.

Disclosure: CoPilot Interview is our product, so weigh what we say about it accordingly. Everything else on this page applies whichever tool you use, or none.

Frequently asked questions

What is RAG in simple terms?

Retrieval-augmented generation retrieves relevant passages from your documents and gives them to a language model with the question, so the answer is grounded in those passages and can cite them.

How do you evaluate a RAG system?

Evaluate retrieval with recall@k and mean reciprocal rank on labelled questions, evaluate generation for faithfulness and relevance, check end-to-end correctness, and monitor user feedback online.

What is hybrid search in RAG?

Combining dense vector search, which matches meaning, with sparse keyword search such as BM25, which matches exact terms, and fusing the results.

Why does RAG return wrong answers?

Either retrieval failed — the right chunk wasn't found because of chunking, wording or filters — or generation failed despite good retrieval, often due to noisy context or weak grounding instructions.