Interview: RAG System Design Questions

What you will be able to do

• ✓Answer the core retrieval-toolbox questions
• ✓Choose evaluation metrics for retrieval and answers
• ✓Run a full RAG design with a fixed framework
• ✓Discuss cost, latency, safety and risks

Intermediate: What chunking strategies do you know?

Fixed size with overlap, sentence or paragraph based, structure based (headings, sections, pages), and semantic chunking that splits where the topic changes. Keep metadata (source, section, page) with each chunk.

• •Structure-based splitting is a strong default for documents with headings.
• •Test on real questions.

Intermediate: Explain hybrid search and how results are combined.

Run keyword search (BM25) and vector search, then merge the lists - often with reciprocal rank fusion, which adds up scores based on each result's rank in each list.

• •Keywords catch codes and names; vectors catch paraphrases.

Intermediate: What is re-ranking?

A second stage after fast retrieval: a cross-encoder reads the question and each candidate together and scores relevance more precisely. Retrieve 20-50, re-rank, keep the best few.

• •Better precision for some extra latency; cannot find what stage one missed.

Intermediate: Which metrics do you use to evaluate RAG?

Retrieval: hit rate, recall@k, MRR or nDCG on questions with known relevant chunks. Answers: correctness, faithfulness to the context, answer relevance, citation accuracy - plus latency and cost.

• •LLM-as-judge for faithfulness, checked against human labels.

Intermediate: How many chunks should you put in the prompt?

Enough to contain the answer, few enough to keep cost, latency and noise low - often 3 to 8. Tune it on the test set; put the most relevant chunks first.

• •More chunks can make answers worse, not better.

Advanced: What guardrails would you add to a RAG system?

Input: permission checks, prompt-injection screening, rate limits. Output: answer only with citations, refuse when no relevant context, remove personal data, validate the format, and block unsafe content.

• •Guardrails run in code, not only in the prompt.

Advanced: Would you use a framework like LangChain or LlamaIndex?

For a prototype, frameworks speed things up. In production, use them where they help (loaders, integrations) but keep the core pipeline understandable and testable; some teams write the core in plain code.

• •Know what the framework does underneath.

Advanced: How do you serve a RAG system as an API?

A web service (for example FastAPI) with authentication, input validation, the query pipeline behind an endpoint, streaming responses, timeouts and concurrency limits, health checks, and tracing.

• •Ingestion runs separately - as background workers.

Professional: Mock design: a customer support bot for an online shop.

Clarify channels, languages, question types and actions. Ingest help articles, policies and product data with structure-based chunks and metadata. Query: clean query -> hybrid retrieval -> re-rank -> answer with citations; order questions go to tools with the user's identity, not RAG. Add "I don't know" and human hand-off. Evaluate on real tickets; track resolution rate and cost per conversation.

• •Separate document knowledge (RAG) from personal data (tools/APIs).

Professional: Mock design: question answering over 50,000 PDFs (contracts and reports).

Focus on parsing: layout-aware extraction, tables, scanned pages with OCR. Chunk by sections with page numbers, add metadata (client, date, type), hybrid search with filters, re-ranking, answers that cite page and section. Add evaluation with lawyers' or analysts' questions, and permissions per client.

• •Parsing quality decides everything for PDFs.

Professional: Mock design: a documentation assistant for developers.

Ingest docs, READMEs, API references and code comments with version metadata; chunk by heading and keep code blocks whole; hybrid search so exact function names match; answers with links to the exact doc section and version; evaluate on real support questions.

• •Versions matter - filter to the version the user runs.

Professional: How do you decide whether a use case needs RAG at all?

Ask whether the answers live in documents and need citations (RAG), in structured data (SQL or APIs), in a fixed process (a workflow), or in the model's general knowledge (plain prompting). Many products combine them.

• •Counts and totals belong to databases, not RAG (Lesson 1.7).

Professional: What are the top risks in your RAG design, and how do you reduce them?

Wrong answers (evaluation, citations, "I don't know"), data leaks (permission filters, isolation tests), prompt injection (layered defences), stale content (incremental ingestion, freshness alerts), cost overruns (budgets, routing), and outages (fallbacks).

• •Ending with risks shows senior judgement.

Interview questions