Interview: Explain RAG End to End

What you will be able to do

• ✓Explain RAG end to end in two minutes
• ✓Describe the ingestion and query pipelines
• ✓Explain why RAG reduces hallucination and where it does not
• ✓Compare RAG with fine-tuning and long prompts
• ✓Answer with the definition-how-example-trade-off structure

Practise this: RAG in two minutes

Say this out loud until it feels natural. Then adapt it to a project you built.

• •1. The problem: an LLM only knows its training data. It does not know our documents or anything recent - and when it does not know, it often invents an answer.
• •2. The idea: before the model answers, we find the relevant parts of our own documents and put them in the prompt. The model answers from that text and cites it.
• •3. Ingestion, done ahead of time: load the documents, clean them, split them into chunks, turn each chunk into an embedding, and store it with its text and source in a vector database.
• •4. Query, for every question: embed the question, find the most similar chunks (often with keyword search too, and a re-ranker), build a prompt with those chunks, and let the model answer with citations - or say "I don't know" if nothing relevant was found.
• •5. The trade-off: answers are only as good as retrieval. So we measure retrieval and answers separately, keep documents fresh, and enforce permissions.

Basic: What is RAG?

Retrieval-Augmented Generation: find the parts of your own documents that are relevant to a question, add them to the prompt, and let the LLM answer from them - with sources.

• •Retrieve -> augment -> generate.
• •Gives the model private and current knowledge without retraining.

Basic: Why do we need RAG? What problem does it solve?

LLMs know only their training data up to a cut-off, never saw your private documents, and invent answers when they do not know. RAG gives them the right text at question time.

• •Private data, recent data, and hallucination.
• •In Lesson 1.4, llama3 invented 6 of 10 bakery answers without RAG and none with it.

Basic: What are the main steps of a RAG system?

Ingestion: load, clean, split into chunks, embed, store. Query: embed the question, retrieve the best chunks, build the prompt, generate the answer, return it with sources.

• •Ingestion runs when documents change; query runs for every question.

Basic: Why should a RAG answer cite its sources?

So users can check the answer, trust it, and find more detail - and so developers can debug which chunk caused a wrong answer.

• •Store the source (file, page, section) with every chunk.
• •Citations are also a quality signal: no source, no claim.

Basic: What should a RAG system do when the documents do not contain the answer?

Say it does not know - and decide that in code when retrieval finds nothing relevant, instead of hoping the model refuses.

• •In Lesson 1.3 the prompt said "say I don't know" and the model still invented gift cards.
• •A code guard fixed it.

Intermediate: RAG vs fine-tuning - when do you use which?

RAG changes what the model sees; fine-tuning changes its weights. Use RAG for knowledge - facts that change, must be cited, or are private. Use fine-tuning for behaviour - style, format, a narrow skill. Many systems use both.

• •RAG facts update by editing a document.
• •A saved Modelfile system prompt is not fine-tuning (Lesson 1.5).

Intermediate: Why not just put all the documents in a long context window?

It works for a few small documents, but cost and latency grow with every token, models use long contexts less reliably, and real collections are far too large. RAG sends only what matters.

• •Lesson 1.5: 255 tokens for all 7 documents vs 79 for the matching one.
• •Lesson 1.2: a prompt over the limit was cut silently.

Intermediate: What are good use cases for RAG, and where does it struggle?

Good: answering from documents - support bots, policy and HR search, product and code documentation, contracts. Struggles: counting and listing across everything, outdated or conflicting documents, wording the search cannot match, and reasoning errors even with the right text.

• •Route counts and totals to SQL or code.
• •Lesson 1.7 broke a RAG app on purpose in five ways.

Intermediate: Which parts of a RAG system can fail?

Ingestion (bad parsing, bad chunks, stale index), retrieval (wrong or missing chunks), prompt building (too much or badly ordered context), and generation (the model misreads or ignores the text).

• •Test retrieval and generation separately.
• •Fix the earliest failing step first.

Advanced: How does RAG reduce hallucination, and why does it not remove it?

Grounding the answer in retrieved text gives the model real facts to copy and cite. But the model can still misread the text, mix in its own knowledge, or answer from an irrelevant chunk when retrieval was wrong.

• •Lesson 1.7: the right refund rule was retrieved and the answer was still wrong (2 days vs 24 hours).
• •Add faithfulness checks and citations.

Advanced: What is agentic RAG, and when is it better than a fixed pipeline?

Retrieval becomes a tool the agent decides to use: it can rewrite the query, search several times or in several sources, and stop when it has enough. Better for complex, multi-part questions; worse for latency, cost and predictability.

• •Start with a fixed pipeline; add agentic steps for questions that need them.

Professional: How would you explain RAG's value and limits to a non-technical client?

"The assistant reads your documents before answering and shows where each answer comes from. It is as good as your documents and our search: if a policy is outdated or missing, the answer will be too. We measure quality on your real questions before launch and keep improving it."

• •Promise measured quality, not perfection.
• •Make document ownership part of the plan.

Professional: A stakeholder asks for "a ChatGPT that knows everything in our company". How do you respond?

Clarify the real goal and users, pick one high-value use case with clean documents, define success metrics and permissions, build a small RAG pilot, measure on real questions, then expand. "Everything" is a direction, not a first release.

• •Scope, data quality and permissions decide success more than the model.

Interview questions