Interview: RAG and Memory

What you will be able to do

• ✓Explain the RAG pipeline end to end
• ✓Discuss chunking, embeddings and vector stores
• ✓Improve retrieval with hybrid search and re-ranking
• ✓Explain agentic RAG
• ✓Design memory for long conversations

The idea, in plain English

Almost every agent needs knowledge it was not trained on, so RAG questions appear in nearly every interview - often with a design follow-up: "how would you improve retrieval quality?".

Memory questions are related: both are about choosing what goes into the limited context window.

Use the RAG pipeline you built in Module 2 (embeddings, vector stores, retrievers) as your example.

Worked example: Fourteen questions on retrieval and memory, tied to the RAG pipeline of Module 2 (Lessons 2.9-2.15) and the memory lessons.

Basic: What is RAG?

Retrieval-Augmented Generation: before the model answers, search your own documents for relevant passages, put them in the prompt, and ask the model to answer from them - ideally citing the source.

• •Gives the model private, current knowledge without retraining.
• •Reduces hallucination, because the answer is grounded in real text.
• •Lesson 2.12 builds one.

Basic: What is an embedding, and why does RAG use it?

An embedding is a vector of numbers that represents the meaning of a text. Texts with similar meaning have vectors that are close, so a question can find passages that mean the same thing even with different words.

• •Compared with cosine similarity or dot product.
• •Embedding model and chat model are different models.

Basic: What is a vector store?

A database that stores vectors with their text and metadata and finds the nearest vectors to a query quickly.

• •Examples: Chroma, FAISS, pgvector, Qdrant, Pinecone.
• •Uses approximate nearest-neighbour indexes to stay fast at scale.
• •Lessons 2.10 and 2.15.

Intermediate: How do you choose a chunk size?

Small chunks match precisely but may lose context; large chunks keep context but dilute the match and use more tokens. Split along the document's structure (sections, paragraphs), add some overlap, and test sizes on real questions.

• •There is no universal best size - measure retrieval quality.
• •Keep the source and position with every chunk.

Intermediate: What is hybrid search?

Combining keyword search (like BM25) with semantic vector search. Keywords catch exact names, codes and numbers; embeddings catch paraphrases. The two result lists are merged, for example with reciprocal rank fusion.

• •Often clearly better than either alone.
• •Especially for product codes, error codes and names.

Intermediate: What is re-ranking?

A second step that takes the top candidates from fast retrieval and re-scores them with a slower, more accurate model (a cross-encoder), which reads the question and passage together.

• •Retrieve many (say 50), re-rank, keep the best few.
• •Improves precision at the cost of some latency.

Intermediate: How do you evaluate a RAG system?

Evaluate retrieval and generation separately. Retrieval: did the right chunks come back (hit rate, recall@k, MRR)? Generation: is the answer correct, faithful to the retrieved text, and does it cite sources?

• •Build a test set of questions with known answers and sources.
• •LLM-as-judge for faithfulness; exact checks where possible.

Intermediate: What is agentic RAG?

Retrieval becomes a tool the agent decides to use. The agent can rewrite the query, search several times, choose between sources, and stop when it has enough - instead of always retrieving once before answering.

• •Better for complex or multi-part questions.
• •More calls and more ways to fail - add limits.

Intermediate: How do you manage memory in a long conversation?

Keep the recent turns in full, summarise older turns, and store important facts in long-term memory to retrieve when relevant - so the context stays within budget.

• •A pure buffer grows until it overflows.
• •Summaries lose detail - keep key facts structured.
• •Lesson 2.8 compares memory types.

Advanced: Your RAG app gives wrong answers. How do you debug it?

First check retrieval: for failing questions, were the right chunks retrieved? If not, fix chunking, embeddings, query rewriting or add hybrid search. If retrieval was right, fix generation: the prompt, the model, or how chunks are presented.

• •Trace every step so you can see the retrieved chunks.
• •Fix the earliest step that is wrong.

Advanced: How do you handle questions that need information from many documents?

Split the question into sub-questions, retrieve for each, then combine; or use map-reduce summarisation; or route aggregate questions (counts, totals) to a database or code instead of RAG.

• •Top-k retrieval sees only a few chunks.
• •Counting and listing are better done with SQL.

Advanced: How do you keep a RAG index up to date and correct?

Re-ingest changed documents incrementally, delete removed ones, store versions or dates in metadata, and remove outdated documents - otherwise old and new facts both get retrieved.

• •Track content hashes to embed only changes.
• •Prefer the newest version when two conflict.

Advanced: What are the security risks of RAG?

Retrieved documents can contain instructions (indirect prompt injection), private data a user should not see, or poisoned content. Filter by user permissions at retrieval time, treat retrieved text as data not instructions, and validate outputs.

• •Access control must be enforced in retrieval, not just in the UI.

Advanced: Long-term memory for an assistant: what would you store, and how?

Store stable, useful facts (preferences, decisions, profile) as structured records or embedded notes with timestamps, decide when to write them (explicitly or with a summariser), retrieve only relevant ones per request, and let users see and delete them.

• •Avoid storing everything - noise lowers retrieval quality.
• •Privacy: consent, retention limits, deletion.

Interview questions