Interview: Troubleshoot Poor Retrieval

What you will be able to do

• ✓Diagnose a wrong RAG answer step by step
• ✓Tell retrieval problems from generation problems
• ✓Choose the right fix for each kind of failure
• ✓Use a test set to prove a fix worked

Basic: The bot gives a wrong answer. What is your first step?

Reproduce it and look at what was retrieved for that question. If the right chunk was not retrieved, it is a retrieval problem; if it was, it is a generation problem.

• •Log or trace retrieved chunks for every answer.

Intermediate: The right document exists but is never retrieved. What could be wrong?

The question uses different words (keyword search) or a different meaning focus (semantic search), the chunk is too big or mixes topics, the text was badly parsed, or a filter excludes it.

• •Lesson 2.7: the shop name in every question pulled results to the wrong document.
• •Lesson 1.7: "return" did not match "refund".

Intermediate: How can a bad document parser hurt retrieval?

Broken tables, headers and footers repeated on every page, menus, hyphenation and HTML leftovers become part of the chunks and pull the vectors away from the real content.

• •Lesson 2.2: HTML leftovers lowered similarity from 0.847 to 0.758; cleaning restored it.

Intermediate: How do chunk size and chunking strategy cause problems?

Too big: one chunk mixes topics and its vector matches nothing well. Too small: the answer is split across chunks and loses context. Splitting in the middle of a sentence or table breaks meaning.

• •Split on structure, add overlap, keep headings with chunks.
• •Test sizes on real questions.

Intermediate: Retrieval was right but the answer is wrong. What now?

Look at the prompt: too many chunks, the key chunk buried in the middle, unclear instructions, or a model that misreads numbers and rules. Fix the prompt, the order and number of chunks, or the model.

• •Lesson 1.7: the right refund rule, the wrong conclusion.
• •Lesson 1.3: a clearer prompt fixed the delivery answer.

Advanced: The answers use outdated information. Why?

The index is stale (documents changed but were not re-ingested), or old and new versions are both indexed and the model picked the old one.

• •Lesson 1.6: a stale index kept the old rule.
• •Lesson 1.7: the 2023 policy won over the current one.

Advanced: What is query rewriting, and when does it help?

Turning the user's question into a better search query - expanding abbreviations, removing noise, splitting multi-part questions, or generating several versions (multi-query). It helps when users write short, vague or chatty questions.

• •Even simple cleaning helps: removing the shop name gave 11/12 in Lesson 2.7.

Advanced: When would you add hybrid search or re-ranking?

Hybrid search when exact terms matter - product codes, names, numbers - that embeddings miss. Re-ranking when the right chunk is in the top 20-50 but not at the top: a cross-encoder reads question and chunk together and reorders them.

• •"What costs Rs 45?" was found by keyword search but missed by embeddings (Lesson 2.7).

Advanced: How do you prove that a fix worked?

Run the same test set before and after - retrieval metrics (hit rate, recall@k, MRR) and answer metrics (correctness, faithfulness) - and check that no other questions got worse.

• •One fixed example is not proof.
• •Add the original failing question to the test set.

Advanced: The bot answers questions it should refuse. What do you do?

Check why something was retrieved at all - semantic search always returns a nearest chunk. Add a relevance check: a re-ranker score threshold, keyword evidence, or an LLM relevance judgement - and an "I don't know" path in code.

• •Raw similarity thresholds were fragile in Lesson 2.7 (a gap of 0.014).

Professional: Walk me through how you would run a quality investigation for a RAG bot in production.

Collect failing cases (negative feedback, escalations, low scores), sample them, label the failing step for each (parsing, chunking, retrieval, generation, missing document), count by cause, fix the biggest cause, measure on the test set, release, and repeat.

• •Group by cause - one fix often solves many failures.
• •Missing documents are a content problem, not a code problem.

Professional: A client says "the AI is wrong 30% of the time". How do you respond?

Ask for examples and measure it properly on a labelled set. Break the errors down by cause, show which are retrieval, generation or missing content, agree on a target and a plan, and report progress with the same metric.

• •Turn a feeling into a number, then into a plan.

Interview questions