Putting It Together: RAG

What you will be able to do

• ✓Explain what RAG is and the problem it solves
• ✓Separate the two phases: indexing and answering
• ✓Load a PDF, split it into chunks, and choose chunk_size and chunk_overlap
• ✓Build the RAG chain with LCEL: retriever, prompt, LLM, parser
• ✓Write a prompt that says "I don’t know" when the context lacks the answer
• ✓Find where a RAG system fails - usually retrieval

The idea, in plain English

A model knows what it was trained on, not your documents. Asked how much annual leave "our company" gives, llama3 said it has no access to your company’s policies. RAG - Retrieval-Augmented Generation - fixes that without changing the model: retrieve the relevant passages, add them to the prompt, and let the model generate an answer from them.

It has two phases. Indexing happens once: load the documents, split them into chunks, embed the chunks, store them (Lessons 2.9 and 2.10). Answering happens per question: the retriever finds the closest chunks (2.11), they are formatted into a prompt with the question, and the LLM answers (2.3 and 2.4).

The whole answering phase is one LCEL chain: {"context": retriever | format_documents, "question": passthrough} | prompt | llm | StrOutputParser(). Ask it a string, get a string back.

Generation in this lesson is real llama3 output. Retrieval needs an embedding model, which this machine does not have (Lesson 2.9) - so the runs use a keyword retriever, BM25, in Chroma’s place. Because a retriever is just question in, Documents out, the rest of the chain is unchanged. The one place keyword search behaves differently turned out to be the most instructive result of the lesson.

Worked example: Ask questions about a PDF.

Why RAG

Without RAG, the model has its training data and your question. Asked "How many days of annual leave do employees at our company receive?", llama3 said it does not have access to your company’s policies and offered general information instead. Honest - and useless. Another model, or another question, might produce a confident invented number.

RAG suits information that is private (policies, customer records), changes often (prices, documentation), is too large for one prompt (a hundred PDFs), or is specific to your domain (course material). The documents stay in your store; the model sees only the few chunks relevant to each question.

Two phases

Indexing: load, split, embed, store. It runs once, and again when documents change - the expensive part, kept out of the question path. Answering: retrieve, augment, generate. It runs per question and touches only the top k chunks.

Keep them as separate scripts or functions. An app that re-indexes every document on every question is a common first mistake - and with persist_directory (Lesson 2.10) there is no need.

Chunking

A whole document is too large to embed as one vector and too large to put in every prompt. RecursiveCharacterTextSplitter cuts text into chunks of at most chunk_size characters, preferring paragraph breaks, then line breaks, then spaces - so it rarely cuts mid-word. Our four one-sentence policies stayed four chunks at chunk_size=500.

chunk_overlap repeats the end of one chunk at the start of the next. On a three-sentence leave policy with chunk_size=100: without overlap, one chunk ended "Unused annual leave of up to 5 days can" and the next began "be carried over to the next year." - the rule split in half. With chunk_overlap=30, the third chunk carried "within 2 working days. Unused annual leave of up to 5 days can be carried over to the next year." whole.

Loading a PDF

PyPDFLoader (pip install langchain-community pypdf) returns one Document per page, with metadata including source, page (counting from 0) and page_label. split_documents() - not create_documents() - splits those Documents and copies the metadata onto every chunk, so each chunk still knows its page.

Our handbook: 2 pages, 5 chunks of 127 to 190 characters. Check what extraction produced before anything else - scanned PDFs, tables and two-column layouts often come out as garbled or empty text, and no later step can recover it. Note: importing from langchain-community printed a notice that the package is being sunset in favour of standalone integration packages; the loader still worked.

The RAG chain

The chain starts with a dict of runnables, run in parallel on the question: "context" is retriever | format_documents - Documents in, one string out - and "question" passes the question through unchanged (lambda x: x or RunnablePassthrough(), both gave identical answers). The dict feeds the prompt, then the model, then StrOutputParser.

format_documents decides what the model sees. Joining page_content with blank lines works; prefixing each chunk with its page - "[page 1] Sick leave..." - lets the answer cite its source.

"If it is not in the context, say you don’t know"

For "What is the company’s maternity leave policy?" the retriever still returned three chunks - annual leave, travel, laptops - because top-k always returns something. With the instruction, llama3 answered "I don’t know." Asked about bringing a dog to the office: "I don’t know."

We also removed the instruction, to see if llama3 would invent a policy. It did not: it said the context did not mention maternity leave, or pets. That is this model on these questions at temperature 0 - not a guarantee. Keep the instruction; it costs nothing and it is the behaviour you want specified, not hoped for.

Where RAG fails: usually retrieval

"How much vacation do I get?" The handbook says 20 days of annual leave. Our keyword retriever returned the laptop and remote-work chunks - "vacation" appears nowhere in the PDF - and llama3, correctly given what it saw, said "I don’t know." The answer was in the document; it never reached the prompt.

This is exactly the gap embeddings close: "vacation" and "annual leave" mean the same thing, and a semantic retriever would match them where keyword search cannot. It is also the general lesson. Most bad RAG answers are retrieval failures, and the model cannot answer from a chunk it was never shown - so when an answer is wrong, print what was retrieved first.

RAG is not training

Indexing a PDF changes the vector store, not the model. The model receives the chunks for one call and keeps nothing. Update a policy and you re-index the document; the next answer uses the new text.

Fine-tuning is different: it changes the model’s weights, needs a training run, and suits style or task behaviour more than facts that change. For questions over documents, RAG is the natural fit.

RAG on a learning platform

Index lesson content with course_id, lesson_id - and, for many academies on one platform, tenant_id - in the metadata. "Ask this course" is then a retriever with filter={"tenant_id": ..., "course_id": ...}; "ask this lesson" narrows the filter. Quiz generation and module summaries are the same chain with a different prompt.

Step-by-step code

Tip: When an answer is wrong, print retriever.invoke(question) before touching the prompt. If the right chunk is not there, the model never had a chance.

Watch out: The "say you don’t know" instruction reduces invented answers; it does not prevent them. Test questions your documents cannot answer, and keep testing when you change the model.

RAG at a glance

PyPDFLoader("file.pdf").load()

RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)

splitter.create_documents(texts)

splitter.split_documents(pages)

vector_store.add_documents(chunks)

{"context": retriever | fmt, "question": ...} | prompt | llm | parser

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

What usually goes wrong

✗ splitter.create_documents([p.page_content for p in pages])

✓ splitter.split_documents(pages)

Quick check before you move on

1. 1.

   "How much vacation do I get?" got "I don’t know" although the handbook lists 20 days of annual leave. Which step failed?

   Show answer
2. 2.

   For the maternity-leave question, the retriever returned three chunks. Why, if none was relevant?

   Show answer
3. 3.

   Why split_documents(pages) rather than create_documents(texts)?

   Show answer
4. 4.

   What does "question": lambda x: x do in the chain?

   Show answer

Interview questions