Lesson 1.3 · RAG Fundamentals

What is Retrieval-Augmented Generation (RAG)?

RAG = find the right documents, put them in the prompt, then let the model answer. Build a working RAG app in 40 lines for a bakery: without it llama3 rambled; with it, correct answers with sources. Then watch the two ways it fails - too strict, then inventing a gift card - and the small fix in code.

Beginner30 min

What you will be able to do

Explain the three steps of RAG: retrieve, augment, generate
Build a tiny RAG app with Python and a local model
Compare answers with and without RAG
Return the source of an answer
Handle "nothing found" safely in code
Understand why each step can fail on its own

The idea, in plain English

Lessons 1.1 and 1.2 showed the problem: a model writes what sounds right, knows nothing after its cut-off, and has never seen your documents. Retrieval-Augmented Generation (RAG) fixes this in three steps.

Retrieve: search your own documents for the ones that match the question. Augment: put those documents into the prompt. Generate: ask the model to answer using them - and to say which document it used.

We build the smallest possible RAG app for a bakery. The search is a simple word match; Module 2 replaces it with embeddings. llama3 on Ollama, temperature 0.

Worked example: Sri Lakshmi Bakery’s four policy documents and three customer questions - answered without RAG, then with RAG in three versions.

workflowRetrieve, augment, generatestep 1 / 3

question

delivery for Rs 1500?

your documents

4 policy files

1. retrieve

best match: delivery.md

2. augment

document + question in the prompt

3. generate

llama3 answers + source

React Flow

1 - Retrieve

The search compares the question with each document and picks delivery.md.

retrieved

['delivery.md']

"How much does delivery cost for an order of Rs 1500?" - the final version, real run.

Words you will see in this lesson

The words of RAG.

Small dictionary

RAGRetrieval-Augmented Generation: answer with the help of documents found for the question.

RetrieveFind the documents (or parts) that match the question.

AugmentAdd them to the prompt.

GenerateLet the model write the answer from them.

ContextThe retrieved text placed in the prompt.

GroundedAn answer based on the given documents, not on the model’s memory.

CitationThe name of the document an answer came from.

An everyday example: an open-book exam

In a closed-book exam, you answer from memory - and guess when you do not know. (A model without RAG.)
In an open-book exam, you first find the right page, then write the answer from it. (RAG.)
A good student also writes "see page 42". (A citation.)
If the book does not cover the question, a good student says so instead of inventing. (Handling "nothing found".)

Example 1 - a tiny RAG app

The whole app, in four parts:

DOCS: the bakery’s own documents. The model has never seen them.
words() and retrieve(): a simple search - the document that shares the most words with the question wins. No match, no result.
answer(): without RAG it sends only the question. With RAG it puts the found document, with its file name, into the prompt, and asks for an answer from it with the source.
Three questions: one about refunds, one that needs a calculation (delivery for Rs 1,500), and one the documents do not cover (gift cards).

tiny_rag.py

import re
import ollama

# 1. The knowledge: Sri Lakshmi Bakery's own documents (the model has never seen them)
DOCS = {
    "refunds.md":   "Refunds: custom cakes can be refunded only if cancelled at least 24 hours before pickup. "
                    "Ready-made items cannot be refunded once they leave the shop.",
    "delivery.md":  "Delivery: we deliver within 8 km of MG Road between 10 am and 8 pm. Delivery costs Rs 50; "
                    "orders above Rs 1000 are delivered free.",
    "hours.md":     "Opening hours: Monday to Saturday 8 am to 9 pm, Sunday 9 am to 2 pm.",
    "allergens.md": "Allergens: all cakes may contain nuts and eggs. Eggless cakes must be ordered two days ahead.",
}
STOP = {"a", "an", "the", "is", "are", "i", "my", "can", "do", "you", "for", "to", "of", "in", "on", "it", "what", "if", "get", "be"}


def words(text):
    return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}


# 2. Retrieve: find the document that shares the most words with the question
def retrieve(question, k=1):
    scored = sorted(DOCS.items(), key=lambda item: len(words(question) & words(item[1])), reverse=True)
    return [(name, text) for name, text in scored[:k] if words(question) & words(text)]


# 3. Augment + 4. Generate
def answer(question, use_rag):
    if not use_rag:
        prompt = question
    else:
        found = retrieve(question)
        context = "\n".join(f"[{name}] {text}" for name, text in found) or "(no documents found)"
        prompt = (f"Answer the question using ONLY the context below. Name the source file in brackets. "
                  f"If the context does not contain the answer, say: I don't know.\n\n"
                  f"Context:\n{context}\n\nQuestion: {question}")
    return ollama.generate(model="llama3", prompt=prompt, options={"temperature": 0}).response.strip()


for question in ["Can I get a refund for a custom cake I cancelled 2 days before pickup?",
                 "How much does delivery cost for an order of Rs 1500?",
                 "Do you sell gift cards?"]:
    print(f"=== {question}")
    print("  without RAG:", answer(question, use_rag=False)[:300].replace("\n", " "))
    print("  retrieved:  ", [name for name, _ in retrieve(question)])
    print("  with RAG:   ", answer(question, use_rag=True)[:300].replace("\n", " "))

Output - version 1

=== Can I get a refund for a custom cake I cancelled 2 days before pickup?
  without RAG: The sweet spot of cake conundrums!  Whether you can get a refund for a custom cake you cancelled 2 days before pickup depends on the bakery's policies and your agreement with them. Here are some possible scenarios:  1. **Bakery's cancellation policy**: Check your agreement or contract with the baker
  retrieved:   ['refunds.md']
  with RAG:    Yes. [refunds.md]
=== How much does delivery cost for an order of Rs 1500?
  without RAG: I'm happy to help! However, I need more information to provide an accurate answer. The delivery cost depends on various factors such as:  1. Location: Where is the delivery being made to? 2. Service provider: Which delivery service is being used (e.g., Swiggy, Zomato, etc.)? 3. Type of delivery: Is
  retrieved:   ['delivery.md']
  with RAG:    I don't know.
=== Do you sell gift cards?
  without RAG: I'm just an AI, I don't have the ability to sell gift cards or any physical products. I exist solely as a digital entity, and my purpose is to provide information and assist with tasks through text-based conversations. I don't have a physical presence or the ability to process transactions. If you'r
  retrieved:   []
  with RAG:    I don't know.

Reading version 1

Without RAG, every answer was generic: "depends on the bakery’s policies", "I need more information". The model does not know this bakery.
Refund: retrieval found refunds.md, and the answer was right and cited: "Yes. [refunds.md]".
Gift cards: nothing matched, and the model said "I don’t know". Correct.
Delivery: retrieval found the right document - but the model said "I don’t know". The document says orders above Rs 1,000 are free. The model did not apply the rule.
So RAG has two parts that can fail on their own: retrieval (find the right text) and generation (use it correctly). Here retrieval worked and generation failed.

Version 2 - a clearer prompt

We changed only the prompt: say who the assistant is, and ask it to apply the rules and numbers:

Delivery is now right: "Since the order value is above Rs 1000, delivery is free. [delivery.md]".
Refund is now a full sentence with the reason.
But gift cards broke: with "(no documents found)" in the context, llama3 answered "Yes, we do sell gift cards!" - and invented a source: [Sri Lakshmi Bakery - Gift Cards].
A less strict prompt answered more - including what it should not.

Version 2 - the new prompt

        prompt = (f"You answer customer questions for Sri Lakshmi Bakery. Use the context below. "
                  f"Read the numbers and rules carefully and apply them to the question. "
                  f"Give a short answer and name the source file in brackets. "
                  f"If the context does not cover the question at all, say: I don't know.\n\n"
                  f"Context:\n{context}\n\nQuestion: {question}")

Output - version 2

=== Can I get a refund for a custom cake I cancelled 2 days before pickup?
  retrieved:   ['refunds.md']
  with RAG:    Yes, you can get a refund for the custom cake since you cancelled at least 24 hours before pickup. [refunds.md]
=== How much does delivery cost for an order of Rs 1500?
  retrieved:   ['delivery.md']
  with RAG:    Since the order value is above Rs 1000, delivery is free. [delivery.md]
=== Do you sell gift cards?
  retrieved:   []
  with RAG:    Yes, we do sell gift cards! [Sri Lakshmi Bakery - Gift Cards]

Version 3 - "nothing found" is handled in code

If retrieval finds nothing, there is nothing to answer from. So do not ask the model at all:

Three new lines: if not found, return "I don’t know - our documents do not cover this."
All three answers are now right: refund, free delivery, and an honest "I don’t know" for gift cards.
The model never gets the chance to guess.
Rule: decide in code what code can decide. Use the model only for the part that needs language.

Version 3 - the guard

        found = retrieve(question)
        if not found:                                  # nothing relevant: do not let the model guess
            return "I don't know - our documents do not cover this."
        context = "\n".join(f"[{name}] {text}" for name, text in found) or "(no documents found)"

Output - version 3

=== Can I get a refund for a custom cake I cancelled 2 days before pickup?
  retrieved:   ['refunds.md']
  with RAG:    Yes, you can get a refund for the custom cake since you cancelled at least 24 hours before pickup. [refunds.md]
=== How much does delivery cost for an order of Rs 1500?
  retrieved:   ['delivery.md']
  with RAG:    Since the order value is above Rs 1000, delivery is free. [delivery.md]
=== Do you sell gift cards?
  retrieved:   []
  with RAG:    I don't know - our documents do not cover this.

Tip: Keep the "I don’t know" path in code even when your prompt says "say I don’t know". In version 2 the prompt said it - and the model still invented gift cards.

What this tiny app leaves out

Our app works for four short documents. The rest of the course turns it into a real system:

Word matching misses meaning: "Can I return my cake?" shares no words with "refund". Embeddings fix that (Module 2).
Real documents are long PDFs and web pages - they must be split into chunks (Module 5).
Thousands of documents need a vector database (Modules 3 and 4).
Quality must be measured, not guessed (Module 11), and the app must be safe (Module 12).

The recipe

1. Retrieve: find the documents that match the question.
2. If nothing is found, answer "I don’t know" in code.
3. Augment: put the documents, with their names, in the prompt.
4. Generate: ask for an answer from the documents, with the source.
5. Check retrieval and generation separately - each can fail on its own.

RAG at a glance

Retrieve

Find matching documents.

found = retrieve(question)

Nothing found

Decide in code.

if not found: return "I don't know..."

Augment

Documents with names.

context = "\n".join(f"[{name}] {text}" for name, text in found)

Generate

Answer from the context.

ollama.generate(model="llama3", prompt=..., options={"temperature": 0})

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Different words

“Ask "Can I return a cake I ordered?" Does the word search find refunds.md? Why not?”

Two documents

“Ask "When can I pick up an eggless cake on Sunday?" - it needs hours.md and allergens.md. Try retrieve(question, k=2).”

Your documents

“Replace DOCS with four short notes about your own work and ask questions about them.”

What usually goes wrong

Letting the model handle "nothing found"

Version 2’s prompt said "say I don’t know" - and it invented gift cards with a fake source.

Blaming the wrong step

In version 1, retrieval was right and generation was wrong. Check each step separately.

No sources

Without a citation, nobody can check the answer. Ask for the file name and show it.

Practice

Write these yourself before opening anything. Getting them wrong first is most of how this sticks.

Add a fifth document, payments.md ("We accept cash, UPI and cards; no cheques."), and two questions about it. Then add a check that prints a warning if the answer does not contain a [file name].

Show hint

A regular expression like re.search(r"\[[\w.]+\]", answer) finds a citation.

Key points

RAG = retrieve the right documents, augment the prompt with them, generate the answer from them.
Without RAG, llama3 gave generic or invented answers about a bakery it never saw.
Version 1 (strict prompt) refused a question it could answer; version 2 (clearer prompt) invented gift cards.
Handling "nothing found" in code made all three answers right.
Retrieval and generation fail independently - test both.

Quick check before you move on

What are the three steps of RAG?

Retrieve, augment, generate.

Why did version 1 say "I don’t know" about delivery?

The model did not apply the "above Rs 1000 is free" rule - generation failed, retrieval was right.

What fixed the invented gift cards?

Returning "I don’t know" in code when retrieval finds nothing.

Interview questions

What is RAG and why use it?

Retrieval-Augmented Generation retrieves relevant documents for a query and adds them to the prompt so the LLM answers from them - giving current, private and citable knowledge without retraining.

How do you handle queries with no relevant documents?

Detect it at retrieval time (no results or scores below a threshold) and return a fallback in code rather than asking the model, which may fabricate an answer.

Quiz

1.
Why does RAG reduce hallucination?
2.
Why will the word search fail for "Can I return my cake?"
3.
Which RAG step does a vector database help with?

Comments

Loading comments...