Lesson 1.6 · RAG Fundamentals

RAG Architecture and End-to-End Workflow

A real RAG system has two pipelines: ingestion (load, split, index, store - run when documents change) and query (retrieve, build the prompt, generate, cite - run for every question). Build both in two small files, follow one question through every stage with timings, and break it on purpose: a changed document that was not re-ingested gave the old answer.

Beginner30 min

What you will be able to do

Name the two pipelines and every stage in each
Split documents into chunks and keep their source
Follow one question through the query pipeline
Read the real prompt the model receives
Measure where the time goes
Keep the index fresh when documents change

The idea, in plain English

Lesson 1.3 did everything in one script, every time. A real RAG system splits the work in two. The ingestion pipeline prepares the documents once: load them, split them into small chunks, make each chunk searchable (the index), and store it. The query pipeline runs for each question: find the best chunks, build the prompt, generate the answer, return it with its sources.

The split matters. Ingestion can be slow - thousands of documents, run at night. Queries must be fast - a person is waiting. And the two meet only through the stored index, so whatever is in the index is what users get, even if the documents have changed since.

We build both pipelines in two files, still with a word index (Module 2 replaces it with embeddings, Modules 3-4 with a vector database). Every later module improves one stage of this picture. llama3 on Ollama, temperature 0.

Worked example: Three bakery documents split into eight chunks, and the full path of "How much does delivery cost for an order of Rs 1500?".

workflowTwo pipelinesstep 1 / 3

documents

3 .md files

ingest.py

load, split, index

index.json

8 chunks

question

delivery for Rs 1500?

query.py

retrieve, prompt

llama3

generate

React Flow

1 - Ingestion (once)

Three documents become eight chunks, one per section, each with its source.

chunks

8 (e.g. policies.md#Delivery)

time

0.001 s

Ingestion prepares the index; each query uses it. Real numbers from our run.

Words you will see in this lesson

A few architecture words.

Small dictionary

PipelineA fixed series of steps; each step’s output is the next step’s input.

IngestionPreparing documents for search: load, split, index, store.

ChunkA small piece of a document - here, one section.

IndexThe stored, searchable form of the chunks.

Top-kThe k best-matching chunks - here k = 2.

StaleOut of date: the index no longer matches the documents.

An everyday example: a library

Before the library opens, librarians label new books and write catalogue cards - ingestion. It happens once per book, and nobody is waiting.
A visitor asks a question; the librarian checks the catalogue, fetches two books and opens them at the right page - the query pipeline.
If a new edition arrives but nobody updates the catalogue, visitors are sent to the old edition - a stale index.

Example 1 - the documents and the ingestion pipeline

Three Markdown files with "## " sections. ingest.py runs four stages:

1. Load: read every .md file in docs/.
2. Split: one chunk per "## " section, so each chunk is about one topic. The source is "file#section", like policies.md#Delivery - so answers can cite a section, not just a file.
3. Index: the words of each chunk, for our word search. Module 2 stores embeddings here instead.
4. Store: write everything to index.json, so questions do not read the documents again.
The main() guard lets query.py import words() without running ingestion - our first version forgot it, and every query re-ingested.

docs/policies.md (shop.md and menu.md are the same shape)

# Sri Lakshmi Bakery - customer policies

## Refunds
Custom cakes can be refunded only if cancelled at least 24 hours before pickup.
Ready-made items cannot be refunded once they leave the shop.

## Delivery
We deliver within 8 km of MG Road between 10 am and 8 pm.
Delivery costs Rs 50; orders above Rs 1000 are delivered free.

## Allergens
All cakes may contain nuts and eggs. Eggless cakes must be ordered two days ahead.

Example 1 - ingest.py

# INGESTION PIPELINE - run once, and again whenever the documents change
import json
import re
import time
from pathlib import Path

STOP = {"a", "an", "the", "is", "are", "i", "my", "can", "do", "does", "you", "for", "to", "of", "in", "on",
        "it", "what", "if", "be", "how", "much", "must", "at", "and", "we", "sri", "lakshmi", "bakery"}


def words(text):
    return sorted({w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP})


def main():
    start = time.perf_counter()

    # 1. Load: read every document
    documents = {path.name: path.read_text() for path in sorted(Path("docs").glob("*.md"))}

    # 2. Split: one chunk per "## " section, so each chunk is about one topic
    chunks = []
    for name, text in documents.items():
        for section in re.split(r"\n## ", text)[1:]:
            title, _, body = section.partition("\n")
            chunks.append({"source": f"{name}#{title.strip()}", "text": body.strip()})

    # 3. Index: what makes each chunk findable (Module 2 replaces this with embeddings)
    for chunk in chunks:
        chunk["words"] = words(chunk["source"] + " " + chunk["text"])

    # 4. Store: save the index, so questions do not re-read the documents
    Path("index.json").write_text(json.dumps(chunks, indent=1))

    print(f"loaded {len(documents)} documents -> {len(chunks)} chunks -> index.json "
          f"({time.perf_counter() - start:.3f} s)")
    for chunk in chunks:
        print(f"  {chunk['source']:<24} {len(chunk['text']):>4} characters")


if __name__ == "__main__":       # so query.py can import words() without re-ingesting
    main()

Output

$ python ingest.py
loaded 3 documents -> 8 chunks -> index.json (0.001 s)
  menu.md#Pastries           79 characters
  menu.md#Cakes             111 characters
  policies.md#Refunds       141 characters
  policies.md#Delivery      120 characters
  policies.md#Allergens      82 characters
  shop.md#Opening hours      53 characters
  shop.md#Contact            79 characters
  shop.md#History            48 characters

Example 2 - the query pipeline

query.py runs for every question, in four stages with a timer each:

1. Load the index from index.json.
2. Retrieve: score every chunk by shared words and keep the best 2. Nothing found - "I don’t know" in code (Lesson 1.3).
3. Build the prompt: instructions, then each chunk with its [source], then the question.
4. Generate: llama3 answers from the prompt.
It prints the retrieved sources, the exact prompt, the answer and the time of each stage.

Example 2 - query.py

# QUERY PIPELINE - runs for every question
import json
import sys
import time
from pathlib import Path

import ollama

from ingest import words  # the same word rules as ingestion

question = sys.argv[1]
timings = {}

t = time.perf_counter()
chunks = json.loads(Path("index.json").read_text())                    # 1. load the index
timings["load index"] = time.perf_counter() - t

t = time.perf_counter()
asked = set(words(question))                                           # 2. retrieve the best 2 chunks
scored = sorted(chunks, key=lambda c: len(asked & set(c["words"])), reverse=True)
found = [c for c in scored[:2] if asked & set(c["words"])]
timings["retrieve"] = time.perf_counter() - t

if not found:                                                          # nothing found: answer in code
    print("I don't know - our documents do not cover this.")
    sys.exit()

t = time.perf_counter()
context = "\n".join(f"[{c['source']}] {c['text']}" for c in found)     # 3. augment: build the prompt
prompt = (f"You answer customer questions for Sri Lakshmi Bakery. Use only the context below. "
          f"Apply the numbers and rules to the question. Answer in one or two sentences "
          f"and name the source in brackets.\n\nContext:\n{context}\n\nQuestion: {question}")
timings["build prompt"] = time.perf_counter() - t

t = time.perf_counter()
reply = ollama.generate(model="llama3", prompt=prompt, options={"temperature": 0})   # 4. generate
timings["generate"] = time.perf_counter() - t

print("retrieved:", [c["source"] for c in found])
print("--- prompt sent to the model ---")
print(prompt)
print("--- answer ---")
print(reply.response.strip())
print("--- time per stage ---")
for stage, seconds in timings.items():
    print(f"  {stage:<13} {seconds * 1000:9.1f} ms")

The full path of one question

"How much does delivery cost for an order of Rs 1500?"

Retrieved: policies.md#Delivery - and menu.md#Pastries, which matched only the word "rs". A harmless extra chunk; better retrieval (Module 8) filters such noise.
The prompt is plain text: our instructions, the two chunks with their sources, and the question. This is everything the model sees - it knows nothing else about the bakery.
The answer applies the rule and cites the section: "delivery is free [policies.md#Delivery]".
Time: generate took 1,305 ms. Loading, retrieving and building the prompt together took under 1 ms. In a small RAG app, the model is almost all of the waiting.

Output

$ python query.py "How much does delivery cost for an order of Rs 1500?"
retrieved: ['policies.md#Delivery', 'menu.md#Pastries']
--- prompt sent to the model ---
You answer customer questions for Sri Lakshmi Bakery. Use only the context below. Apply the numbers and rules to the question. Answer in one or two sentences and name the source in brackets.

Context:
[policies.md#Delivery] We deliver within 8 km of MG Road between 10 am and 8 pm.
Delivery costs Rs 50; orders above Rs 1000 are delivered free.
[menu.md#Pastries] The best seller is the pineapple pastry at Rs 45. Chocolate pastry costs Rs 55.

Question: How much does delivery cost for an order of Rs 1500?
--- answer ---
Since the order value is above Rs 1000, delivery is free [policies.md#Delivery].
--- time per stage ---
  load index          0.1 ms
  retrieve            0.0 ms
  build prompt        0.0 ms
  generate         1305.5 ms

When documents change: a stale index

The owner changes the rule to "free only above Rs 2000" in docs/policies.md - and nobody re-runs ingestion:

The query still said "delivery is free" - it reads index.json, which holds the OLD text.
After python ingest.py, the answer was "Delivery costs Rs 50".
Rule: every document change must reach the index. Real systems re-ingest on a schedule or when a file changes, and only the changed files (Module 5).

Output

# the owner changes the rule: free delivery only above Rs 2000
$ sed -i "s/orders above Rs 1000/orders above Rs 2000/" docs/policies.md
$ python query.py "How much does delivery cost for an order of Rs 1500?"   # without re-ingesting
Since the order value is above Rs 1000, delivery is free [policies.md#Delivery].
$ python ingest.py
loaded 3 documents -> 8 chunks -> index.json (0.002 s)
$ python query.py "How much does delivery cost for an order of Rs 1500?"
Delivery costs Rs 50 for an order of Rs 1500. [Source: policies.md#Delivery]

The map for the rest of the course

Every later module improves one box of these two pipelines:

Split - Module 5 (document ingestion and chunking).
Index - Module 2 (embeddings); store - Modules 3 and 4 (vector databases).
Retrieve - Module 8 (retrieval optimization) and Module 9 (advanced patterns).
Build prompt and generate - Module 6 (your first full RAG app) and Module 7 (frameworks).
Around everything - Module 10 (APIs), 11 (evaluation), 12 (security), 14 (observability and cost).

RAG architecture at a glance

Ingestion

When documents change.

load -> split -> index -> store

Query

For every question.

retrieve -> build prompt -> generate -> cite

Chunk source

Cite a section.

"policies.md#Delivery"

Nothing found

Answer in code.

if not found: print("I don't know ...")

Keep it fresh

Re-run after changes.

python ingest.py

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Another question

“Run python query.py "Is writing a name on a cake free?". Which chunks were found?”

Only one chunk

“Change scored[:2] to scored[:1]. Does the delivery answer stay right?”

A new document

“Add docs/offers.md with a "## Diwali" section, re-run ingest.py, and ask about it.”

Watch the time

“Ask three different questions. Which stage is always the slowest?”

What usually goes wrong

Doing ingestion on every question

Our first query.py re-ran ingestion on import. Keep the two pipelines separate.

Forgetting to re-ingest

The index kept the old rule - "free" instead of Rs 50.

Losing the source when splitting

Store "file#section" with every chunk, or answers cannot be cited.

Optimising the wrong stage

Generation took 1,305 ms; the rest under 1 ms. Measure before you speed anything up.

Practice

Write these yourself before opening anything. Getting them wrong first is most of how this sticks.

Make ingest.py incremental: store each file’s last-modified time in index.json, and on the next run re-split only the files that changed (and drop chunks of deleted files). Print how many files were skipped, updated and removed. Test it by editing one document.

Show hint

path.stat().st_mtime gives the last-modified time. Keep a {"files": {...}, "chunks": [...]} structure in index.json.

Key points

RAG = an ingestion pipeline (offline) + a query pipeline (online).
Ingestion: load, split into chunks with sources, index, store.
Query: retrieve top chunks, build the prompt, generate, cite.
The prompt is all the model sees - print it when debugging.
In a small app almost all the time is generation.
A changed document is invisible until ingestion runs again.

Quick check before you move on

How many chunks did three documents become?

Eight - one per "## " section.

Why was menu.md#Pastries retrieved for a delivery question?

It shared the word "rs" with the question.

Which stage took the most time?

generate - 1,305 ms; the others under 1 ms.

What did the query answer after the document changed but before re-ingesting?

The old rule: delivery is free.

Interview questions

Describe the architecture of a RAG system.

An offline ingestion pipeline (load, clean, chunk, embed, store in a vector index with metadata) and an online query pipeline (embed the query, retrieve and optionally re-rank top-k chunks, assemble a prompt with citations, generate, post-process), plus evaluation and monitoring around both.

How do you keep a RAG index up to date?

Incremental ingestion triggered by change events or schedules, tracking document versions or hashes, upserting changed chunks, deleting removed ones, and monitoring index freshness.

Quiz

1.
Why are ingestion and query separate pipelines?
2.
Why store "file#section" with each chunk?
3.
What does the main() guard in ingest.py prevent?

Comments

Loading comments...