Lesson 1.5 · RAG Fundamentals

RAG vs Fine-Tuning

Two ways to give a model your knowledge: change the model (fine-tuning) or give it the documents (RAG). See why most question-answering apps choose RAG - a changed fee was used on the very next question - what an Ollama "custom model" really is (the same weights file), and when fine-tuning is the right tool.

Beginner25 min

What you will be able to do

Explain fine-tuning and RAG in plain words
Tell a system prompt or Modelfile apart from fine-tuning
Compare prompt size: all documents vs only the matching ones
See how fast each approach picks up a changed fact
Choose between RAG, fine-tuning and both

The idea, in plain English

There are two ways to make a model answer with your knowledge. Fine-tuning trains the model further on your examples, so the knowledge goes INTO its weights - the billions of numbers from Lesson 1.2. RAG leaves the model as it is and gives it the right documents in the prompt, every time.

Think of fine-tuning as teaching and RAG as handing over the book. Teaching changes how someone thinks and writes, but it is slow and they may misremember. A book can be updated tonight and shown as proof - but someone has to find the right page.

Real fine-tuning needs special training tools and, for a model the size of llama3, a strong GPU and hours - so we do not run it here. What we can run and measure: what a "custom model" in Ollama really is, the cost of pasting every document into every prompt, and how each approach handles a fact that changes. llama3 on Ollama.

Worked example: Our bakery’s facts given to the model three ways - a Modelfile system prompt, all documents in every prompt, and RAG - compared on size, freshness and invented answers.

workflowTwo ways to add knowledgestep 1 / 3

your knowledge

bakery documents

fine-tuning

into the weights

RAG

into the prompt

llama3

answers

React Flow

1 - A Modelfile is not fine-tuning

ollama create bakery-bot made a new name, but its weights are the same file as llama3’s.

weights file

sha256-6a0746... (same)

uncovered question

refund rule invented

Where the knowledge lives - and what that means. Real results from our bakery runs.

Words you will see in this lesson

A few words about changing a model.

Small dictionary

WeightsThe billions of numbers a model learned in training - its "memory".

Fine-tuningTraining a finished model further on your own examples, which changes its weights.

LoRAA cheaper kind of fine-tuning that trains a small add-on instead of all the weights.

System promptInstructions sent before every question. It changes no weights.

ModelfileOllama’s recipe for a named model: a base model plus a system prompt and settings.

An everyday example: a new employee

You hire a new person for the bakery counter:

Training (fine-tuning): two weeks of lessons on how to greet customers, write orders, speak politely. It changes how they work - but if the delivery fee changes, you must train them again, and they may still remember the old fee.
A price list on the counter (RAG): they read it for every customer. Change the list tonight, and tomorrow every answer is right - and the customer can see the list.
A note stuck on the wall with today’s rules (a system prompt): quick, but it does not make them any better trained.
Good shops do both: training for how to work, a price list for the facts.

Example 1 - an Ollama "custom model" is a saved prompt

Many people think ollama create fine-tunes a model. We made bakery-bot from a Modelfile with three facts in its SYSTEM prompt:

ollama create said "creating new layer" and "success".
ollama list shows bakery-bot with the same size as llama3, 4.7 GB.
ollama show --modelfile for both models points to the SAME weights file: sha256-6a0746a1... Nothing was trained. The new layer holds only the system prompt and the setting.
That is still useful - a saved prompt with a name. But the knowledge is in the prompt, not in the model.

Modelfile

FROM llama3
PARAMETER temperature 0
SYSTEM """You answer questions for Sri Lakshmi Bakery, MG Road, Hyderabad.
Facts: delivery costs Rs 50; orders above Rs 1000 are delivered free.
Sunday hours are 9 am to 2 pm. Phone 040 2345 6789."""

Output

$ ollama create bakery-bot -f Modelfile
creating new layer sha256:b8961dbee8a5d802a3b10d38c949abc6ee96f27585dc6d9af696a186687c53e0 
writing manifest 
success 
$ ollama list
NAME                 ID              SIZE      MODIFIED               
bakery-bot:latest    21653094db7c    4.7 GB    Less than a second ago    
llama3:latest        365c0bd3c000    4.7 GB    5 months ago              
gemma3:latest        a2af6cc3eb7f    3.3 GB    9 months ago              
$ ollama show llama3 --modelfile | grep "^FROM"
FROM /Users/nandigam/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
$ ollama show bakery-bot --modelfile | grep "^FROM"
FROM /Users/nandigam/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa

What bakery-bot knows - and invents

Delivery for Rs 1500: "delivery is free!" - right, the fact is in its system prompt.
A refund question, which its prompt does not cover: "custom cakes are non-refundable". Invented - our real rule allows a refund up to 24 hours before pickup.
This is exactly how a fine-tuned model behaves too: outside what it learned, it still guesses. Training does not teach a model to say "I don’t know".

bot.py

import ollama

for question in ["How much is delivery for an order of Rs 1500?",
                 "Can I get a refund for a custom cake I cancelled 2 days before pickup?"]:
    reply = ollama.generate(model="bakery-bot", prompt=question)
    print(f"Q: {question}\nA: {reply.response.strip()[:220]}\n   (prompt tokens: {reply.prompt_eval_count})\n")

Output

Q: How much is delivery for an order of Rs 1500?
A: Since the order value is above Rs 1000, delivery is free!
   (prompt tokens: 83)

Q: Can I get a refund for a custom cake I cancelled 2 days before pickup?
A: I'm Sri Lakshmi Bakery, MG Road, Hyderabad. As per our policy, custom cakes are non-refundable. Since you cancelled the order 2 days before pickup, we regret to inform you that we cannot provide a refund. However, we wou
   (prompt tokens: 87)

Example 2 - all documents vs RAG, and a changed fact

compare.py asks one question in two ways, then changes the fee:

All 7 documents in the prompt: 255 prompt tokens. Only the matching document (RAG): 79 tokens. Same right answer.
500 documents of this size would need about 12,000 tokens - more than llama3’s 8,192 window (and Lesson 1.2 showed what happens then: a silent cut).
Then the owner changes the rule: Rs 60, free only above Rs 2000. We edited the document text - and the very next answer was "Delivery costs Rs 60". No training, no restart.
A fine-tuned model would keep saying "free" until it is trained again - and you could not easily prove where its answer came from.

compare.py

import ollama

from bakery import DOCS

QUESTION = "How much is delivery for an order of Rs 1500?"


def ask(docs):
    context = "\n".join(f"[{name}] {text}" for name, text in docs.items())
    reply = ollama.generate(model="llama3", prompt=f"Context:\n{context}\n\nAnswer in one sentence and name the source file in brackets.\nQuestion: {QUESTION}",
                            options={"temperature": 0})
    return reply.response.strip(), reply.prompt_eval_count


# 1. Every document in every prompt (like a big system prompt)
answer, tokens = ask(DOCS)
print(f"all {len(DOCS)} documents:  {tokens} prompt tokens -> {answer}")

# 2. RAG: only the matching document
answer, tokens = ask({"delivery.md": DOCS["delivery.md"]})
print(f"only delivery.md: {tokens} prompt tokens -> {answer}")

# 3. 500 documents of the same size would not even fit
per_doc = round(sum(len(text) for text in DOCS.values()) / len(DOCS) / 4)     # about 4 characters per token
print(f"500 documents: about {500 * per_doc:,} tokens - llama3 holds 8,192")

# 4. The fee changes. RAG: edit the document - the next answer uses it.
DOCS["delivery.md"] = ("Delivery: we deliver within 8 km of MG Road between 10 am and 8 pm. Delivery costs Rs 60; "
                       "orders above Rs 2000 are delivered free.")
answer, tokens = ask({"delivery.md": DOCS["delivery.md"]})
print(f"after the change: {answer}")

Output

all 7 documents:  255 prompt tokens -> Delivery is free for orders above Rs 1000, so for an order of Rs 1500, delivery is free [delivery.md].
only delivery.md: 79 prompt tokens -> Delivery is free [delivery.md].
500 documents: about 12,000 tokens - llama3 holds 8,192
after the change: Delivery costs Rs 60, as the order value is below the free delivery threshold of Rs 2000. [delivery.md]

When to use which

Ask what you want to change: what the model KNOWS, or how it BEHAVES.

RAG for knowledge: facts that change (prices, policies, stock), facts that must be cited, private documents, many documents, and anything you may need to delete (a customer’s data).
Fine-tuning for behaviour: a fixed output format, a house writing style, a special vocabulary (medical, legal), or making a small model good at one narrow task.
A system prompt first: try instructions before training anything - it is free and instant.
Both: a model fine-tuned to follow your format and cite sources, fed by RAG with the facts. Many production systems end up here.
Fine-tuning costs: training data (hundreds to thousands of good examples), GPU time, evaluation, and doing it again when the base model changes.

RAG vs fine-tuning at a glance

Where knowledge lives

RAG: documents. Fine-tuning: weights.

prompt context  vs  model weights

Update a fact

RAG: edit a file. Fine-tuning: train again.

DOCS["delivery.md"] = "... Rs 60 ..."

Sources

RAG can cite; weights cannot.

[delivery.md]

Best for

RAG: facts. Fine-tuning: style and format.

what it knows  vs  how it behaves

Not fine-tuning

A saved prompt.

ollama create bakery-bot -f Modelfile

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Make your own bot

“Create a Modelfile for a shop you know, then compare the FROM lines of both models.”

Grow the prompt

“Copy the 7 documents 20 times into compare.py. How many prompt tokens now? Does the answer change?”

Change a fact twice

“Change the Sunday hours in the documents and ask again. How long until the answer is right?”

Clean up

“Remove the custom model with ollama rm bakery-bot. Does llama3 still work?”

What usually goes wrong

Thinking a Modelfile fine-tunes

bakery-bot used the same weights file as llama3. It is a saved system prompt.

Fine-tuning to teach changing facts

Every price change would need new training - and you could not cite the source.

Pasting every document into every prompt

255 tokens for 7 documents; about 12,000 for 500 - over the window.

Expecting training to stop hallucination

Outside what it learned, a trained model still guesses - bakery-bot invented a refund rule.

Practice

Write these yourself before opening anything. Getting them wrong first is most of how this sticks.

For each of these five needs, write RAG, fine-tuning, system prompt or both - and one sentence why: (1) answer questions from 2,000 HR policy pages; (2) always reply in a fixed JSON format; (3) a shop’s prices that change weekly; (4) write in a hospital’s formal medical style; (5) a support bot that must show where each answer came from.

Show hint

Ask for each: is it about what the model knows (facts) or how it behaves (style, format)? And does it change often?

Key points

Fine-tuning changes the model’s weights; RAG changes what is in the prompt.
An Ollama Modelfile is a saved system prompt, not fine-tuning - same weights file.
RAG sends only the matching documents: 79 instead of 255 tokens here.
With RAG a changed fact is used on the next question.
Use RAG for knowledge, fine-tuning for behaviour - and often both.

Quick check before you move on

Which weights file did bakery-bot use?

The same one as llama3 - sha256-6a0746a1...

What did bakery-bot say about refunds?

That custom cakes are non-refundable - invented.

How many prompt tokens: all 7 documents vs only the match?

255 vs 79.

What happened after the fee was edited in the document?

The next answer said Rs 60 - no training needed.

Interview questions

When would you choose fine-tuning over RAG?

To change behaviour - format, tone, domain language, or a narrow task for a smaller model - when the knowledge is stable; RAG is better for changing, private, citable or deletable facts. Often combine them.

What are the costs and risks of fine-tuning?

Curating quality training data, GPU and evaluation costs, retraining when facts or the base model change, possible loss of general ability, no citations, and difficulty removing specific information.

Quiz

1.
Why can RAG cite a source but a fine-tuned model cannot?
2.
Give one need that suits fine-tuning better than RAG.
3.
Why does pasting every document stop working?

Comments

Loading comments...