Problems with LLM Knowledge and Hallucinations
How often does a model invent facts? We asked llama3 ten questions about our bakery and graded every answer in code: 6 invented, 1 lucky guess with a fake source, 2 no answer, 1 right. With RAG: 8 right and 2 misses - and not one invented answer. Plus five different phone numbers for the same question, and a training cut-off you can see.
What you will be able to do
- Name the three knowledge problems of an LLM
- Explain what a hallucination is and why it happens
- Measure hallucinations with a small graded test
- See RAG remove the invented answers - and what it still misses
- Use "ask several times" as a quick hallucination check
The idea, in plain English
An LLM knows only what was in its training text, up to a date - the cut-off. That gives three problems. It does not know your private data (your shop, your company). It does not know anything newer than the cut-off. And when it does not know, it often does not say so: it writes a fluent, confident answer anyway.
That last problem has a name: hallucination. The model is a next-token predictor (Lesson 1.2). It writes what a good answer usually looks like - and a good answer about a phone number contains a phone number. So it writes one.
In this lesson we stop guessing and count. Ten questions about our bakery, an automatic grader, and two runs: without RAG and with it. llama3 on Ollama, temperature 0.
Worked example: Ten questions about Sri Lakshmi Bakery, graded automatically - first without, then with the documents.
1 - Without RAG
llama3 answers from memory. It has never seen our bakery.
The same ten questions, graded in code. Real results.
Words you will see in this lesson
A few words about what models know.
HallucinationA confident answer that is not true - an invented fact, name, number or source.Training cut-offThe date after which the model has seen no text.Private dataYour own documents - never in the model’s training text.GraderCode that checks an answer and labels it, so we can count instead of guess.Ground truthThe correct answers we compare against - here, the bakery’s documents.An everyday example: the over-helpful stranger
- You ask a stranger on the street for the way to a small shop.
- They have never heard of it - but they do not want to seem unhelpful, so they point confidently: "Second left, near the temple."
- Ask three strangers and you may get three different directions.
- A shopkeeper who checks a map before answering is RAG: the answer comes from a real source.
Three knowledge problems
An LLM fails on knowledge in three different ways:
- Private data - it never saw your documents. Our bakery is not on the internet, so llama3 cannot know its phone number.
- Training cut-off - it saw nothing after a date. Asked for the latest Python, llama3 said 3.10.7. Python 3.11, 3.12, 3.13 and 3.14 have come out since.
- Hallucination - when it does not know, it often answers anyway, with confidence and sometimes with a fake source.
- Asked for today’s date, llama3 correctly said it cannot know. So it can say "I don’t know" - it just does not do it reliably.
Q: What is the latest stable version of Python? Answer in one short sentence.
A: The latest stable version of Python is Python 3.10.7.
Q: What is today's date? Answer in one short sentence.
A: I'm a large language model, I don't have access to real-time information, so I don't know today's date.Example 1 - ten questions and a grader
bakery.py holds the truth: seven documents and ten questions with the words a right answer must contain. Eight questions are covered by the documents; two (gift cards, parking) are not.
- grade.py labels each answer. RIGHT: it contains the true fact. LUCKY GUESS: the true fact, but with an invented source. INVENTED: a made-up fact or source. no answer: it said it does not know.
- For the two uncovered questions, the only right answer is "I don’t know".
- An invented source is found with a simple pattern: "according to ..." (but not a real [file]), a +91 phone number, a www. address.
- count_answers.py asks every question and prints the label and the start of the answer. With --rag it uses the tiny RAG app from Lesson 1.3 (word search, top 2 documents, "I don’t know" in code).
# Sri Lakshmi Bakery's own documents - the truth we check answers against
DOCS = {
"refunds.md": "Refunds: custom cakes can be refunded only if cancelled at least 24 hours before pickup. "
"Ready-made items cannot be refunded once they leave the shop.",
"delivery.md": "Delivery: we deliver within 8 km of MG Road between 10 am and 8 pm. Delivery costs Rs 50; "
"orders above Rs 1000 are delivered free.",
"hours.md": "Opening hours: Monday to Saturday 8 am to 9 pm, Sunday 9 am to 2 pm.",
"allergens.md": "Allergens: all cakes may contain nuts and eggs. Eggless cakes must be ordered two days ahead.",
"contact.md": "Contact: phone 040 2345 6789, email hello@srilakshmibakery.in. The owner is Lakshmi Rao.",
"menu.md": "Menu: the best seller is the pineapple pastry at Rs 45. A 1 kg black forest cake costs Rs 650.",
"history.md": "History: the bakery opened in 1998 on MG Road, Hyderabad.",
}
# Ten questions. "expect" = words a correct answer must contain; None = the documents do not cover it.
QUESTIONS = [
("What is the phone number of Sri Lakshmi Bakery on MG Road, Hyderabad?", ["2345 6789"]),
("Who owns Sri Lakshmi Bakery on MG Road, Hyderabad?", ["lakshmi rao"]),
("In which year did Sri Lakshmi Bakery on MG Road, Hyderabad open?", ["1998"]),
("What time does Sri Lakshmi Bakery on MG Road close on Sunday?", ["2 pm"]),
("How much does a 1 kg black forest cake cost at Sri Lakshmi Bakery, MG Road?", ["650"]),
("What is the best seller at Sri Lakshmi Bakery, MG Road?", ["pineapple"]),
("How far does Sri Lakshmi Bakery on MG Road deliver?", ["8 km"]),
("How many days ahead must I order an eggless cake at Sri Lakshmi Bakery, MG Road?", ["two days", "2 days", "2-day"]),
("Does Sri Lakshmi Bakery on MG Road sell gift cards?", None),
("Does Sri Lakshmi Bakery on MG Road have parking?", None),
]import re
NO_ANSWER = ["i don't", "i do not", "i'm not aware", "not sure", "i'm just an ai", "don't have access",
"do not cover", "no information"]
# Signs of an invented source: "according to ..." (but not a real [file.md]), a phone number, a web address
FAKE_SOURCE = re.compile(r"according to (?!\[)|\+91|www\.|directories")
def grade(text, expect):
low = text.lower()
fake = bool(FAKE_SOURCE.search(low))
if expect is None: # the documents do not cover it
if fake or not any(p in low for p in NO_ANSWER):
return "INVENTED"
return "RIGHT (no guess)"
if any(word in low for word in expect):
return "LUCKY GUESS" if fake else "RIGHT"
if fake:
return "INVENTED"
return "no answer" if any(p in low for p in NO_ANSWER) else "INVENTED"import re
import sys
import ollama
from bakery import DOCS, QUESTIONS
from grade import grade
STOP = {"a", "an", "the", "is", "are", "i", "my", "can", "do", "does", "you", "for", "to", "of", "in",
"on", "it", "what", "if", "get", "be", "how", "who", "which", "much", "many", "must", "at",
"sri", "lakshmi", "bakery", "mg", "road", "hyderabad", "have", "did", "time", "year", "far"}
def words(text):
return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}
def retrieve(question, k=2):
scored = sorted(DOCS.items(), key=lambda item: len(words(question) & words(item[1])), reverse=True)
return [(name, text) for name, text in scored[:k] if words(question) & words(text)]
def ask(question, use_rag):
if not use_rag:
return ollama.generate(model="llama3", prompt=question, options={"temperature": 0}).response
found = retrieve(question)
if not found:
return "I don't know - our documents do not cover this."
context = "\n".join(f"[{name}] {text}" for name, text in found)
prompt = (f"You answer customer questions for Sri Lakshmi Bakery. Use the context below. "
f"Read the numbers and rules carefully and apply them to the question. "
f"Give a short answer and name the source file in brackets.\n\n"
f"Context:\n{context}\n\nQuestion: {question}")
return ollama.generate(model="llama3", prompt=prompt, options={"temperature": 0}).response
use_rag = sys.argv[1:] == ["--rag"]
counts = {}
for question, expect in QUESTIONS:
reply = ask(question, use_rag).strip()
result = grade(reply, expect)
counts[result] = counts.get(result, 0) + 1
print(f"{result:<17} | {question}")
print(f"{'':<17} | -> {reply[:140].replace(chr(10), ' ')}")
print("\nTOTAL:", counts)Result 1 - without RAG: 6 of 10 invented
- It invented a phone number "according to online directories", an owner, a best seller ("Butter Cake"), a delivery price table, and parking details.
- It moved our Hyderabad bakery to Bangalore - three times.
- The Sunday-hours answer began "I’m not sure..." - and then still gave a phone number, +91 80 2222 2222.
- Eggless cakes: "2-day advance order" is right - by luck. The same answer cited "Sri Lakshmi Bakery’s website" and gave another invented number and web address. A right fact with a fake source is still not trustworthy.
- Two answers honestly said it did not know (opening year, cake price), and gift cards was fully right - the rest were invented.
$ python count_answers.py
INVENTED | What is the phone number of Sri Lakshmi Bakery on MG Road, Hyderabad?
| -> I'm happy to help! According to online directories and reviews, the phone number of Sri Lakshmi Bakery on MG Road, Hyderabad is +91 98480 44
INVENTED | Who owns Sri Lakshmi Bakery on MG Road, Hyderabad?
| -> A popular question! Sri Lakshmi Bakery on MG Road, Hyderabad is owned by Sri Lakshmi Narasimha Swamy, a renowned bakery chain in Hyderabad.
no answer | In which year did Sri Lakshmi Bakery on MG Road, Hyderabad open?
| -> I'm not aware of any information about a Sri Lakshmi Bakery on MG Road in Hyderabad. It's possible that it may not be a well-known or popula
INVENTED | What time does Sri Lakshmi Bakery on MG Road close on Sunday?
| -> I'm not sure what the exact closing time of Sri Lakshmi Bakery on MG Road is on Sundays. However, I can suggest some ways for you to find ou
no answer | How much does a 1 kg black forest cake cost at Sri Lakshmi Bakery, MG Road?
| -> I'm just an AI, I don't have have access to real-time pricing information or specific details about Sri Lakshmi Bakery on MG Road. Additiona
INVENTED | What is the best seller at Sri Lakshmi Bakery, MG Road?
| -> Sri Lakshmi Bakery on MG Road is a popular spot for bakery items in Bangalore! According to reviews and customer feedback, the best-seller a
INVENTED | How far does Sri Lakshmi Bakery on MG Road deliver?
| -> A question about a popular bakery in Bangalore! According to Sri Lakshmi Bakery's website and online ordering platforms, they offer deliver
LUCKY GUESS | How many days ahead must I order an eggless cake at Sri Lakshmi Bakery, MG Road?
| -> A specific question! According to Sri Lakshmi Bakery's website and reviews, they typically require a 2-day advance order for eggless cakes.
RIGHT (no guess) | Does Sri Lakshmi Bakery on MG Road sell gift cards?
| -> I'm not aware of any information about Sri Lakshmi Bakery on MG Road selling gift cards. It's possible that they may not offer gift cards or
INVENTED | Does Sri Lakshmi Bakery on MG Road have parking?
| -> According to online reviews and ratings, Sri Lakshmi Bakery on MG Road in Bangalore does not have a dedicated parking space. However, there
TOTAL: {'INVENTED': 6, 'no answer': 2, 'LUCKY GUESS': 1, 'RIGHT (no guess)': 1}Result 2 - with RAG: nothing invented
- Six answers right, each with its source file.
- Gift cards and parking: "I don’t know" - from code, because the search found nothing. Both right.
- Owner and opening year: "I don’t know" - but our documents DO have these answers. The word search missed them: "owns" is not "owner", and "open" is not "opened".
- So RAG removed every invented answer. Its remaining mistakes come from retrieval - which Module 2 improves with embeddings that match meaning, not exact words.
$ python count_answers.py --rag
RIGHT | What is the phone number of Sri Lakshmi Bakery on MG Road, Hyderabad?
| -> The phone number of Sri Lakshmi Bakery on MG Road, Hyderabad is 040 2345 6789. [contact.md]
no answer | Who owns Sri Lakshmi Bakery on MG Road, Hyderabad?
| -> I don't know - our documents do not cover this.
no answer | In which year did Sri Lakshmi Bakery on MG Road, Hyderabad open?
| -> I don't know - our documents do not cover this.
RIGHT | What time does Sri Lakshmi Bakery on MG Road close on Sunday?
| -> Sri Lakshmi Bakery on MG Road closes at 2 pm on Sunday. [hours.md]
RIGHT | How much does a 1 kg black forest cake cost at Sri Lakshmi Bakery, MG Road?
| -> A 1 kg black forest cake costs Rs 650. [menu.md]
RIGHT | What is the best seller at Sri Lakshmi Bakery, MG Road?
| -> The best seller at Sri Lakshmi Bakery, MG Road is the pineapple pastry, which costs Rs 45. [menu.md]
RIGHT | How far does Sri Lakshmi Bakery on MG Road deliver?
| -> Sri Lakshmi Bakery delivers within 8 km of MG Road. [delivery.md]
RIGHT | How many days ahead must I order an eggless cake at Sri Lakshmi Bakery, MG Road?
| -> According to [allergens.md], eggless cakes must be ordered two days ahead. Therefore, you must order an eggless cake at Sri Lakshmi Bakery,
RIGHT (no guess) | Does Sri Lakshmi Bakery on MG Road sell gift cards?
| -> I don't know - our documents do not cover this.
RIGHT (no guess) | Does Sri Lakshmi Bakery on MG Road have parking?
| -> I don't know - our documents do not cover this.
TOTAL: {'RIGHT': 6, 'no answer': 2, 'RIGHT (no guess)': 2}A quick test: ask five times
A real fact comes out the same every time. An invented one changes. We asked for the phone number five times at temperature 0.8:
- Five runs, five different numbers. None is the real one (040 2345 6789).
- This is a cheap check you can use anywhere: if repeated answers disagree, the model is guessing.
- It does not prove an answer is right - a model can repeat the same wrong fact - but disagreement is a strong warning.
import re
import ollama
QUESTION = "What is the phone number of Sri Lakshmi Bakery on MG Road, Hyderabad? Reply with only the number."
numbers = []
for run in range(1, 6):
reply = ollama.generate(model="llama3", prompt=QUESTION, options={"temperature": 0.8}).response.strip()
found = re.findall(r"\+?\d[\d -]{7,}\d", reply)
numbers.append(found[0] if found else reply[:40])
print(f"run {run}: {numbers[-1]}")
print("different answers:", len(set(numbers)), "of 5")run 1: 9948111111
run 2: 040-2323 4444
run 3: 9392222222
run 4: 040-2324-5555
run 5: 040-2324-2222
different answers: 5 of 5Knowledge problems at a glance
Private dataNever in training.
phone number of our bakery -> invented
Cut-offNothing after a date.
"latest Python" -> 3.10.7
HallucinationConfident, not true.
owner, best seller, prices -> invented
Measure itGrade answers in code.
grade(reply, expect)
Quick checkAsk several times.
5 runs at 0.8 -> 5 different numbers
Try it yourself
The code does not change. Swap the content string and the program does something else entirely.
“Ask llama3 about a small local shop or office you know. How much does it invent?”
“Run count_answers.py with gemma3 instead of llama3. Is the count different?”
“Add "If you do not know, say I don’t know" to every question without RAG. Does the count of INVENTED go down?”
“Add "owns" and "open" as extra words in the documents. Do the two RAG misses become right?”
What usually goes wrong
llama3 cited "online directories" for a number it made up.
The eggless answer had the right number and an invented source. Check the source too.
Count over many questions with a grader - the numbers tell a clearer story.
With RAG, the two wrong answers came from the word search, not from llama3.
Practice
Write these yourself before opening anything. Getting them wrong first is most of how this sticks.
Add five more questions to bakery.py - three that the documents answer and two that they do not - and run both modes. Then add a "repeat check" to count_answers.py: ask each question 3 times at temperature 0.8 without RAG, and print SUSPICIOUS when the three answers differ. Does SUSPICIOUS match INVENTED?
Show hintHide hint
Compare the answers after lower() and strip(); for numbers, compare the digits only.
Key points
- LLMs do not know private data or anything after their training cut-off.
- When they do not know, they often invent an answer - a hallucination - sometimes with a fake source.
- Measured without RAG: 6 of 10 invented, 1 lucky guess.
- Measured with RAG: 0 invented; the 2 misses came from retrieval.
- Asking several times and comparing is a cheap hallucination warning.
Quick check before you move on
Interview questions
What causes LLM hallucinations?
Models are trained to produce likely text, not verified facts; gaps in training data, private or post-cut-off topics, and pressure to be helpful lead to fluent fabrications.
How would you measure hallucination in a RAG system?
A labelled question set with ground-truth answers and sources, automatic checks for correctness, faithfulness to retrieved context and citation validity, plus separate retrieval metrics - tracked over time.
Quiz
- 1.
Why does an LLM invent a phone number instead of saying "I don’t know"?
- 2.
Why is a right fact with an invented source still a problem?
- 3.
Which step caused RAG’s two misses, and which module improves it?
Comments
Sign in to leave a comment. Your name and photo come from Google; nothing else is shared.
Loading comments...