Lesson 1.2 · RAG Fundamentals

What is a Large Language Model (LLM)?

An LLM predicts the next piece of text, again and again. Watch llama3 choose " Paris" with 64% probability, count tokens (Telugu cost five times more than English), read the model’s size and limits, and see a long prompt silently cut to a third - the fact in the middle was lost.

Beginner25 min

What you will be able to do

Explain an LLM as a next-token predictor
Read the probabilities a model gives its next-token choices
Explain what tokens are and why their count matters
Read a model’s size, context length and abilities
Explain the context window - and the silent cut that breaks RAG
Explain the training cut-off

The idea, in plain English

A large language model (LLM) is a program trained on a huge amount of text to do one thing: given some text, predict what comes next. It does not look anything up. It produces one small piece - a token - then adds it to the text and predicts again.

"Large" means billions of numbers (parameters) learned during training. llama3, the model we use, has 8 billion. Those numbers hold patterns of language - grammar, style, many facts - but not a database you can check.

In this lesson we open the box a little: we watch the predictions, count tokens, read the model’s limits, and break a prompt by making it too long. Everything ran on llama3 with Ollama 0.34.1.

Worked example: llama3 completing "The capital of France is" one token at a time, four sentences counted in tokens, and a secret code word hidden in a long prompt.

workflowOne token at a timestep 1 / 3

text so far

The capital of France is

llama3

predicts the next token

pick one

temperature 0: the top one

React Flow

Step 1

The model scores every possible next token. " Paris" is most likely.

" Paris"

64%

" a"

13%

" the"

"The capital of France is" - llama3’s real choices and probabilities.

Words you will see in this lesson

The words for how a model works.

Small dictionary

LLMLarge language model: a model trained on huge amounts of text to predict the next token.

TokenA piece of text the model reads and writes: a word, part of a word, or a symbol.

ParametersThe numbers learned in training. llama3 has 8 billion.

ProbabilityHow likely the model thinks each next token is.

Context windowThe most tokens the model can look at at once - prompt and answer together.

Training cut-offThe date the training data ends. The model knows nothing after it.

An everyday example: your phone’s keyboard

When you type "See you", your phone suggests "tomorrow", "soon", "later". (Next-token prediction.)
Tap a suggestion again and again, and it writes a whole sentence. (Generating, token by token.)
An LLM is that idea, trained on far more text, with billions of parameters - so it can continue almost anything.
Your keyboard does not check facts. Neither does an LLM.

Example 1 - watch the model predict

We ask for ONE token at a time, with its four best alternatives, and add the chosen token to the text:

raw=True: no chat template - just continue the text as it is.
num_predict: 1 - generate exactly one token.
logprobs=True, top_logprobs=4 - also return the four most likely tokens and their scores.
math.exp(logprob) turns a score into a probability (0.64 = 64%).

next_token.py

import math
import ollama

text = "The capital of France is"
print(f"prompt: {text!r}")
for step in range(1, 7):
    reply = ollama.generate(model="llama3", prompt=text, raw=True,           # raw: no chat template, just continue
                            options={"temperature": 0, "num_predict": 1},     # predict ONE token
                            logprobs=True, top_logprobs=4)
    candidates = reply.logprobs[0].top_logprobs
    choices = ", ".join(f"{c.token!r} {math.exp(c.logprob):.0%}" for c in candidates)
    chosen = reply.logprobs[0].token
    print(f"step {step}: chose {chosen!r:10}  top 4: {choices}")
    text += chosen
print(f"result: {text!r}")

Output

prompt: 'The capital of France is'
step 1: chose ' Paris'    top 4: ' Paris' 64%, ' a' 13%, ' the' 5%, ' known' 2%
step 2: chose ','         top 4: ',' 46%, '.' 34%, '.\n' 15%, ' (' 1%
step 3: chose ' which'    top 4: ' which' 43%, ' but' 13%, ' and' 12%, ' the' 11%
step 4: chose ' is'       top 4: ' is' 94%, ' has' 4%, ' was' 0%, ' lies' 0%
step 5: chose ' located'  top 4: ' located' 56%, ' the' 10%, ' also' 8%, ' situated' 7%
step 6: chose ' in'       top 4: ' in' 99%, ' on' 1%, ' at' 0%, ' along' 0%
result: 'The capital of France is Paris, which is located in'

Reading Example 1

" Paris" was the most likely next token - but only 64%. "a" (13%) could start "a city..."
Tokens often include the space before a word: " Paris", " which".
Some steps are almost certain (" in" 99%), others are close ("," 46% vs "." 34%).
At temperature 0 the model always takes the top token. Higher temperature sometimes takes a lower one - that is the variety you saw in Lesson 1.1.
Nothing here checks facts. "Paris" won because it is the most common continuation in the training text.

Example 2 - counting tokens

Models read and write tokens, not letters or words. Cost, speed and limits are all counted in tokens. Ollama reports how many tokens a prompt used:

Common English words are often one token each.
The same greeting in Telugu used 37 tokens for 22 characters - about five times the English count. Many languages need more tokens, so they cost more and fill the context sooner.
A rough rule for English: about 4 characters per token.

tokens.py

import ollama

for text in ["Hello, how are you?",
             "హలో, మీరు ఎలా ఉన్నారు?",
             "Retrieval-Augmented Generation",
             "The quick brown fox jumps over the lazy dog."]:
    reply = ollama.generate(model="llama3", prompt=text, raw=True, options={"num_predict": 1})
    print(f"{reply.prompt_eval_count:3} tokens  {len(text):3} characters  {text}")

Output

tokens   19 characters  Hello, how are you?
tokens   22 characters  హలో, మీరు ఎలా ఉన్నారు?
tokens   30 characters  Retrieval-Augmented Generation
tokens   44 characters  The quick brown fox jumps over the lazy dog.

Example 3 - the model’s size and limits

ollama show describes a model:

parameters 8.0B - 8 billion learned numbers.
context length 8192 - at most 8,192 tokens at once.
quantization Q4_0 - the numbers are stored in 4 bits to make the file small (4.7 GB) and fast on a laptop.
Capabilities: completion only. That is why llama3 cannot call tools natively or make embeddings in Ollama - you will need an embedding model in Module 2.

ollama show llama3

  Model
    architecture        llama
    parameters          8.0B
    context length      8192
    embedding length    4096
    quantization        Q4_0

  Capabilities
    completion

Example 4 - the context window, and a silent cut

RAG puts documents into the prompt. So what happens when the prompt is too long? We hid a code word in the MIDDLE of about 5,600 tokens of text:

With Ollama’s default settings, the prompt counted only 2,060 tokens. The rest was cut - silently. The answer: "the". Wrong.
With num_ctx=8192 (llama3’s full window), all 5,634 tokens were read. The answer: MANGO-42. Right.
No error, no warning. In a RAG app the cut part would be your retrieved documents.
When the code word was at the START, the default run still answered correctly - Ollama keeps the beginning. Facts in the middle are the ones lost.

context2.py

import ollama

lines = [f"Line {i}: the shop opens at nine and closes at eight." for i in range(400)]
lines.insert(200, "The secret code word is MANGO-42.")              # the fact is in the MIDDLE
question = "What is the secret code word? Answer with the code word only."
prompt = "\n".join(lines) + "\n" + question

for num_ctx in [None, 8192]:
    options = {"temperature": 0} if num_ctx is None else {"temperature": 0, "num_ctx": num_ctx}
    reply = ollama.generate(model="llama3", prompt=prompt, options=options)
    print(f"num_ctx={num_ctx or 'default':7} prompt tokens counted: {reply.prompt_eval_count:5}  answer: {reply.response.strip()[:70]!r}")

Output

num_ctx=default prompt tokens counted:  2060  answer: 'the'
num_ctx=   8192 prompt tokens counted:  5634  answer: 'MANGO-42'

Watch out: Ollama’s default context was much smaller than llama3’s 8,192 tokens, and it cut long prompts without an error. In every RAG app, set num_ctx and compare prompt_eval_count with what you sent.

The training cut-off

A model learns from text collected up to a date - its training cut-off. After that, it knows nothing.
In Lesson 1.1 llama3 said "it’s still 2023". It cannot know newer events, prices or products.
It also never saw your private documents.
Both problems have the same fix: put the facts in the prompt. That is RAG (Lesson 1.3).

The recipe

1. Think of an LLM as "predict the next token", repeated.
2. Count in tokens: prompt_eval_count (input) and eval_count (output).
3. Know your model: ollama show - parameters, context length, capabilities.
4. Set num_ctx for long prompts, and check nothing was cut.
5. Never expect knowledge after the cut-off, or of your own data.

LLMs at a glance

One token

With alternatives.

ollama.generate(..., raw=True, options={"num_predict": 1}, logprobs=True, top_logprobs=4)

Probability

From a logprob.

math.exp(c.logprob)

Tokens used

Input / output.

reply.prompt_eval_count / reply.eval_count

Model facts

Size, context, abilities.

ollama show llama3

Full context

For long prompts.

options={"num_ctx": 8192}

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Other sentences

“Run next_token.py with "My favourite food is". How sure is the model at each step?”

Your language

“Count tokens for a sentence in your language and the same sentence in English.”

Start vs middle

“Move the code word to the end of the prompt. Does the default setting find it?”

What usually goes wrong

Thinking the model looks things up

It predicts likely text. " Paris" won with 64% because it is common, not because it was checked.

Counting words instead of tokens

Costs and limits are in tokens. Telugu used five times more tokens than English.

Trusting the default context

A long prompt was cut to 2,060 tokens without an error, and the answer was wrong.

Practice

Write these yourself before opening anything. Getting them wrong first is most of how this sticks.

Write a function safe_generate(prompt) that sets num_ctx=8192, and raises an error if prompt_eval_count is much smaller than an estimate of the prompt’s tokens (characters / 4).

Show hint

Compare reply.prompt_eval_count with len(prompt) / 4; warn if it is less than half.

Key points

An LLM predicts the next token again and again; llama3 has 8 billion parameters.
" Paris" came first with 64% - likely, not checked.
Everything is counted in tokens; Telugu took about 5x more tokens than English.
llama3’s context length is 8,192 tokens; Ollama’s default cut a long prompt silently, and the answer was wrong.
A model knows nothing after its training cut-off, and nothing about your private data.

Quick check before you move on

What does an LLM do at each step?

Predicts the next token, adds it to the text, and repeats.

What is a token?

A piece of text: a word, part of a word, or a symbol.

What happened to the long prompt with default settings?

It was cut to 2,060 tokens without an error, and the answer was wrong.

Interview questions

How does an LLM generate text?

Autoregressively: it computes a probability distribution over the vocabulary for the next token given the context, selects one (greedy or sampled by temperature), appends it, and repeats.

What is a context window and why does it matter for RAG?

The maximum number of tokens the model attends to at once. Retrieved context, instructions, question and answer must fit; exceeding it truncates input, losing retrieved facts.

Quiz

1.
Why does a long prompt in another language fill the context sooner?
2.
What does "Q4_0" in ollama show mean?
3.
How do you check that a prompt was not cut?

Comments

Loading comments...