Lesson 2.3 · Embeddings and Semantic Search

How Embedding Models Work

Embedding models and chat models are built for different jobs. llama3 (8 billion parameters, "completion") refused to embed; nomic-embed-text (137 million, "embedding") gives one fixed-size vector per text - the same every time, 7 documents in about 80 ms. But it captures topic, not logic: "can be refunded" and "cannot be refunded" scored 0.971.

Intermediate25 min

What you will be able to do

Explain the difference between an encoder and a decoder model
Read a model’s capabilities in Ollama
Explain why llama3 could not embed here
See that embeddings are fixed-size and repeatable
Know what embeddings miss: negation and word order

The idea, in plain English

A chat model like llama3 is a decoder: it reads the text so far and predicts the next token, again and again (Lesson 1.2). An embedding model is an encoder: it reads the whole text at once and turns it into one vector. It writes nothing.

Embedding models are trained on pairs of texts: a question and its answer, a title and its article. Training pulls the vectors of matching pairs together and pushes others apart. That is why "refund" and "return" ended up close - and also why the vector remembers the topic of a sentence better than its exact logic.

We compare llama3 and nomic-embed-text in Ollama, see why llama3 refused to embed, and test what embeddings capture and what they miss. Ollama 0.34.1.

Worked example: llama3 and nomic-embed-text side by side - their details, their speed, and three sentence pairs that show what an embedding notices and what it misses.

workflowTwo kinds of modelstep 1 / 3

text

"Can I return my cake?"

llama3 (decoder)

8.0B, completion

✗

nomic-embed-text (encoder)

137M, embedding

new text

one token at a time

one vector

768 numbers

React Flow

1 - A chat model writes

llama3 predicts tokens. Asked to embed, this server answered 501.

capabilities

completion

embed

501 does not support embeddings

Same text in, very different outputs. Real details from ollama show.

Words you will see in this lesson

A few words about model types.

Small dictionary

EncoderA model that reads a whole text and produces a representation - here one vector.

DecoderA model that produces text one token at a time - like llama3.

CapabilitiesWhat Ollama says a model can do: completion, embedding, tools...

PoolingCombining the vectors of all tokens into one vector for the whole text.

NegationWords that turn a meaning around: not, never, cannot.

An everyday example: a librarian and a writer

A librarian reads a new book once and writes a short catalogue card: topic, shelf, keywords. That is an encoder - one summary for the whole book.
A writer produces new pages word by word. That is a decoder.
You would not ask the writer to catalogue a thousand books, or the librarian to write a novel.
And a catalogue card says "about refund policies" - it does not say whether the answer is yes or no.

Example 1 - the two models in Ollama

ollama show prints a model’s details:

llama3: architecture llama, 8.0B parameters, context 8,192, capabilities: completion.
nomic-embed-text: architecture nomic-bert (a BERT-style encoder), 137M parameters - about 60 times smaller - context 2,048, capabilities: embedding.
llama3 also has an "embedding length" of 4,096: inside, every model turns tokens into vectors. But this Ollama does not offer them as embeddings for llama3 - asking gave "This server does not support embeddings" (status 501). And even where a chat model can return vectors, they are not trained for search the way an embedding model is.

Output

$ ollama show llama3
  Model
    architecture        llama    
    parameters          8.0B     
    context length      8192     
    embedding length    4096     
    quantization        Q4_0     

  Capabilities
    completion    

  Parameters
    num_keep    24                       
$ ollama show nomic-embed-text
  Model
    architecture        nomic-bert    
    parameters          137M          
    context length      2048          
    embedding length    768           
    quantization        F16           

  Capabilities
    embedding    

Output - ollama.embed(model="llama3", input="refund")

ResponseError: This server does not support embeddings. Start it with `--embeddings` (status code: 501)

Example 2 - how an embedding model behaves

how_it_works.py tests four things:

Example 2 - how_it_works.py

import time

import ollama

from bakery import DOCS


def embed(texts):
    return ollama.embed(model="nomic-embed-text", input=texts).embeddings


def sim(a, b):
    return sum(x * y for x, y in zip(a, b))


# 1. Every text becomes the same number of values
short, long = embed(["cake", " ".join(DOCS.values()) * 3])
print("1 word ->", len(short), "numbers;  a long text ->", len(long), "numbers")

# 2. The same text always gives the same vector (no temperature, no randomness)
first, second = embed(["Can I return my cake?", "Can I return my cake?"])
print("same text twice: biggest difference =", max(abs(a - b) for a, b in zip(first, second)))

# 3. Speed: embedding all 7 documents vs writing one answer
start = time.perf_counter()
embed(list(DOCS.values()))
embed_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter()
ollama.generate(model="llama3", prompt="In one sentence: what is a bakery?", options={"temperature": 0})
generate_ms = (time.perf_counter() - start) * 1000
print(f"embed 7 documents: {embed_ms:.0f} ms;  llama3 writes one sentence: {generate_ms:.0f} ms")

# 4. What it captures - and what it misses
for a, b in [("Custom cakes can be refunded.", "Custom cakes cannot be refunded."),
             ("The dog bit the man.", "The man bit the dog."),
             ("Custom cakes can be refunded.", "We deliver within 8 km of MG Road.")]:
    va, vb = embed([a, b])
    print(f"{sim(va, vb):.3f}  {a!r} vs {b!r}")

Output

1 word -> 768 numbers;  a long text -> 768 numbers
same text twice: biggest difference = 0.0
embed 7 documents: 78 ms;  llama3 writes one sentence: 2774 ms
0.971  'Custom cakes can be refunded.' vs 'Custom cakes cannot be refunded.'
0.968  'The dog bit the man.' vs 'The man bit the dog.'
0.336  'Custom cakes can be refunded.' vs 'We deliver within 8 km of MG Road.'

Reading the results

Fixed size: one word and a long text both gave 768 numbers. The model combines all token vectors into one (pooling), so every text lands on the same "map".
Repeatable: the same text twice gave exactly the same vector - no temperature, no randomness. Store a vector once and reuse it.
Fast: all 7 documents in about 80 ms; llama3 needed about 2.8 s for one sentence. Embedding thousands of chunks is cheap; generating is the slow part (as in Lesson 1.6).
Topic, not logic: "can be refunded" vs "cannot be refunded" scored 0.971, and "the dog bit the man" vs "the man bit the dog" 0.968. Different topics scored 0.336.

What this means for RAG

Retrieval finds text about the right TOPIC. It does not check whether the text says yes or no.
That is fine - the chat model reads the retrieved text and works out the answer. Retrieval finds, generation decides.
But a document that says the opposite will be retrieved just as happily - another reason to remove outdated documents (Lesson 1.7).
Pick the embedding model for search, and the chat model for answers. They are chosen separately.

Model types at a glance

Decoder (chat)

Writes text.

llama3: 8.0B, capabilities: completion

Encoder (embedding)

Text -> one vector.

nomic-embed-text: 137M, capabilities: embedding

Check a model

Size, context, abilities.

ollama show nomic-embed-text

Fixed size

Any length in.

len(vector) == 768

Blind spot

Negation, word order.

can / cannot be refunded -> 0.971

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Other pairs

“Compare "Delivery is free" with "Delivery is not free". Then with "Delivery costs Rs 50".”

gemma3

“Run ollama show gemma3. What are its capabilities?”

Speed at scale

“Embed the 7 documents 20 times in one batch. How many milliseconds per document?”

Questions vs answers

“Compare "When do you close on Sunday?" with "Sunday 9 am to 2 pm" and with "When do you open on Sunday?".”

What usually goes wrong

Using a chat model for embeddings

llama3 refused here; even where possible, chat-model vectors are not trained for search.

Expecting retrieval to understand "not"

"can" and "cannot be refunded" scored 0.971. The chat model must read the text.

Re-embedding the same text

The vector is always the same - store it once.

Practice

Write these yourself before opening anything. Getting them wrong first is most of how this sticks.

Build a small "negation test": write five sentence pairs where only one word turns the meaning around (can/cannot, is/is not, always/never...). Embed each pair and print the similarity. Then ask llama3 for each pair: "Do these two sentences mean the same thing? Answer yes or no." Compare the two results.

Show hint

Use temperature 0 for llama3 and keep its answer short with "Answer only yes or no."

Key points

Chat models (decoders) write tokens; embedding models (encoders) turn a whole text into one vector.
nomic-embed-text: 137M parameters, 2,048-token context, 768 numbers, capability "embedding".
Every text gives a vector of the same size, and the same text always gives the same vector.
Embedding is fast; generating is slow.
Embeddings capture topic, not logic - negation and word order barely change them.

Quick check before you move on

What did Ollama answer when asked to embed with llama3?

"This server does not support embeddings" - status 501.

How similar were "can be refunded" and "cannot be refunded"?

0.971.

How many numbers did a single word and a long text each give?

768 both.

Interview questions

How do embedding models differ from generative LLMs?

Embedding models are usually bidirectional encoders trained contrastively to map texts to vectors where semantic similarity is geometric closeness; generative LLMs are decoders trained to predict next tokens. They are optimised for different tasks and chosen separately.

What are known weaknesses of dense embeddings?

Weak sensitivity to negation, word order, exact numbers and rare identifiers, domain or language mismatch, and length limits - often mitigated with hybrid search and re-ranking.

Quiz

1.
Why is it fine that retrieval ignores "not"?
2.
Why can you store an embedding and reuse it?
3.
Which model type writes the answer in RAG, and which finds the chunks?

Comments

Loading comments...