Lesson 2.1 · Embeddings and Semantic Search

What are Vector Embeddings?

An embedding turns a text into a list of numbers - 768 of them here - so that texts with similar meaning get similar numbers. See "refund" land next to "return" and "money back", and "Can I return my cake?" find the refunds document although they share no useful word - the exact question word search failed on in Lesson 1.7.

Beginner25 min

What you will be able to do

Explain an embedding in one sentence
Create embeddings with a local model
Read a similarity score
See meaning matched without shared words
Know why embeddings matter for RAG

The idea, in plain English

Computers cannot compare meaning directly - only numbers. An embedding model reads a text and gives back a fixed-length list of numbers, called a vector. The model was trained so that texts with similar meaning get vectors that point in a similar direction.

So "refund" and "return" get similar vectors, and "refund" and "chocolate cake" get different ones. To compare two texts, we compare their vectors and get one score: close to 1.0 means very similar, lower means less related.

For RAG this is the fix for Lesson 1.7’s first failure: "Can I return my cake?" shares no useful word with the refunds document, but it shares meaning. We use nomic-embed-text on Ollama - a small embedding model (274 MB) that runs on a laptop.

Worked example: Five words compared with each other, and a customer question matched to the right bakery document by meaning.

workflowFrom text to a scorestep 1 / 3

text A

"refund"

text B

"return"

embedding model

nomic-embed-text

two vectors

768 numbers each

similarity

0.698

React Flow

1 - Text in, numbers out

Every text becomes 768 numbers. "refund" starts 0.06, 0.039, -0.169...

numbers per text

768

Two texts become two vectors; comparing the vectors gives one score. Real numbers.

Words you will see in this module

Module 2 is about embeddings. These words come up in every lesson.

Small dictionary

EmbeddingA list of numbers that stands for the meaning of a text.

VectorA list of numbers with a direction - the embedding itself.

DimensionOne position in the vector. nomic-embed-text gives 768.

SimilarityOne number saying how close two vectors are; 1.0 = the same direction.

Embedding modelA model that turns text into vectors instead of writing text.

Semantic searchSearch by meaning, using embeddings - not by exact words.

An everyday example: a map of a city

Every shop in a city has a place on the map: two numbers, how far east and how far north.
Shops of the same kind are often close together - all the sweet shops in one street, all the phone shops in another.
An embedding is a place on a "map of meaning" - not with 2 numbers but with 768.
"refund", "return" and "money back" sit in the same street; "chocolate cake" is in another part of town.

Step 1 - get the model

nomic-embed-text is a small model made only for embeddings. One command downloads it:

274 MB - small next to llama3’s 4.7 GB.
It returns vectors only. It cannot write an answer - and llama3 cannot make embeddings here (Lesson 2.3).

Install

ollama pull nomic-embed-text

Example 1 - five words, compared

similar.py embeds five words and prints how similar each pair is:

ollama.embed(model, input=[...]) sends a list of texts and returns one vector for each.
Each vector has 768 numbers. One number alone means nothing - only the whole list does (Lesson 2.5).
similarity() multiplies the two lists number by number and adds it all up. Lesson 2.6 explains why this works.
The table is symmetric, and every word scores 1.000 with itself.

Example 1 - similar.py

import ollama


def embed(texts):
    return ollama.embed(model="nomic-embed-text", input=texts).embeddings


def similarity(a, b):                     # how close two vectors point: 1.0 = same direction
    return sum(x * y for x, y in zip(a, b))       # (the vectors are already length 1 - Lesson 2.5)


words = ["refund", "return", "money back", "delivery", "chocolate cake"]
vectors = embed(words)

print("vector for 'refund':", len(vectors[0]), "numbers, starting", [round(x, 3) for x in vectors[0][:5]])
print()
print(" " * 16 + "".join(f"{w:>16}" for w in words))
for word, a in zip(words, vectors):
    print(f"{word:<16}" + "".join(f"{similarity(a, b):16.3f}" for b in vectors))

Output

vector for 'refund': 768 numbers, starting [0.06, 0.039, -0.169, 0.007, 0.032]

                          refund          return      money back        delivery  chocolate cake
refund                     1.000           0.698           0.652           0.461           0.331
return                     0.698           1.000           0.675           0.470           0.338
money back                 0.652           0.675           1.000           0.409           0.307
delivery                   0.461           0.470           0.409           1.000           0.436
chocolate cake             0.331           0.338           0.307           0.436           1.000

Reading the table

"refund" is closest to "return" (0.698) and "money back" (0.652) - three ways to say the same idea.
"delivery" is in the middle (0.461) - a different shop topic, still about orders.
"chocolate cake" is far away (0.331).
The scores are not percentages. 0.698 does not mean "70% the same" - compare scores with each other, not with a fixed meaning.

Example 2 - the question word search missed

In Lesson 1.7, "Can I return my cake?" found the menu instead of the refund rules. return_cake.py compares the question with all seven bakery documents:

The only word the question shares with refunds.md is "can" - a word our search ignored on purpose.
By meaning, refunds.md still came first with 0.759, clearly ahead of allergens.md (0.610).
This is the whole idea of semantic search: find the text that MEANS the same, not the text that uses the same words.

Example 2 - return_cake.py

import ollama

from bakery import DOCS

question = "Can I return my cake?"
names = list(DOCS)
vectors = ollama.embed(model="nomic-embed-text", input=[question] + [DOCS[n] for n in names]).embeddings
q, docs = vectors[0], vectors[1:]

print(f"question: {question}")
print("shared words with refunds.md:", set(question.lower().strip("?").split()) & set(DOCS["refunds.md"].lower().split()))
for score, name in sorted(((sum(a * b for a, b in zip(q, d)), n) for d, n in zip(docs, names)), reverse=True):
    print(f"  {score:.3f}  {name}")

Output

question: Can I return my cake?
shared words with refunds.md: {'can'}
759  refunds.md
610  allergens.md
538  menu.md
476  history.md
409  contact.md
388  delivery.md
358  hours.md

Where embeddings fit in RAG

Remember the two pipelines from Lesson 1.6:

Ingestion: each chunk is embedded once and its vector is stored (Lesson 2.4).
Query: the question is embedded, and the stored vectors closest to it are the chunks we send to the model.
The model that writes the answer (llama3) and the model that makes vectors (nomic-embed-text) are two different models.
Embeddings are not perfect. Lessons 2.3 and 2.7 show where they fail.

Embeddings at a glance

Get the model

274 MB.

ollama pull nomic-embed-text

Embed texts

One vector per text.

ollama.embed(model="nomic-embed-text", input=["refund", "return"]).embeddings

Compare

For length-1 vectors.

sum(x * y for x, y in zip(a, b))

Read a score

Higher = more similar.

refund/return 0.698 > refund/cake 0.331

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Your words

“Add "cancel", "discount" and "birthday" to the word list. Where do they land?”

Sentences

“Compare "When do you open?" with "What are your hours?" and with "Where is the shop?".”

What usually goes wrong

Reading scores as percentages

0.698 is not "70% the same". Compare scores with each other.

Expecting the chat model to embed

llama3 here only writes text. Use an embedding model for vectors.

Looking for meaning in one number

Only the whole vector carries meaning.

Practice

Write these yourself before opening anything. Getting them wrong first is most of how this sticks.

Write find.py that takes a question from the command line, embeds it and all seven bakery documents, and prints the top 3 documents with scores. Try five questions written in your own words (no words copied from the documents) and count how often the right document is first.

Show hint

Reuse return_cake.py; read the question from sys.argv[1].

Key points

An embedding is a list of numbers that captures a text’s meaning.
Similar meaning gives similar vectors and a higher similarity score.
"refund" was closest to "return" (0.698) and far from "chocolate cake" (0.331).
By meaning, "Can I return my cake?" found refunds.md with no useful word in common.
RAG embeds every chunk once and every question at query time.

Quick check before you move on

How many numbers does nomic-embed-text give per text?

768.

Which word was closest to "refund"?

"return", with 0.698.

Which document did "Can I return my cake?" find by meaning?

refunds.md, with 0.759.

Interview questions

What is an embedding?

A dense vector representation of text (or other data) produced by a trained model so that semantic similarity corresponds to geometric closeness, typically compared with cosine similarity.

Why are embeddings used in RAG?

They let retrieval match the meaning of a question to relevant chunks even when the wording differs, which keyword search misses.

Quiz

1.
Why could word search not match "return" with "refund"?
2.
Is a score of 0.7 "70% the same"?
3.
Which two models does a RAG app with embeddings use?

Comments

Loading comments...