Lesson 2.2 · Embeddings and Semantic Search

Tokens and Text Preprocessing

What goes into the embedding model matters. Measured: HTML leftovers lowered the match from 0.847 to 0.758, menu and footer text to 0.789 - and a small clean() brought both back. Spaces and CAPITALS changed nothing. Two hard limits: text past 2,048 tokens is cut off silently, and this English model turned Telugu words into one identical vector.

Beginner25 min

What you will be able to do

Count tokens for the embedding model
Measure how messy text changes similarity
Write a small clean() function
Know the embedding model’s length limit and what happens past it
Check that the model supports your language

The idea, in plain English

An embedding model reads tokens (Lesson 1.2), not raw text. Everything you send becomes tokens and moves the vector - including HTML tags, website menus, cookie notices and page footers. Text from real sources is rarely clean.

Every embedding model also has a length limit - nomic-embed-text reads at most 2,048 tokens - and a vocabulary built for certain languages. What falls outside either is lost without an error.

We measure all of this on the bakery’s refund policy: four messy versions, a clean() function, a fact hidden past the length limit, and two Telugu words. nomic-embed-text on Ollama.

Worked example: The bakery’s refund policy as clean text, as scraped HTML, with a website menu and footer - and a gift-card fact hidden at the end of a long page.

workflowClean before you embedstep 1 / 3

scraped text

tags, menu, footer

clean()

strip, unescape, join

embed

max 2,048 tokens

vector

compared to the question

✗

React Flow

1 - Messy text in

Tags and website menus are tokens too. They pull the vector away from the meaning.

html leftovers

0.758 (102 tokens)

menu and footer

0.789

Similarity to "Can I get a refund for a custom cake?". Real scores.

Words you will see in this lesson

A few words about preparing text.

Small dictionary

PreprocessingCleaning text before embedding it.

BoilerplateText repeated on every web page: menus, footers, cookie notices.

HTML entityA code like   for a special character - here a space.

TruncationCutting text that is too long. Here it happens silently.

Unknown tokenThe single token a model uses for anything outside its vocabulary.

An everyday example: photocopying a page

You photocopy a recipe from a magazine to keep in a folder.
If the copy also has the advertisements, the page number and a coffee stain, the recipe is harder to find later - that is HTML and boilerplate.
If the recipe continues on the next page and you copy only one page, the end is simply missing - nobody tells you. That is truncation.
And a recipe in a language you cannot read is just shapes to you - that is a model without your language.

Tokens in an embedding model

ollama.embed also returns prompt_eval_count - how many tokens were read. count_tokens.py tries a few texts:

An empty text still counts 2: the model adds a start and an end token to every text.
"cake", "cakes" and "Hyderabad" are one token each (3 with the two extra); "refundable" is split into three pieces.
"కేక్" (cake in Telugu) is ONE token. That looks efficient - the next section shows it is not.

count_tokens.py

import ollama

for text in ["", "cake", "cakes", "refundable", "Hyderabad", "black forest cake", "కేక్"]:
    tokens = ollama.embed(model="nomic-embed-text", input=text or " ").prompt_eval_count
    print(f"{text!r:<22} {tokens} tokens")

Output

''                     2 tokens
'cake'                 3 tokens
'cakes'                3 tokens
'refundable'           5 tokens
'Hyderabad'            3 tokens
'black forest cake'    5 tokens
'కేక్'                 3 tokens

Example 1 - messy text, measured

messy.py takes the refund policy in five versions and compares each with the question "Can I get a refund for a custom cake?" - before and after clean():

clean() does five things: drop <script> blocks, drop all other tags, turn entities like   into real characters, drop lines with "|" (our menu and footer), and join everything with single spaces.
Real projects use a proper HTML parser (Module 5). This small version is enough to see the effect.

Example 1 - messy.py

import html
import re

import ollama

QUESTION = "Can I get a refund for a custom cake?"
CLEAN = ("Refunds: custom cakes can be refunded only if cancelled at least 24 hours before pickup. "
         "Ready-made items cannot be refunded once they leave the shop.")

VERSIONS = {
    "clean": CLEAN,
    "html leftovers": ("<div class=\"policy\"><p><b>Refunds:</b>&nbsp;custom&nbsp;cakes can be refunded only if "
                       "cancelled at least 24&nbsp;hours before pickup.</p><p>Ready-made items cannot be refunded "
                       "once they leave the shop.</p></div><script>trackPage('policy');</script>"),
    "extra spaces": CLEAN.replace(" ", "     ").replace(". ", ".\n\n\n\t"),
    "ALL CAPITALS": CLEAN.upper(),
    "menu and footer": ("Home | Menu | Cakes | Contact | Login | Cart (0)\n" + CLEAN +
                        "\n© 2026 Sri Lakshmi Bakery. All rights reserved. | Privacy | Terms | Follow us on Instagram"),
}


def clean(text):
    text = re.sub(r"<script.*?</script>", " ", text, flags=re.S)     # drop scripts
    text = re.sub(r"<[^>]+>", " ", text)                             # drop tags
    text = html.unescape(text)                                       # &nbsp; -> a real space
    lines = [line for line in text.splitlines() if "|" not in line]  # drop menu and footer lines
    return re.sub(r"\s+", " ", " ".join(lines)).strip()              # one space between words


def embed(text):
    reply = ollama.embed(model="nomic-embed-text", input=text)
    return reply.embeddings[0], reply.prompt_eval_count


q, _ = embed(QUESTION)
print(f"{'version':<17}{'tokens':>7}{'similarity':>12}{'after clean()':>15}")
for name, text in VERSIONS.items():
    vector, tokens = embed(text)
    cleaned, _ = embed(clean(text))
    score = sum(a * b for a, b in zip(q, vector))
    after = sum(a * b for a, b in zip(q, cleaned))
    print(f"{name:<17}{tokens:>7}{score:>12.3f}{after:>15.3f}")

Output

version           tokens  similarity  after clean()
clean                 37       0.847          0.847
html leftovers       102       0.758          0.847
extra spaces          37       0.847          0.847
ALL CAPITALS          37       0.847          0.847
menu and footer       73       0.789          0.847

Reading the results

HTML leftovers: 102 tokens instead of 37, and the similarity fell from 0.847 to 0.758. Most of the tokens were tags and code, not policy.
Menu and footer: 73 tokens and 0.789. "Home | Menu | Login" and "All rights reserved" are not about refunds.
After clean(), every version scored 0.847 - the same as the clean text.
Extra spaces and ALL CAPITALS changed nothing: this model ignores both (its tokenizer lowercases and splits on spaces). Other models may not - measure yours.

Example 2 - past the length limit

nomic-embed-text reads at most 2,048 tokens. too_long.py hides the fact "Gift cards are sold at the counter" in a long page of filler:

The fact alone: 0.730 for "Do you sell gift cards?".
At the START of a long text: 0.519. Found, but diluted by everything around it.
At the END of a long text: 0.323 - exactly the same as the long text with NO fact. Everything after token 2,048 was cut off, without any error.
This is why real documents are split into chunks before embedding (Module 5): small enough to fit, focused enough to match.

Example 2 - too_long.py

import ollama

QUESTION = "Do you sell gift cards?"
FACT = "Gift cards are sold at the counter for Rs 500 and Rs 1000."
FILLER = "Our cakes are baked fresh every morning with butter, flour and sugar from local farms. " * 150


def embed(text):
    reply = ollama.embed(model="nomic-embed-text", input=text)
    return reply.embeddings[0], reply.prompt_eval_count


q, _ = embed(QUESTION)
for label, text in [("fact alone", FACT),
                    ("fact at the START of a long text", FACT + " " + FILLER),
                    ("fact at the END of a long text", FILLER + FACT),
                    ("long text with NO fact", FILLER)]:
    vector, tokens = embed(text)
    print(f"{label:<34} tokens counted: {tokens:>5}   similarity: {sum(a * b for a, b in zip(q, vector)):.3f}")

Output

fact alone                         tokens counted:    16   similarity: 0.730
fact at the START of a long text   tokens counted:  2048   similarity: 0.519
fact at the END of a long text     tokens counted:  2048   similarity: 0.323
long text with NO fact             tokens counted:  2048   similarity: 0.323

Example 3 - the model’s languages

Telugu "కేక్" was one token. telugu.py compares two unrelated Telugu words, "cake" and "house":

Similarity 1.000 - the model gave both words the SAME vector. Its vocabulary is English; every Telugu word became the same "unknown" token.
In English, "cake" vs "house" scored 0.415 - different, as expected.
So a Telugu question would match every Telugu document equally. Search would be random.
For other languages, choose a multilingual embedding model - and test it like this before you trust it.

Example 3 - telugu.py

import ollama

pairs = [("కేక్", "ఇల్లు"),          # "cake" and "house" in Telugu
         ("cake", "house")]
for a, b in pairs:
    va, vb = ollama.embed(model="nomic-embed-text", input=[a, b]).embeddings
    print(f"{a} vs {b}: similarity {sum(x * y for x, y in zip(va, vb)):.3f}")

Output

కేక్ vs ఇల్లు: similarity 1.000
cake vs house: similarity 0.415

Preprocessing at a glance

Count tokens

Includes 2 extra.

ollama.embed(model=..., input=text).prompt_eval_count

Drop tags

And scripts first.

re.sub(r"<[^>]+>", " ", text)

Fix entities

  -> space.

html.unescape(text)

One space

Tidy whitespace.

re.sub(r"\s+", " ", text).strip()

Length limit

nomic-embed-text.

2,048 tokens - the rest is cut silently

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Your own page

“Copy the HTML of any web page with a policy, and compare the similarity before and after clean().”

Find the limit

“Change the filler count in too_long.py until the fact at the end starts to count again.”

Hindi or Tamil

“Run telugu.py with two words in another language you know. Same vector?”

A cookie notice

“Add "We use cookies to improve your experience. Accept all" to the clean text. How much does the score drop?”

What usually goes wrong

Embedding scraped HTML as it is

Tags and menus lowered the match from 0.847 to 0.758.

Embedding whole long documents

Past 2,048 tokens the text is cut silently - the fact at the end was lost.

Assuming every model reads your language

Two Telugu words got the same vector.

Cleaning what does not matter

Spaces and capitals changed nothing for this model. Measure before you add steps.

Practice

Write these yourself before opening anything. Getting them wrong first is most of how this sticks.

Write check_text(text) that prints warnings before embedding: more than 2,000 tokens ("will be cut"), any "<" tag left, more than 20% of characters outside the English alphabet ("check the model’s languages"), and lines that look like menus. Run it on the five versions in messy.py and on the long text in too_long.py.

Show hint

Use prompt_eval_count for tokens; for letters, count characters with ch.isascii().

Key points

Everything you send becomes tokens and moves the vector.
Remove tags, scripts, entities and boilerplate before embedding.
nomic-embed-text reads at most 2,048 tokens; the rest is dropped without an error.
Long texts dilute a fact even when it fits - split documents into chunks.
Check that the embedding model supports your language.

Quick check before you move on

What did HTML leftovers do to the similarity?

Lowered it from 0.847 to 0.758.

What score did a fact at the end of a long text get?

0.323 - the same as no fact at all.

What did two unrelated Telugu words score?

1.000 - the same vector.

Interview questions

How do you prepare text for embedding?

Extract the main content (strip markup, boilerplate, navigation), normalise whitespace and entities, keep useful structure like headings, chunk to the model’s token limit, and validate language support and token counts.

What happens if text exceeds an embedding model’s context length?

Depending on the API it errors or truncates; silent truncation loses information, so chunk documents and monitor token counts.

Quiz

1.
Why did capitals change nothing here?
2.
Why is a fact at the start of a long text weaker than alone?
3.
What happens to text past the length limit in Ollama’s embed?

Comments

Loading comments...