Semantic Search vs Keyword Search
Semantic search should beat keyword search - and on 14 bakery questions it LOST, 7 to 9. Every question said "Sri Lakshmi Bakery on MG Road, Hyderabad", and that pulled them all to the one document about the bakery itself. Removing the shop name brought it to 11 of 12. It still missed "What costs Rs 45?", and it never says "nothing found" - a score cut-off had a gap of only 0.014.
What you will be able to do
- Compare keyword and semantic search on the same questions
- Explain where each one wins
- Find why semantic search can fail on repeated words
- Test fixes by measuring, not guessing
- Understand why "nothing found" is hard with semantic search
The idea, in plain English
Keyword search matches exact words. It is precise with names, numbers and codes, and it can say "nothing matched". It fails when people use other words - "return" for "refund", "egg" for "eggless".
Semantic search matches meaning through embeddings. It handles other words easily - but every question gets a nearest document, even when none is relevant, and exact numbers matter less to it. And as we found, words that appear in EVERY question can pull all of them in one direction.
We test both on 14 questions with known answers: the ten from Lesson 1.4 plus four in other words. Then we measure three ways to fix what broke. nomic-embed-text with its search prefixes.
Worked example: Lesson 1.4’s questions plus four paraphrases, searched both ways - then three fixes tested side by side.
1 - Keyword search
Right for names and numbers, wrong for paraphrases.
Right document found first. Real results on 14 questions.
Words you will see in this lesson
A few search words.
Keyword searchSearch by exact shared words (also called lexical search).Semantic searchSearch by meaning, using embeddings (also called dense search).ParaphraseThe same meaning in different words.ThresholdA cut-off score: below it, treat the result as "nothing found".Hybrid searchKeyword and semantic search combined (Module 8).An everyday example: two shop assistants
- Assistant K only listens for exact product names. Ask for "item 45" and he finds it at once; ask for "something to give my mother on her birthday" and he is lost.
- Assistant S understands what you mean. "A birthday gift for my mother" - she shows you cakes and hampers. But ask for "the Rs 45 one" and she shows you something "similar".
- And if every customer starts with "Hello, Sri Lakshmi Bakery!", assistant S may hear only that and walk everyone to the shop’s history wall.
- A good shop uses both assistants.
Example 1 - 14 questions, both searches
compare_search.py has 14 questions, each with the document that holds its answer (None for the two the documents do not cover):
- keyword_search: the document with the most shared words (the Lesson 1.4 search); no shared word means None.
- semantic_search: the document with the highest similarity, using the search prefixes.
- For each question it prints ok or X for both, and the semantic score.
import re
import ollama
from bakery import DOCS
STOP = {"a", "an", "the", "is", "are", "i", "my", "can", "do", "does", "you", "for", "to", "of", "in",
"on", "it", "what", "if", "get", "be", "how", "who", "which", "much", "many", "must", "at",
"sri", "lakshmi", "bakery", "mg", "road", "hyderabad", "have", "did", "time", "year", "far"}
# question -> the document that holds the answer (None = no document does)
TESTS = {
"What is the phone number of Sri Lakshmi Bakery on MG Road, Hyderabad?": "contact.md",
"Who owns Sri Lakshmi Bakery on MG Road, Hyderabad?": "contact.md",
"In which year did Sri Lakshmi Bakery on MG Road, Hyderabad open?": "history.md",
"What time does Sri Lakshmi Bakery on MG Road close on Sunday?": "hours.md",
"How much does a 1 kg black forest cake cost at Sri Lakshmi Bakery, MG Road?": "menu.md",
"What is the best seller at Sri Lakshmi Bakery, MG Road?": "menu.md",
"How far does Sri Lakshmi Bakery on MG Road deliver?": "delivery.md",
"How many days ahead must I order an eggless cake at Sri Lakshmi Bakery, MG Road?": "allergens.md",
"Can I return my cake?": "refunds.md",
"Is there a charge for bringing the order to my house?": "delivery.md",
"Do you make cakes without egg?": "allergens.md",
"What costs Rs 45?": "menu.md",
"Does Sri Lakshmi Bakery on MG Road sell gift cards?": None,
"Does Sri Lakshmi Bakery on MG Road have parking?": None,
}
def words(text):
return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}
def keyword_search(question):
score, name = max((len(words(question) & words(text)), name) for name, text in DOCS.items())
return name if score > 0 else None
def main():
names = list(DOCS)
doc_vectors = ollama.embed(model="nomic-embed-text", input=["search_document: " + DOCS[n] for n in names]).embeddings
def semantic_search(question):
q = ollama.embed(model="nomic-embed-text", input="search_query: " + question).embeddings[0]
return max((sum(a * b for a, b in zip(q, d)), n) for d, n in zip(doc_vectors, names)) # (score, name)
right = {"keyword": 0, "semantic": 0}
print(f"{'question':<52}{'keyword':<16}{'semantic':<15}score")
for question, expected in TESTS.items():
kw = keyword_search(question)
score, sem = semantic_search(question)
right["keyword"] += kw == expected
right["semantic"] += sem == expected
mark = lambda found: "ok" if found == expected else "X "
print(f"{question[:50]:<52}{mark(kw)} {str(kw):<13}{mark(sem)} {sem:<12} {score:.3f}")
print(f"\nright: keyword {right['keyword']}/{len(TESTS)}, semantic {right['semantic']}/{len(TESTS)}")
if __name__ == "__main__": # fixes.py imports TESTS without running this
main()question keyword semantic score
What is the phone number of Sri Lakshmi Bakery on ok contact.md X history.md 0.773
Who owns Sri Lakshmi Bakery on MG Road, Hyderabad? X None X history.md 0.763
In which year did Sri Lakshmi Bakery on MG Road, H X None ok history.md 0.873
What time does Sri Lakshmi Bakery on MG Road close ok hours.md X history.md 0.720
How much does a 1 kg black forest cake cost at Sri ok menu.md ok menu.md 0.750
What is the best seller at Sri Lakshmi Bakery, MG ok menu.md X history.md 0.689
How far does Sri Lakshmi Bakery on MG Road deliver ok delivery.md ok delivery.md 0.761
How many days ahead must I order an eggless cake a ok allergens.md ok allergens.md 0.694
Can I return my cake? X menu.md ok refunds.md 0.794
Is there a charge for bringing the order to my hou X None ok delivery.md 0.576
Do you make cakes without egg? X refunds.md ok allergens.md 0.780
What costs Rs 45? ok menu.md X delivery.md 0.642
Does Sri Lakshmi Bakery on MG Road sell gift cards ok None X history.md 0.734
Does Sri Lakshmi Bakery on MG Road have parking? ok None X history.md 0.727
right: keyword 9/14, semantic 7/14Reading the result: semantic search lost
- Semantic search won the paraphrases: "Can I return my cake?", "a charge for bringing the order to my house", "cakes without egg" - all three missed by keyword search.
- But it chose history.md for SEVEN questions - right only for the opening year, wrong for phone number, owner, Sunday hours, best seller, gift cards and parking. history.md says "the bakery opened in 1998 on MG Road, Hyderabad".
- Why: those questions all contain "Sri Lakshmi Bakery on MG Road, Hyderabad". In a short question, the shop name is most of the meaning - and history.md is the document most "about" the shop and its location.
- Keyword search did not have this problem: our STOP list removed the shop name.
- "What costs Rs 45?" - keyword search found the exact "45" in the menu; semantic search chose delivery.md (it mentions money too).
- Gift cards and parking: keyword search said None (right). Semantic search ALWAYS returns a nearest document - here history.md with 0.734.
Example 2 - three fixes, measured
fixes.py runs the 12 answerable questions three ways and also prints scores, to see whether a cut-off could catch the two unanswerable ones:
- 1. As before: 7 of 12.
- 2. Shop name removed from the questions: 11 of 12. Only "What costs Rs 45?" still missed.
- 3. Shop name added to every document instead: 10 of 12 - better, but not as good as fix 2.
- Fix 2 is query cleaning: if every question is about the same shop, the shop name carries no information for search - remove it before embedding.
import re
import ollama
from bakery import DOCS
from compare_search import TESTS
names = list(DOCS)
SHOP = re.compile(r"(at |of |does )?sri lakshmi bakery(,| on)? mg road(, hyderabad)?", re.I)
def run(label, doc_texts, clean_question):
vectors = ollama.embed(model="nomic-embed-text", input=["search_document: " + t for t in doc_texts]).embeddings
right, wrong_answerable = 0, []
unanswerable, right_scores = [], []
for question, expected in TESTS.items():
q_text = clean_question(question)
q = ollama.embed(model="nomic-embed-text", input="search_query: " + q_text).embeddings[0]
score, found = max((sum(a * b for a, b in zip(q, d)), n) for d, n in zip(vectors, names))
if expected is None:
unanswerable.append(f"{score:.3f}")
else:
right += found == expected
if found == expected:
right_scores.append(score)
if found != expected:
wrong_answerable.append(q_text[:28])
print(f"{label:<44} right {right}/12 still wrong: {wrong_answerable}")
print(f"{'':<44} lowest score of a right answer: {min(right_scores):.3f} "
f"best score for gift cards / parking: {unanswerable}")
plain_docs = [DOCS[n] for n in names]
run("1. as before", plain_docs, lambda q: q)
run("2. shop name removed from questions", plain_docs, lambda q: SHOP.sub("", q).replace(" ", " "))
run("3. shop name added to every document", [f"Sri Lakshmi Bakery, MG Road, Hyderabad. {t}" for t in plain_docs], lambda q: q)1. as before right 7/12 still wrong: ['What is the phone number of ', 'Who owns Sri Lakshmi Bakery ', 'What time does Sri Lakshmi B', 'What is the best seller at S', 'What costs Rs 45?']
lowest score of a right answer: 0.576 best score for gift cards / parking: ['0.734', '0.727']
2. shop name removed from questions right 11/12 still wrong: ['What costs Rs 45?']
lowest score of a right answer: 0.569 best score for gift cards / parking: ['0.555', '0.498']
3. shop name added to every document right 10/12 still wrong: ['Who owns Sri Lakshmi Bakery ', 'What is the best seller at S']
lowest score of a right answer: 0.560 best score for gift cards / parking: ['0.839', '0.838']Why "nothing found" is hard
A threshold would say: "best score below X - answer I don’t know". Look at the numbers:
- Fix 1: a right answer scored as low as 0.576, but gift cards scored 0.734. No cut-off can separate them.
- Fix 3: right answers from 0.560, gift cards 0.839. Impossible again.
- Fix 2: right answers from 0.569, gift cards 0.555 and parking 0.498. A cut-off at about 0.56 would work - with a gap of only 0.014. One new question could break it.
- So with semantic search, "nothing found" needs more than one fixed number: a re-ranking step, a check by the chat model, or keyword evidence - Module 8.
When to use which
- Keyword search: exact names, prices, product codes, error codes, IDs - and when you need a clear "no match".
- Semantic search: questions in the user’s own words, synonyms, paraphrases.
- Real systems combine both - hybrid search, Module 8.
- Always measure on your own questions. Our "obvious" winner lost until we found the cause.
Keyword vs semantic at a glance
KeywordExact words.
9 of 14 here; found "Rs 45"; missed paraphrases
SemanticMeaning.
7 of 14; 11 of 12 after cleaning the questions
Query cleaningRemove words every question shares.
SHOP.sub("", question)"Nothing found"Easy for keyword, hard for semantic.
threshold gap: 0.014
Try it yourself
The code does not change. Swap the content string and the program does something else entirely.
“Add five questions in your own words to TESTS. Which search wins them?”
“Add a document "Order code CAKE-2041 is the Diwali hamper" and ask "What is CAKE-2041?" both ways.”
“Use the keyword result when it found something, otherwise the semantic one. How many of 14 now?”
“With fix 2, set a cut-off of 0.56. Which questions turn into "I don’t know"?”
What usually goes wrong
It lost 7 to 9 until we cleaned the questions.
The shop name pulled seven questions to history.md.
The gap was 0.014 in the best case and negative in the others.
"What costs Rs 45?" went to delivery.md.
Practice
Write these yourself before opening anything. Getting them wrong first is most of how this sticks.
Build a first hybrid search: give each document a keyword score (shared words divided by question words) and a semantic score, and rank by 0.5 * keyword + 0.5 * semantic. Run it on the 14 questions with the shop name removed. Report how many are right, and whether "What costs Rs 45?" is fixed. Then try weights 0.3 / 0.7.
Show hintHide hint
Normalise the keyword score to 0-1 so it can be added to the similarity.
Key points
- Keyword search matches exact words; semantic search matches meaning.
- Semantic search won every paraphrase; keyword search won the exact price.
- Words repeated in every question can pull semantic search the wrong way - remove them.
- Measured: keyword 9/14, semantic 7/14, semantic after cleaning 11/12.
- Semantic search always returns something; score cut-offs are fragile.
- Combine both (hybrid search) and measure on your own questions.
Quick check before you move on
Interview questions
Compare keyword (BM25) and dense semantic retrieval.
Keyword retrieval excels at exact terms, identifiers and rare words, is cheap and explainable, but misses synonyms; dense retrieval captures paraphrases and meaning but can miss exact matches, is sensitive to query noise and always returns neighbours. Hybrid search with re-ranking combines their strengths.
How do you decide that no document is relevant in semantic search?
Raw similarity thresholds are unreliable; use calibrated re-ranker scores, keyword evidence, an LLM relevance check, or evaluation-tuned thresholds per collection, and design the answer step to say "I don’t know".
Quiz
- 1.
Why did the shop name pull questions to history.md?
- 2.
Why can keyword search say "nothing found" easily?
- 3.
Why did a threshold not work in fix 1?
Comments
Sign in to leave a comment. Your name and photo come from Google; nothing else is shared.
Loading comments...