0:00Last time, we cut documents into chunks, the passages a search ranks against a question.
0:06There's more than one way to decide which passages match. We'll try two, and then combine them.
0:11First, keyword search, which scores a passage by the words it shares with the question.
0:17Then semantic search, which compares embeddings, so a passage can match a question that uses different words.
0:23Next, a question that names an error code and also says what we're trying to do.
0:28Then we merge the two rankings into one, using each passage's position instead of its score.
0:35Finally, how to choose between the methods, and how to check whether combining them helped. Let's start with the script.
0:42This script searches six short passages from some API documentation.
0:47Two are about rate limiting, then caching, a timeout setting, token expiry and batching. The output numbers them P1 to P6, in this order.
0:58For keyword search, each passage is lowercased and split on whitespace into tokens, and those tokens build the index. So API_TIMEOUT stays one token, and a query that types it as two words won't match it.
1:12The keyword scores come from BM25. A passage earns score for each query word it contains, and rare words count for more than common ones.
1:22Both searches end with the same function, a little further down.
1:26It sorts the passages by score, highest first, and keeps the top k, which is three by default.
1:33Let's start with a query that names an exact error code.
1:37That's the status code a server returns when a client sends requests too fast.
1:43Each row is a rank, a passage, its score, and the passage's first words. Only P1 scores above zero, and it's the passage that explains this status code.
1:54It contains every word of the query, so for a query like this, matching words works well.
2:00The other two rows score zero. The function always returns three, so these are just the first two of five passages tied at zero, not the second and third best.
2:12Now a question in plain words, with no code in it.
2:16We ask keyword search how to make our application run faster.
2:20Again only one passage scores above zero. It's P4, about the timeout setting, which doesn't answer the question.
2:28The only word it shares with the question is how. No other passage contains that word, so BM25 gives it some weight.
2:35Let's look back at the passage list for the one that does answer it.
2:39It's P3, about caching frequent lookups to cut response latency. It shares no words with the question, so keyword search scored it zero and didn't return it.
2:51This problem has a name, vocabulary mismatch.
2:54A user describes a need one way, the document describes it another way, and keyword search has no tokens to match.
3:01Here, the user asks about speed, and the passage talks about latency and caching.
3:07Semantic search is meant for this. It compares embeddings, which capture what a text is about rather than its exact words.
3:16The semantic half of the script starts with a small embedding model.
3:20It encodes all six passages once, up front. The vectors are normalized to length one, so a dot product between two of them is their cosine similarity.
3:30At search time, the query is encoded with the same model, and one matrix product scores it against all six passages.
3:38The same top k function keeps the best three. Let's put the speed question to it.
3:43The keyword ranking for this question is still above. Now the same question goes to semantic search.
3:49This time three passages come back with close scores, in a different order.
3:53First is P6, on batching several operations into one call, which is also a real way to make an application faster.
4:01Second is P3, the caching passage that keyword search scored zero.
4:06Matching on meaning found the answer here. Does it still find an exact code, though?
4:12We search by meaning for just the code, and ask for all six passages.
4:17P1, the passage with that code, comes first, well ahead of the rest.
4:21Semantic search has no rule that an exact token must match, so this result isn't guaranteed. On this run, the passage with the code still came first.
4:33Real questions often need both kinds of match. The next one names the error code and also says what we're trying to do.
4:41It asks how to stop getting that error when our calls go out too fast. Keyword search goes first.
4:50All three passages score above zero this time.
4:53P1 explains the error, and P2, the backoff advice, says what to do about it. An answer needs both, and keyword search ranked them first and second.
5:03Now we send the same question to semantic search, which matches on meaning instead of shared words.
5:10Semantic search puts the same two passages first and second. Only third place differs.
5:16To use both searches, we need one ranking, and averaging their scores won't work. A BM25 score has no upper limit.
5:23A cosine similarity stays between minus one and one. In an average, the larger scale would decide the order.
5:31So we combine the passages' positions instead of their scores. The method is called reciprocal rank fusion.
5:37This function takes several rankings and adds up a score for each passage across all of them.
5:43Each ranking gives a passage one over sixty plus its rank. The plus one is there because enumerate counts from zero, and ranks start at one.
5:53The sixty keeps first and second place close together. So in short lists like ours, a passage that both rankings include beats one that only a single ranking put first.
6:04The hybrid search takes the top three from each method, and fuses those two lists.
6:09Let's run it on the mixed question.
6:12The keyword and semantic rankings for this question are above. The hybrid command runs both searches again and fuses their results.
6:20Four passages come back, and each row ends with the search, or searches, that found it.
6:26P1 and P2 were first and second in both rankings, so each collects credit twice, and they stay on top.
6:32P3 came only from keyword search, and P6 only from semantic search. Both were third, so they tie.
6:39For this question, fusion added no passage the answer needs. The keyword ranking above already had both of them at the top.
6:46Fusion also only sees each search's top three. A passage outside both cutoffs gets no credit, however relevant it is. So when is a second search worth having?
6:58It depends on the queries your users actually send.
7:01When queries carry codes, names or identifiers, keyword search matches them directly, so start there.
7:07When users and documents describe the same things in different words, try semantic search, and check it with questions people really ask.
7:16When queries need both, try hybrid search, and ask whether it brings in passages the simpler method missed.
7:23Run that check over many queries, with the number of results you'll actually use. If hybrid keeps finding more, keep it. If not, you save a second index and a second model.
7:35Keyword search matches tokens and semantic search matches meaning, so they can rank the same passage differently.
7:42You can now explain why the two disagree, and merge their rankings without mixing their scores. Then check whether the merge found anything the simpler search missed.
7:53Every search in this video used the question exactly as it was typed. Next, we look at rewriting the question before we search.

Keyword, Semantic and Hybrid Search

Created by Abhyuday DesaiPublished 4 Oct, v1Runs 8 min 8 sec

0 views

About this video

In Episode 03 of Retrieval Explained, we search six passages of API documentation two ways. BM25 keyword search finds the passage on an exact error code, and misses the caching passage when the question asks how to make an application faster, because the two share no words. Semantic search compares embeddings and finds it. For a question that names the error and also says what we are trying to do, we merge both rankings with reciprocal rank fusion, which combines positions instead of scores, and check whether the merge found anything the simpler search missed.

You'll learn how to:

Score passages with BM25, and see how splitting text into tokens decides what can match
Recognize vocabulary mismatch, where a user and a document describe the same need in different words
Search by meaning with normalized embeddings and one matrix product
Merge two rankings with reciprocal rank fusion, and see why averaging BM25 and cosine scores does not work
Choose between keyword, semantic and hybrid search from the queries your users actually send

This video is for developers choosing a search method for a retrieval system. The code runs from the companion repo.

What changed

v104 Oct 2026abhydesai8 min 8 secb9e8014

First published version.

0 views0 likes on this version