0:00Last time, search ranked passages by how close their vectors were to the question's.
0:05Those passages are chunks, pieces cut from longer documents. This video is about where to make the cuts.
0:11First, we cut a document every hundred characters, and one fact ends up split across two chunks.
0:18Then we change the two settings of that splitter, the chunk size and the overlap, and see what each one costs.
0:25Next, we cut at the document's own headings, with a size budget for long sections.
0:30Finally, we give each chunk the document's title, its heading and where it came from. Let's start with the document.
0:39This is the document we'll cut, a short travel policy with a title and three sections, on flights, meals and ground transport.
0:47Our question for this whole video is about the meals section: what is the domestic meal allowance? The answer here is fifty dollars a day.
0:56The simplest way to cut it is every so many characters. Let's open that splitter.
1:02Sizes here are counted in characters, so we can see exactly where each cut lands.
1:07It moves forward by the chunk size minus the overlap, so with no overlap, each chunk starts where the last one ended.
1:15At each step, it slices off one chunk's worth of text.
1:19The size and the overlap come from the command line, and each chunk prints under its range of characters.
1:25Let's run it.
1:26We ask for chunks of one hundred characters.
1:29The policy is four hundred thirty nine characters long, so it comes back as five chunks.
1:34Let's look at chunks one and two.
1:36Chunk one ends at character two hundred, partway through the domestic sentence. It names domestic trips, then stops before the amount.
1:45Chunk two opens with the amount, fifty dollars a day. The word domestic stayed behind in chunk one.
1:51Ask this collection for the domestic allowance, and neither chunk answers on its own. One has the subject, and the other has the amount.
2:01That problem is called fragmentation.
2:04A boundary lands between two pieces of information that only make sense together.
2:09So fitting under the model's input limit is not enough. A useful chunk has to be found by the question, and make sense without its neighbors.
2:17Let's see what a bigger chunk size does.
2:21This time we ask for chunks of a thousand characters, more than the whole policy.
2:26So the policy comes back as one chunk, with nothing cut.
2:30The domestic rule is whole, with the subject and the amount together.
2:35But the flight and transport rules came too. In a real collection, extra text like this can weaken the match, and it takes up room in the prompt.
2:44This is called a mixed chunk. Now let's go back to a hundred characters and add twenty characters of overlap.
2:51Each window is still a hundred characters wide but moves forward only eighty, so neighbors share twenty.
2:58That gives six windows this time.
2:60Let's look at windows two and three.
3:02Window two starts at character one hundred sixty, so it holds the domestic sentence whole. The window just starts in a different place.
3:11But the international rule is cut in both windows. The twenty shared characters are the overlap, and neither window holds the whole rule.
3:19So overlap gives text near a cut a second chance, without guaranteeing a complete idea, and every shared character is stored and ranked twice. So what should these settings be in practice?
3:33Our character counts are toy numbers, chosen to make the cuts visible. Real systems measure chunks in tokens.
3:39A common starting point is a few hundred tokens, somewhere between two hundred and eight hundred.
3:45With ten to fifteen percent of that as overlap, where the cuts are arbitrary.
3:50Then check the chunks against your users' real questions. You want the smallest size that still holds a complete answer.
3:56Before tuning numbers, though, try cutting where the document already has boundaries.
4:03Each section of the policy starts with a second-level heading, a line beginning with two hash marks.
4:09These are the places where the author already separated one topic from the next.
4:14So let's cut right before each one.
4:17This splitter does it with one regular expression.
4:20It splits at every line break followed by two hash marks and a space.
4:24The part in parentheses only checks for the heading, so the heading stays with its section.
4:30Let's run it.
4:31There's no size this time. The headings decide where the cuts go.
4:35Four chunks come back: the title alone, then the three sections, each one whole.
4:40The meals section is one chunk now, with both allowances and the line about alcohol.
4:46That last line matters, because anyone asking about the allowance needs to know it doesn't cover alcohol.
4:52The international allowance came too, which is fine, because it's the same topic.
4:57So a good chunk can hold several facts, as long as they belong to one topic.
5:02When a section runs long, we also need a size budget.
5:07This recursive splitter, from the LangChain text splitters package, cuts at headings and keeps to a size budget.
5:14This list is its order of preference: a heading, then a blank line, then a line break, then a space, and finally anywhere.
5:23The budget is four hundred characters. It cuts at headings first, moves down the list only for pieces that are still too long, then joins small pieces while they fit.
5:34Further down, a second splitter runs on the meals section alone.
5:38Its budget is only a hundred characters.
5:41Let's run it.
5:43It prints each chunk's length and its headings, not the full text.
5:47The whole policy gives two chunks, three hundred thirty three characters and a hundred and three.
5:53The first holds the title, the flight rules and the meals section. They fit the budget together, so the splitter joined them across a heading.
6:01The meals section is a hundred seventy six characters, too long for a hundred. So the splitter falls back to line breaks, and returns three pieces.
6:10So set a budget that can hold a complete answer, put your heading style first in the list, and read what comes out.
6:17Each chunk also needs a record of where it came from.
6:22So this script builds a record for every section, instead of a bare string.
6:27The identifier combines the file name and the heading, and the source and section are kept as fields too.
6:33The text field is the section as written, for display.
6:37The embed text puts the title and heading in front of the body, and that's the version we turn into a vector.
6:43Let's run it.
6:44It prints the identifiers, then the meals record's embed text.
6:49There's one record for each section.
6:51When an answer is wrong, an identifier like this traces it back to the exact section of the document.
6:57Because the title and heading are embedded with the body, they can change what search finds. Stored only as metadata, they wouldn't.
7:05Put the same framing into the model's context too. A title used only for embedding is never seen by the model writing the answer.
7:14Let's put the choices in order.
7:16Begin with the boundaries the document already has.
7:19Choose a budget that holds a complete answer, not the smallest chunk you can make.
7:24Use a little overlap only where cuts fall at arbitrary points.
7:28Give every chunk its document's title, its heading and its source.
7:32Then read a few chunks with nothing around them, and put real user questions through the search.
7:38Every passage a search system returns starts as a decision about where to cut.
7:42You can now see what a cut costs, a fact split in two or unrelated topics mixed in. And you can build chunks that make sense alone and point back to their source.
7:54Next, we compare searching by words with searching by meaning, and then combine the two.

Chunking Documents for Search

Created by Abhyuday DesaiPublished 4 Oct, v1Runs 8 min 6 sec

0 views

About this video

In Episode 02 of Retrieval Explained, we cut one short travel policy into chunks four ways, and ask each version the same question: what is the domestic meal allowance? Cutting every hundred characters splits the answer across two chunks. A larger size keeps it whole but mixes in unrelated rules, and overlap gives text near a cut a second chance without guaranteeing a complete idea. Cutting at the document's own headings keeps each section whole, a recursive splitter adds a size budget for long sections, and each chunk becomes a record with its title, heading and source.

You'll learn how to:

Recognize fragmentation, where a cut separates two facts that only make sense together
See what chunk size and overlap each cost, and where common starting settings sit in tokens
Split at headings with one regular expression, and keep to a size budget with a recursive splitter
Build chunk records with an identifier, a source, and an embed text that puts the title and heading in front of the body
Choose chunking settings in order, and check them against real questions

This video is for developers building a search over their own documents. The code runs from the companion repo.

What changed

v104 Oct 2026abhydesai8 min 6 secb9e8014

First published version.

0 views0 likes on this version