0:00This video is about the middle of an entity resolution pipeline. Several systems hold a record for one company, Acme. We need to decide which pairs of records are worth comparing, and how much evidence each pair carries.
0:14Blocking is how you avoid comparing every pair. You pick a cheap key, and you only compare records that share it.
0:21It runs straight after normalization, on the comparison values rather than the raw ones.
0:27Here the key is the normalized phone number.
0:29Three of our records carry it. Those three go into one block, and they're the only records compared with each other.
0:35That's three pairs instead of the six that four records would otherwise need.
0:40Real pipelines run several passes and take the union. So add a second pass on the first few letters of the normalized name.
0:48And the union comes to six, which is every pair there was, because all four names normalize to the same value. On a table this small the second pass buys nothing at all. So let's make the table bigger.
1:03Four records is too small to show what blocking is actually for. So here's a slightly bigger set, with one deliberate change.
1:11Support's phone number is gone. Same record, same key, one field removed, and nothing else about the original four changes.
1:19Then two records for a second company, Beta, from the same two systems. Both of them normalize to the same name and the same phone.
1:28Six records, which means fifteen pairs if you compare everything.
1:32Now run the phone pass on its own.
1:34It admits two pairs out of fifteen. C-R-M with billing, and Beta with Beta. That is a real saving.
1:41But support never got a phone key, so it isn't in any block at all. Both of the true matches involving it are gone before scoring even starts, and no scorer, however good, can recover a pair it never sees.
1:56Now put the second pass back in, the one on the first few letters of the normalized name.
2:01Block runs both passes and takes the union of whatever they propose.
2:05Seven candidate pairs now, out of fifteen. Still well under comparing everything, but more work than the phone pass on its own.
2:13And this is what the extra work bought. Support is back, paired with both of the records it really belongs with. The name pass found it because its name still normalizes to acme, even with no phone at all.
2:26Three of those seven pairs are Acme lookalikes that will turn out not to match.
2:32That's the trade. Every extra pass recovers matches you would have missed, and hands the scorer more pairs to throw away.
2:39That stress test is finished.
2:41Everything from here runs on the four records we started with.
2:45Blocking has handed us candidate pairs. Scoring decides how much evidence each pair actually has.
2:50Take this pair, C-R-M and billing. It came out of the block, and nothing has been decided about it yet.
2:56The scorer compares three fields and gives each one a weight. Name similarity counts for half, an exact email for three tenths, an exact phone for two tenths.
3:07This pair agrees on all three, so it scores one. Worth noticing that the name comparison here came out exact after normalization, even though the scorer is perfectly willing to accept a fuzzy name match.
3:19And those weights are illustrative. They're numbers somebody chose, so what comes out is a match score, not a probability that the two records are the same company.
3:31That pair had every field agreeing. Most pairs don't.
3:35Here's the one we just scored, with its contributions written out beside it.
3:39Now support. Its name normalizes to the same value and its phone agrees exactly, so those two signals contribute in full.
3:48Its email is missing, though, so that signal contributes nothing. This scorer doesn't spread the missing weight across the other two, which means less available evidence simply gives you a lower total.
4:00And the lookalike, C-R-M one oh two. The name agrees, so half a point.
4:05Its email and its phone both disagree outright. That's a different situation from support's. The evidence isn't absent here, it's pointing the other way.
4:16And agreement on a name alone is weak evidence, because a name is the one field two unrelated companies can easily share.
4:25One more thing about name similarity, because the weight we gave it treats every name the same.
4:31Two records both say Smith. That agreement is real, but thousands of unrelated people share that surname, so it barely narrows anything down.
4:40Two records both say shim bor ska. The same kind of agreement, on a name almost nobody else has. That makes it far more likely these two records are about one person.
4:51Richer scorers measure how often each value occurs, and weigh a rare agreement higher than a common one.
4:57Ours doesn't. It gives name similarity half a point whatever the name is. That's fine for a worked example, and worth knowing about before you ship one.
5:06Which raises the obvious question about embeddings. They do earn a place here, and it helps to be precise about which one.
5:13Character comparison struggles here. International Business Machines and I-B-M share almost no letters in the same order. An embedding can still put them close together, because it works from meaning rather than spelling.
5:27But it puts these two close together as well. A U-K company and a German company, separately registered, genuinely related, and not the same legal entity.
5:39So this pair is a non-match. Similarity caught that the two names are about related things. It didn't catch the boundary we actually care about, which is legal identity.
5:52Which is why similarity belongs inside the pipeline rather than instead of it.
5:57Use vector retrieval as another candidate pass, alongside the blocking keys, to propose pairs that exact keys would miss entirely.
6:05And use the similarity value as one signal when you score. The decision still comes from the combined evidence and the rules you set, not from a distance on its own.
6:16So here's where that leaves you.
6:18You can cut the comparisons down with blocking, and run more than one pass so a missing key doesn't lose you a true match.
6:25And you can turn the evidence in a pair into one score, where a missing field counts for nothing and a disagreeing field counts against.
6:34One thing to carry out of this. The score measures how much evidence a pair has, and the decision about identity is still to be made.

Entity Resolution: Blocking and Scoring

Created by Abhyuday DesaiPublished 3 Oct, v1Runs 6 min 48 sec

0 views

About this video

The middle of an entity resolution pipeline. Blocking picks a cheap key and compares only records that share it; a stress test shows what a single pass silently loses and a second pass recovers. Scoring then combines several fields with weights, treats a missing field differently from a disagreeing one, and gives rare agreement more credit than common agreement. Embeddings earn a place as a candidate pass and a signal, not as the decision.

You'll learn how to:

Block on a cheap key and take the union of several passes
Measure what blocking misses, since no scorer can recover a pair it never sees
Combine evidence into a match score, and read that score as evidence rather than probability
Use vector similarity inside the pipeline without letting it decide identity

For developers building or tuning a matching pipeline.

What changed

v103 Oct 2026abhydesai6 min 48 sec21ba95b

First published version.

0 views0 likes on this version