0:00This video is about entity resolution, which is deciding that several records in different systems describe one real thing. We'll see why ordinary deduplication can't do that, and the two cheapest steps that start to.
0:14Let's start with where duplicate records come from in the first place.
0:18A C-R-M holds this record for a company.
0:21That key identifies a roe inside the C-R-M, and nothing outside it. No other system agreed to use it.
0:27And the name is whatever the person entering it typed that day. Separate systems with separate histories rarely leave you one shared key.
0:35Billing has its own record for the same company.
0:38It spells the name out in full, in capitals. Nobody coordinated that, and nobody was ever going to.
0:45The phone number is the same digits in a different format. A join on either of those fields finds nothing.
0:51Support has a third record, with a third spelling.
0:55It also has no email. A field that's missing is not a field that disagrees, and later on that difference decides an outcome.
1:04All three of these describe one company. When a system keeps them as separate entities, that's splitting, and everything you count downstream is counting the same thing more than once.
1:18So how far does ordinary deduplication actually get us?
1:22Four records go into a table called customers, the three we've seen plus one more, and we ask for the distinct names.
1:28Three come back. Two of those four records stored the identical string, so they folded into one.
1:35That's exact deduplication. It removes equal values, and the three spellings of one company survive it, because as strings they genuinely are different.
1:47Now the fourth record. This is the C-R-M record we started with.
1:51And this is the other one. Same system, different key.
1:54The names are identical, character for character. Exact matching would put these two together without hesitating.
2:02The contact details disagree completely, though. Different email domain, different phone number. In this example they're two different companies.
2:10So there are two ways to get identity wrong, and they pull in opposite directions. Splitting scatters one real thing across several identities. Incorrect merging collapses two real things into one.
2:24So we need something better than comparing strings. Matching works pair by pair: take two records, compare their evidence, and decide.
2:32So here's one pair: the C-R-M record, and one candidate for it over in billing. Nothing has been scored yet, and nothing has been decided.
2:42The trouble is how many pairs there are. For n records, the number of unordered pairs is n times n minus one, over two.
2:49Put a million records in, which is a small customer table by most standards, and that's nearly five hundred billion comparisons.
2:57Nobody runs that. So before any detailed comparison happens, something has to cut the pairs down to the ones that were ever plausible.
3:07Cutting the pairs down is called blocking, and it compares cleaned-up values rather than raw ones. So the first stage is normalization, and it's the cheapest win in the whole pipeline.
3:18These are the three spellings our systems stored. Case, punctuation, a company suffix, some stray whitespace. None of that says anything about which company this is.
3:29So normalize strips it, and all three land on the same comparison value. The records themselves are untouched, which matters later when you have to explain a decision.
3:39The cleanup does have to fit the language and the domain. A rule that strips Latin accents helps with some names and quietly destroys names written in other scripts.
3:51Phone numbers get the same treatment. Two formats, one string of digits, and now two records agree on a field that looked different a moment ago.
4:01Ten digits is a United States convention. Country context decides what normalizing a phone number even means, so this is a rule you write per domain, not once.
4:13Now that the values are comparable, some fields turn out to be strong enough to decide a pair on their own. Worth using those before anything expensive runs.
4:22Here are our three records again, shown as the systems stored them.
4:26And here's a deterministic rule. If two records share an exact email address, call them a match.
4:32C-R-M and billing do. The same mailbox, character for character, sitting in two systems that never talked to each other.
4:39So that's a match, decided with no scoring at all. A verified tax I-D behaves the same way, and so does a person's normalized name together with a birth date.
4:50Support has no email, though.
4:52So the rule says nothing about it. That isn't a disagreement and it isn't evidence against a match. It's silence, and the pipeline has to carry it forward as silence rather than as a no.
5:06So here's where that leaves you.
5:08You can name the two ways identity goes wrong. Splitting scatters one real thing across several records, and an incorrect merge collapses two real things into one.
5:20You can also take the first two steps of a resolution pipeline. Normalize the values so comparisons are fair, and let a field like an exact email decide a pair when it can.
5:32One thing to carry out of this. Matching strings is not the same as matching things, and a missing value is silence rather than a no.

Entity Resolution: Why Deduplication Isn't Enough

Created by Abhyuday DesaiUpdated 4 Oct, v2Runs 5 min 46 sec

1 view

About this video

Entity resolution is deciding that several records in different systems describe one real thing. This video starts where duplicates come from, shows why SELECT DISTINCT removes equal strings and nothing else, names the two failures that pull in opposite directions, and takes the first two steps of the pipeline: normalize the values, then let exact evidence decide a pair when it can.

You'll learn how to:

Recognize splitting and incorrect merging as the two ways identity goes wrong
See why comparing every pair of records is impossible at scale
Normalize names and phone numbers for comparison while keeping the originals
Use deterministic rules on exact evidence, and carry a missing value forward as silence rather than a no

For developers who reached for a name match and found it insufficient.

What changed

v204 Oct 2026abhydesai5 min 46 secba24b25

Re-recorded from the same source. The first take sat on the closing card for half a minute after the narration ended; this one ends when the episode does.

Same chapters and transcript. 28 sec shorter.

0 views0 likes on this version

v103 Oct 2026abhydesai6 min 14 sec21ba95b

First published version.

1 view0 likes on this version

Watch v1