0:00This video is about the second half of entity resolution. Three records for one company, Acme, have been scored against each other. Now a number has to become a decision, and the decision has to become an identity that lasts.
0:14A score on its own doesn't do anything. Something has to turn it into an action.
0:20That's the threshold stage. It takes a number and returns one of three decisions.
0:26Here's the range our scores live in, zero to one.
0:29And here are two cut points, at point six and point eight five. Both of them are choices rather than measurements, which is why they're labelled illustrative.
0:39They divide the range into three. Below point six, reject automatically. Point eight five and above, merge automatically. Anything in between goes to a person.
0:51Our billing pair scored one, so it merges.
0:54Support scored point seven, which lands in the review band. The missing email is what put it there, and the pipeline doesn't get to settle this one on its own.
1:03And a lookalike, a different company whose name also normalizes to acme, scored point five. That's below the lower cut, so it's rejected.
1:13Notice that exactly point six belongs to review rather than to reject. Somebody has to write down which side of each boundary the boundary itself falls on.
1:24Those two cut points were just numbers a moment ago. Choosing them is where the real work is.
1:31The scale is the same one. What changes is where you put the lines.
1:36Move the lower line up and you reject more pairs automatically, including some true matches.
1:42Move the upper line down and you merge more pairs automatically, including some wrong ones.
1:48A model can tell you how the scores are distributed. It can't tell you which of those two mistakes you'd rather make.
1:54That depends entirely on what the records are. Missing a duplicate in a mailing list costs you a second envelope.
1:59Merging two patients' histories into one can hurt somebody. Same pipeline, tuned completely differently.
2:06And to tune it, you have to measure it. Score labelled pairs where you already know the answer.
2:12Then check the clusters those decisions produce, because pairwise accuracy can still assemble nonsense groups.
2:20Measure blocking recall as well. Pairs that never reached the scorer don't appear in any scoring metric, and no threshold you pick will bring them back.
2:30Matching is done. Now the second half of the job: assigning an identity, and keeping it.
2:35Our three records, back where they started.
2:37Support's pair scored point seven, which put it in the review band. So before anything gets canonicalized, that review has to actually happen.
2:46In this example a reviewer looks at it and confirms that support belongs with the other two. Point seven never authorized that on its own. A person did.
2:57Now the canonical entity exists. One node, one identifier, standing for all three records at once.
3:04The worked example calls it ent oh oh oh one. In the graph it's org acme.
3:09Those are two names for one entity in two different namespaces, and the mapping between them is written down rather than guessed from the name.
3:19The entity has an identity now, but it has no values yet. Survivorship decides which source supplies each one.
3:27For the stored name, this pipeline prefers C-R-M. That's a rule somebody wrote down: most trusted source, or most recent, or most complete, depending on the field.
3:38So the canonical record's name field becomes Acme Corp, the exact string C-R-M stored.
3:44The email is a separate decision, and billing wins that one.
3:48So the entity now carries a name from one system and an email from another. That's what field by field means. Nothing forces a single source to supply everything.
3:58And notice the node is still called Acme. The short display label and the surviving name field are two different things, and they don't have to agree.
4:08The phone is a third decision, and it shows something the first two didn't.
4:13Billing wins the phone field, so the value credited to it is billing's exact string, punctuation and all.
4:20That's what survivorship selected. Not a cleaned up version of it, the thing that was actually stored.
4:27Normalizing happens after the selection rather than instead of it. The canonical store gets the digits.
4:34So the entity's phone field holds the normalized form, and billing's record still reads exactly as it always did.
4:41The lineage keeps both: the source string that was chosen, and the transformation applied to it.
4:47It never pretends billing stored the normalized version. When somebody later asks where a value came from, that's the difference between an answer and a guess.
4:58Assigning an identity isn't the end of it. What gets stored about the decision is what makes it maintainable.
5:04Here are the three records and the entity they resolved to.
5:08Each record keeps a link to that entity. Dashed, because a record supports an identity rather than asserting a fact about it.
5:16The link itself is a roe somewhere. Which record, which entity, which source system, and the score that was computed for it.
5:25And separately, the decision. Which records were compared, what was decided, and on what basis. Reviewer approval is not a score, and storing the score in place of it loses the only part a person actually contributed. A crosswalk on its own is not an audit log.
5:45Now suppose later evidence arrives and support turns out to belong somewhere else. That's a revised decision rather than a deletion.
5:53Support's membership moves to a provisional identity and goes back for review. All three source records are exactly where they were.
6:01Version the memberships, recompute whatever canonical values depended on the one that changed, and you can revise this again tomorrow.
6:09One more safeguard, and this is a different example.
6:12Three records from a registry, all carrying the same name. The middle one has no birth year recorded at all.
6:19The first two look like a match. Same name, and nothing disagrees, because the middle record has nothing to disagree with.
6:25The second and third look like a match for exactly the same reason.
6:30But the first and the third were born twenty years apart. They are not the same person, and nothing ever compared them directly.
6:38Real identity is transitive. If A is B and B is C, then A is C.
6:42Approximate pairwise evidence isn't, and a system that closes those chains automatically will build clusters out of records that contradict each other.
7:23So here's where that leaves you.
7:25You can turn a score into an action with two cut points, and you know those cut points follow from what a wrong merge and a missed match each cost you.
7:34And you can keep the identity once you've assigned it. Choose the surviving values field by field, keep the links back to every source record, and keep every decision revisable.
7:47One thing to carry out of this. Identity is a decision, not a fact the data hands you. Keep the evidence, and you can always decide again.

Entity Resolution: From Score to Identity

Created by Abhyuday DesaiPublished 3 Oct, v1Runs 8 min

0 views

About this video

The second half of entity resolution. Two cut points divide the score scale into reject, review and merge, and choosing them depends on which error costs more. A reviewer confirms the pair in the middle band, a canonical entity is assigned, and its values are chosen field by field from the source records, selecting the raw value and then normalizing. Every decision is stored with its provenance so a merge can be revised later, and a final example shows why approximate matches are not transitive.

You'll learn how to:

Set and justify two thresholds by what a wrong merge and a missed match each cost
Assign a canonical identity and keep the mapping to each system's local id explicit
Choose surviving values field by field, with lineage that never pretends a source stored the cleaned value
Keep merges reversible, and avoid building clusters from chains of pairwise evidence

For developers who own the identity layer under a graph or a warehouse.

What changed

v103 Oct 2026abhydesai8 min21ba95b

First published version.

0 views0 likes on this version