How to Define Truth Keys for Fuzzy Duplicates

Last updated 2026-08-07.

Definitional boundary. This page defines a truth key and walks through choosing one. It does not cover how to generate the underlying synthetic records the truth key labels, or how to score match rules once a truth set is labeled; those are separate pages.

A truth key is not the primary key your source system already assigns each record. A source-system key, a customer ID, an account number, a record GUID, identifies one row. A truth key identifies one real-world entity across every row that refers to it, including the rows whose source-system keys are all different because each came from a different system, a different intake form, or a different typo. In a labeled dataset, that entity-level identity is usually carried by a separate column, a Master ID, that every known duplicate of the same entity shares, while each row keeps its own source-system key untouched. Confusing the two is the most common way a labeling effort produces a truth set that cannot actually answer "are these the same person or organization," because it never separated "same row" from "same entity" in the first place.

The distinction matters more once a dataset spans more than one source system. A person who exists in a CRM, a support ticketing system, and a billing system will have three different primary keys, one per system, none of which point at each other. A truth key exists specifically to bridge that gap: it is the label saying "these three otherwise-unrelated rows are the same real-world entity," independent of what identifier each source system happened to assign. Without that bridge, a match rule has nothing to be checked against except its own output, which is the exact problem an independent truth set exists to solve.

The fields that carry duplicate signal in a labeled corpus are typically the fields your match rules already key on: name, address, phone, and email, plus a field like nickname where a formal and informal identity diverge. Pick the fields your own match rules read before deciding which fields the labeled dataset needs to vary. A truth key built on fields your rules never touch cannot tell you anything about how those rules perform.

Start from the rule set, not from the schema. List every field your match rules currently key on, individually or in combination, and treat that list as the minimum set of fields the truth key needs to vary. If a rule matches on name and address together, the labeled dataset needs duplicate pairs where name and address vary independently of each other, not just pairs where both happen to change at once, or the scoring pass will never exercise the rule the way production data eventually will.

Once the candidate fields are set, every record that represents the same entity gets tagged with the same Master ID, a canonical identity value distinct from any source-system key. All rows sharing a Master ID represent the same synthetic identity; rows with different Master IDs represent different entities, even if some of their fields happen to look similar. This Master ID is the mechanism, not a label on top of it: it is what lets a scoring pass check whether a match rule correctly grouped rows that share a Master ID and correctly kept apart rows that don't.

Keep the Master ID and the source-system key in separate columns, never merged or overwritten into one field. A row keeps its own source-system key exactly as that system generated it, alongside the Master ID that ties it to its known duplicates. That separation is what makes the scoring pass mechanical instead of judgment-based afterward: a rule's output pair either shares a Master ID or it doesn't, and either that agrees with what the rule predicted or it doesn't. No reviewer has to look at the pair again and decide.

Alongside the Master ID, each known duplicate pair gets a Duplicate Type label describing how the two records differ from each other. The sourced duplicate types are: exact (formatting-identical duplicates), typo or edit-distance variants, phonetic variants, token-reorder (fields with word order changed), nickname or alias variants, and shared-account cases (two people sharing one login or identifier). Classifying by type matters because a match rule that catches exact duplicates and phonetic variants can still miss shared-account cases entirely, and a truth set with only one duplicate type in it cannot expose that gap.

Each type stresses a different piece of a match rule. Exact duplicates check whether the rule handles the trivial case at all. Typo or edit-distance variants check whether a fuzzy-matching threshold is tuned too tight or too loose. Phonetic variants check whether the rule accounts for names that sound alike but are spelled differently. Token-reorder cases check whether a rule that compares fields token by token survives a word-order change it was not built to expect. Nickname or alias variants check whether the rule has any concept of "Bill" and "William" as the same identity, which an edit-distance check alone will not catch. Shared-account cases check something different again: whether the rule can tell two people apart when they share a single identifier, the inverse problem of finding duplicates in the first place. A truth key that only encodes one or two of these types leaves the rest of the rule set unexamined, however clean the resulting precision number looks.

Building truth keys and tagging Master IDs by hand is exactly the labor a labeled dataset exists to remove. A Match-Ready Sample delivers a Labeled Ground-Truth Dataset with the truth key already chosen, the Master ID already tagged, and the Duplicate Type already classified for a given domain, so the fields above are decisions you can review rather than build from a blank sheet. The output is a Known-Duplicate Pair File: every duplicate pair tagged with its Master ID and Duplicate Type, ready to score match rules against instead of ready to be built from a blank spreadsheet.