Duplicate detection in a CRM that catches real duplicates
A school's front desk was creating four duplicate contacts per week — and merging two different parents once per term. The merging was the real cost. When two people get collapsed into one record, you lose the distinction between the parent who paid last week and the parent who never responds. The front desk staff weren't being careless; they were given a list of suggested merges and told to click through them.
The problem with most duplicate-detection logic is that it treats every match as a yes-or-no decision. Match on name and phone, merge. Match on name and email, merge. That works for a 500-person test database. In a school CRM with 3,000 families, 9,000 students and 12 years of admission history, it creates a steady trickle of false merges.
We rebuilt the detection as a scoring system that surfaces likely duplicates for human review but never auto-merges above a configurable threshold.
The core is a weighted similarity score across four fields: legal name (35%), phone (25%), email (25%), and address (15%). Each field produces a score between 0 and 1, and the weighted sum becomes the match score. Name comparison uses a normalised Levenshtein ratio after stripping titles, middle initials, and transliteration variants — "Smt. Rajeshwari Iyer" and "Rajeshwari Iyer" should compare as the same person. Phone numbers are normalised to E.164 format before comparison, so "+91 98xxx 12345" and "098xxx12345" match exactly.
The code that scores a candidate pair against an existing contact:
from rapidfuzz import fuzz
import phonenumbers
def score_duplicate(a, b):
name_ratio = fuzz.partial_ratio(
normalize_name(a.name), normalize_name(b.name)
) / 100.0
phone_ratio = 0.0
pa, pb = e164(a.phone), e164(b.phone)
if pa and pb:
phone_ratio = 1.0 if pa == pb else 0.0
email_ratio = 0.0
if a.email and b.email:
email_ratio = 1.0 if a.email.lower() == b.email.lower() else 0.0
addr_ratio = fuzz.partial_ratio(
a.address.lower().strip(),
b.address.lower().strip()
) / 100.0
return (0.35 * name_ratio + 0.25 * phone_ratio
+ 0.25 * email_ratio + 0.15 * addr_ratio)A score above 0.85 flags the pair for review. Below 0.60, no suggestion at all. The gap between 0.60 and 0.85 is where most of the work happens — the front desk sees a yellow warning, not a merge prompt, and makes the call.
The weights came from two weeks of running the scorer against contacts the school's office manager had manually classified. The original split was 40/25/25/10, but address turned out to be a strong disambiguator in cases where two siblings lived at the same address and shared a phone number — which happens regularly in Indian joint families. Bumping address to 15 and dropping name to 35 reduced false suggestions by about a third.
The tradeoff with this approach is latency. Scoring a new contact against 9,000 existing ones runs roughly 11,000 comparisons. On Postgres, we cut the candidate set first with a trigram index on the name column so the Python scorer only sees pairs with a name similarity above 0.3. That brings the average down to 40-60 comparisons per new contact, which runs in under 50 milliseconds.
One thing we did not do: block matching on just phone or email as a hard merge. Schools have shared phones, shared emails, and parents who use the child's email for everything. A hard match on a single identifier would have recreated the original problem with a different label.
The front desk now flags about six contacts per week for review. The office manager reviews them in under two minutes total. No auto-merge happens. The false-merge rate — the one that cost staff hours to undo — is zero since the system went live in March.