Duplicate rows skew totals, inflate counts, and break anything that assumes one row per real-world entity. But "duplicate" isn't always as obvious as it sounds, this guide covers both the easy case and the messy one.

Exact Duplicates vs. Near-Duplicates

Exact duplicates

John Smith,[email protected]
John Smith,[email protected]

Every field matches, byte for byte. Straightforward to detect and remove.

Near-duplicates

John Smith,[email protected]
Jon Smith,[email protected]

Same underlying person, different spelling. An exact match won't catch this.

Step 1: Decide What Counts as a "Duplicate"

This is the step people skip, and it's the one that actually matters. Deduplicating on the entire row means two records with even one different field (say, a slightly different "last updated" timestamp) won't be caught. Deduplicating on a specific key column (like email or a unique ID) is usually what you actually want, but you have to choose that key deliberately.

Common mistake: deduplicating on a column that isn't actually a reliable identifier (e.g. "name" alone, when two different people can share a name). Always check whether your chosen key is actually unique per real-world entity before trusting the result.

Step 2: Handle Exact Duplicates First

Remove exact row-level duplicates before anything else, they're unambiguous and removing them first reduces the dataset size for the more expensive fuzzy-matching step that may come next.

Step 3: Decide If You Need Fuzzy Matching

If your data comes from manual entry, multiple source systems, or user-submitted forms, near-duplicates are almost guaranteed: typos, different formatting ("St." vs "Street"), or different capitalization. Fuzzy deduplication compares values by similarity rather than exact equality, using a threshold you control.

Threshold too loose: distinct records get merged incorrectly (e.g. "John Smith" and "Jane Smith" treated as the same person).
Threshold too strict: real duplicates slip through because the similarity score falls just short.

Start conservative (a higher similarity threshold), review a sample of what gets flagged, then loosen it gradually while spot-checking results.

Step 4: Decide Which Row to Keep

When two rows are duplicates, which one survives? Common strategies: keep the first occurrence, keep the most recently updated, or keep whichever row has the fewest missing values (the more "complete" record). Pick this deliberately, the default "keep first" isn't always the right answer.

Step 5: Verify the Result

After deduplication, spot-check a sample of removed rows to confirm they really were duplicates, and check that the row count dropped by a plausible amount. A near-total collapse in row count usually means your key column or similarity threshold was too loose.

Doing This in How To CSV

Logical Dedupe for exact and key-based duplicate removal with control over which columns define uniqueness
Fuzzy Dedupe for near-duplicate detection with an adjustable similarity threshold

Have duplicate rows to clean up?

Remove exact or near-duplicate rows in your browser, no upload required.

Deduplicate Your Data

Turn this into a saved workflow

Create a free account to save the steps from this guide as a reusable workflow and re-run it on any file, from any device.

How to Remove Duplicate Rows from a CSV File