Data Deduplicator

About Data Deduplicator

Data Deduplicator is a powerful online tool that identifies and removes duplicate rows from CSV and JSON files. Pick key columns (such as email or customer ID), optionally ignore letter case and normalize whitespace to catch near-duplicates that differ only by stray spaces, then choose to keep the first or last occurrence and export clean unique records. All processing happens locally in your browser, so even large mailing lists, CRM exports, and confidential datasets never leave your device.

How does duplicate detection work?

The tool compares rows based on the columns you select (key columns). If two or more rows have identical values in all selected columns, they are considered duplicates. You can choose to compare all columns or only specific ones, making it flexible for different data cleaning scenarios.

What's the difference between keeping first vs last occurrence?

When duplicates are found, you can choose which copy to keep. 'Keep first occurrence' retains the first row that appears in the file and removes subsequent duplicates. 'Keep last occurrence' keeps the most recent duplicate and removes earlier copies. This is useful when newer data should replace older entries.

Is my data secure?

Yes. All deduplication processing happens locally in your browser using JavaScript. Your files never leave your device, ensuring complete privacy for sensitive datasets like customer records, financial data, or confidential lists.

What file formats are supported?

Data Deduplicator supports CSV files (with various delimiters: comma, semicolon, tab, pipe) and JSON files (arrays of objects). Both formats can be deduplicated and exported to either CSV or JSON format after processing.

Can I see which rows were duplicates?

Yes. The tool provides two separate views: Unique Records (rows that will be kept) and Duplicate Records (rows that were removed). This lets you review both datasets before downloading, ensuring the deduplication worked as expected.

What does case-sensitive comparison do?

When enabled, 'Apple' and 'apple' are treated as different values. When disabled (default), uppercase and lowercase letters are considered identical. This is useful when your data may have inconsistent capitalization but you want to treat similar entries as duplicates.

What does 'Normalize whitespace before comparison' do?

When enabled, the tool trims leading and trailing spaces and collapses runs of internal spaces, tabs, and line breaks into a single space before comparing. So ' John Smith ' and 'John Smith', or '[email protected] ' and '[email protected]', are detected as the same record. This is essential for CRM, mailing-list, and spreadsheet exports, where stray whitespace is the most common reason exact-match dedup misses real duplicates. Combine it with case-insensitive comparison for the cleanest results.

Which key columns should I pick for email or CRM deduplication?

For mailing lists, select only the email column as the key so contacts are merged whenever the address matches, regardless of differing names or tags. For CRM records, use a stable unique identifier such as customer ID, or a small combination like email plus phone, rather than all columns; selecting every column only removes rows that are identical in every field and will leave true duplicates that differ in a single note or timestamp. Enabling whitespace normalization and case-insensitive matching on these key columns catches the typical variations in exported data.

Can it handle large files, and why does the preview stop at 100 rows?

All rows in your file are deduplicated, and the full result is included in every CSV or JSON download. Only the on-screen preview is capped at the first 100 rows of each tab to keep the interface fast and responsive on large datasets; the count shown next to each tab (for example 'showing 100 of 24,500') reflects the true totals. Note that Total rows equals Unique rows plus Duplicate rows, so you can verify the split at a glance before exporting.