How to use CSV Deduplicator
- Import a CSV and select every column needed to identify a record.
- Choose exact or normalized matching, then keep the first or last occurrence.
- Review kept and removed counts. Download the cleaned file and keep your original backup.
What the matching switches do
Trim removes surrounding Unicode whitespace from selected key values. Ignore case applies JavaScript Unicode lowercase conversion; it is not locale-specific linguistic matching. Neither option changes the original exported cell text. Punctuation, internal spaces and accent differences remain significant. Selecting all columns removes fully matching rows under the chosen rules.
First and last refer to file order
Keep last does not mean newest unless your CSV was already ordered that way. The retained rows preserve their original relative order. This tool does not sum quantities or merge non-key fields. If duplicate rows carry conflicting quantities, choose a rule only after inspecting the source; retaining one record discards the other record’s non-key values.
Empty values are a matching decision
Empty selected key cells count as equal. Several rows with an empty SKU can collapse into one if SKU is the only key. Add another identifying field or repair missing identifiers before running the tool. The summary reports how many rows are removed; the preview contains retained rows only and is limited to the first 50.
CSV format and safe export
Use comma-delimited files with a unique, non-empty header row. Quoted commas, escaped quotes, quoted line breaks, CRLF and UTF-8 BOM are supported. Whitespace-only rows are skipped. Choose the correct encoding before import; non-UTF-8 content is not guessed. Parsing and processing run in a browser Worker. Only the first 50 result rows appear in the preview; downloads contain the complete result. Outputs are also limited to 100,000 rows, 100 columns, 1 million cells and 32 MiB; expanded comparisons or merges may hit a conservative size limit earlier. Reduce fields or split files if an output limit is reached.
Exports include a UTF-8 BOM and CRLF line endings. Cells starting with =, +, − (ASCII hyphen), or @ after whitespace/control characters, and cells starting with a tab or line break, receive a leading apostrophe. Headers are protected too. Negative numbers may therefore export as text. This changes potentially dangerous cell content; the result reports the number of protected cells. Keep an untouched original.