🧹 Remove Duplicates
Remove duplicate lines from text with sorting and trimming options
✓ Free Forever · ✓ No Signup · ✓ No Upload · ✓ Works Offline
🔒 Your text never leaves your device. All processing happens locally in your browser using JavaScript Set-based deduplication. No upload, no server, no tracking.
Features
- Line-by-Line Deduplication — Two lines count as duplicates only when they match exactly — optionally ignoring case and leading/trailing spaces. The first copy stays where it was, and you can sort the result. Plain string comparison, no guessing.
- Statistics — See how many original, unique, and duplicate lines were found.
- Flexible Options — Ignore or respect capitals, ignore spaces at the start and end of a line, drop empty lines, and sort A→Z (numbers in number order). Change an option after a run and the result updates.
- Copy & Download — Copy the deduplicated result or download as a text file.
How It Works
- Paste your text — Enter text with duplicate lines — one item per line.
- Configure options — Set case sensitivity, trim whitespace, and sort preferences.
- Get unique lines — Click Remove Duplicates and copy or download the result.
Use Cases
- Deduplicate email lists before sending campaigns
- Clean up copied data with repeated entries
- Remove duplicate log entries for analysis
Frequently Asked Questions
Is the comparison case-sensitive by default?
No. By default "Hello", "hello" and "HELLO" count as the same line, and the first one is kept exactly as you wrote it. Turn on Case Sensitive to keep all three.
Does it change the lines it keeps?
No. It only removes whole repeated lines — the copy that stays is the first one, character for character, indentation and inner spaces included. "Ignore spaces at start/end" only decides what counts as the same line; it doesn't trim your text.
Does it preserve the original order?
Yes, by default. Turn on Sort Output to sort the unique lines A→Z, with numbers in number order (item2 before item10). A sorted list leaves empty lines out.
What happens to empty lines?
They stay where they were and are never counted as duplicates, so paragraphs keep their spacing. Turn on Remove empty lines to drop them all.
Notes from Milo 🐨
I pasted a bit of Python in to drop a repeated line and got it back with every indent gone: "Trim whitespace" didn't just compare lines without their spaces, it rewrote every line it kept. Now the tool only removes whole repeated lines, and the copy that stays is exactly what you wrote. The page also said comparison was case-sensitive by default when it wasn't — fixed. New: a Remove empty lines switch, sorting that puts item2 before item10, a result that follows your options and edits after the first run, an Open file button (or drop a .txt on the box), and a duplicate list that shows each repeated line once with how many times it appeared — a 200,000-line log no longer freezes the tab.
Related Tools
- 🔢 Word Counter — Analyze text — word count, characters, reading time, and top words
- 🔤 Case Converter — Convert text between uppercase, lowercase, camelCase, and more
- 🔍 Regex Tester — Test and debug regular expressions with real-time matching
💬 Bug, idea, or just to say hi? Tell Milo 🐨 · [email protected]