Remove Duplicate Lines
Strip repeats, keep the first of each, keep the order.
The answer appears here as you type.
Guide: Order is preserved1 min read
Order is preserved
The first occurrence of each line stays where it was. Sorting to find duplicates is the obvious approach and it destroys whatever order the list was in, which is frequently the thing that mattered.
If you do want it sorted, run it through the sort tool afterwards — two steps that each do one thing.
Trailing spaces are invisible duplicates
apple and apple are different strings and look identical on screen. A list pasted out of a spreadsheet or an email is full of them.
Trimming is on by default for that reason. Turn it off when the spacing is meaningful — indented code, for instance, where two lines differing only in leading whitespace are genuinely different lines.
Case is a judgment call
Email@example.com and email@example.com are the same mailbox in practice, and different strings to a computer.
Ignoring case is off by default because it is destructive in the other direction: identifiers, passwords and codes are frequently case-sensitive. Turn it on when you know your data is not.
Nothing is sent anywhere
Everything on this page happens in your own browser. No request is made, nothing is stored between visits, and the page keeps working with the connection switched off.
Frequently asked questions
Which duplicate is kept?
The first. Later copies are removed, so the earliest position in your list wins.
Does it remove blank lines?
Repeated blank lines collapse to one, since a blank line is a duplicate of a blank line. A single blank line between blocks survives.
How many lines can it handle?
Hundreds of thousands without trouble. It runs in your browser, so the limit is memory rather than an upload cap.
Is my list sent anywhere?
No. It stays in this tab. Worth knowing given how often these lists are email addresses.