Remove duplicate lines

How to remove duplicate lines and keywords

1. 1

   Paste the list or drop a file

   One keyword or item per line, pasted into the box or in a .txt, .csv or .md file of up to 5 MB.

2. 2

   Choose what counts as a duplicate

   Keep the first of each, the last, only lines that repeat or only lines that appear once. More settings adds Ignore case and Ignore punctuation.

3. 3

   Check the count and download

   Each column shows its line count, the right one previews the first 400 lines, and Download TXT saves the whole list.

Removing duplicate keywords from a list

Keyword lists pick up repeats when they are merged: two research exports, last month's list pasted under this one, the same phrase pulled from several tools. Put one phrase per line and the page keeps the first copy of each, drops the rest and leaves the order alone. A CSV export is compared line by line too. Two rows with the same keyword and different search volumes are different lines, so paste only the keyword column. On October 20, 2026, a test list of 10,000 phrases (241 kB) came back cleaned in about a fifth of a second in Chrome on a Mac.

What counts as the same line

By default, two lines match when their characters match once spaces, tabs and non-breaking spaces at either end are set aside. Ignore case makes Running shoes and running shoes one line. Ignore punctuation removes every punctuation mark and symbol before comparing, so [running shoes], "running shoes", +running +shoes and running shoes? count as one. Drop empty lines, also on by default, removes blank lines and lines of nothing but spaces. None of this changes what you download: every line that stays is written exactly as it was.

Duplicates it does not catch

Some lines are the same to a person and different to an exact comparison. The words can come in another order (buy running shoes and running shoes buy) or in another form (shoe and shoes). There can be two spaces between words, or a letter with and without an accent (café and cafe). No setting merges these, because the page compares characters, not meaning. We built a 10,000-line test list with known repeats: with every setting on, 300 lines with a double space and 200 with the words reversed stayed. Spaces and letters can be fixed first in Find and Replace. Choose Regular expression, put [ ]{2,} in Find and one space in Replace with, and add a pair such as é and e for each letter. Then pick Remove Duplicate Lines under the result; on the test list that took 7,500 lines down to 7,200. Word order and word forms have no fix on either page.

The order does not change

Many ways of removing duplicates sort the list first, because sorting puts identical lines side by side. That throws away the order you had. Here the list is read once, and each line is checked against the lines already seen. The lines that stay come out in the order they went in. The last of each keeps the latest copy of every line instead. That matters when later lines correct earlier ones, as in a log or an export with updated rows. If you want the result sorted as well, Sort Lines is one of the tools offered under the result.

Repeats, single lines and counts

Two modes turn the job around. Only lines that repeat keeps one copy of each line that appears two or more times. That shows repeated IDs, double bookings or keywords two campaigns share. Only lines that appear once keeps the lines without a partner. Put two clean lists one under the other, and you get what is in only one of them. Put the count before each line writes how many times each line occurred, then a tab, then the line. In Sort Lines, As numbers on column 1 with Largest first then puts the most repeated lines on top.

Frequently asked questions