Cleaning messy lists in the browser: removing duplicate lines

Duplicate lines are everywhere: a list merged from two exports, an order report that repeats the order ID on every line item, or a log file that prints the same error again and again. Removing them by hand is slow, and the duplicates that differ by one capital letter or a trailing space are the easiest to miss.

I built a small browser tool that removes duplicate lines without uploading anything, and this post walks through the logic behind it.

Keep the first occurrence, drop the rest

The core of a duplicate remover is a Set. Walk the lines in order, keep a record of what you have seen, and skip anything that repeats:

function removeDuplicateLines(text, { caseInsensitive = false } = {}) {
  const lines = text.split(/\r?\n/);
  const seen = new Set();
  const result = [];

  for (const line of lines) {
    const key = caseInsensitive ? line.trim().toLowerCase() : line;
    if (!seen.has(key)) {
      seen.add(key);
      result.push(line);
    }
  }

  return result.join("\n");
}

Two details matter more than they look:

• Preserve order. Sorting the output is a separate feature. Most people want their original list back, minus the repeats.
• Case sensitivity is a choice. To JavaScript, Apple and apple are different strings. For a name list you may want them merged; for product codes or usernames you probably do not. Expose it as an option instead of guessing.

That is the whole algorithm. Everything else is interface.

Why not just sort -u?

If you live in a terminal, this does the job:

sort -u input.txt > output.txt

But it sorts, it only matches exact strings, and a single trailing space silently keeps a duplicate alive. A file that "didn't deduplicate" is almost always a whitespace problem, not a logic problem.

The browser tool exists for the cases where you want to see the before and after, and choose the comparison mode without writing a one-off script.

The near-duplicate trap

The duplicates that cause trouble are often not identical:

If your list came from a spreadsheet, a form, or several people typing, expect near-duplicates. Trim spaces and normalise case first, then deduplicate. Otherwise the tool does exactly what you asked and removes only the lines that are truly identical.

A worked example

Input:

apples
bananas
Apples
bananas
cherries

With exact matching and preserve order, the output is:

apples
bananas
Apples
cherries

Apples survives because the capital A makes it a different line. bananas appears once. Turn on case-insensitive mode and the output becomes:

apples
bananas
cherries

The first spelling wins, which is why checking inconsistent capitalisation before you delete is worth the extra minute.

When you should not remove duplicates

Removing duplicates is not always correct. Keep them when the repetition carries information:

• Order and invoice lines — the same product can legitimately appear twice; the quantity matters.
• Logs and audit trails — a repeated error at 10:00 and 11:00 is a pattern, not noise.
• Votes, reactions, and counts — one entry per event is the data.
• Survey answers — two people can give the same answer and both count.

A simple test: if deleting the second copy loses a fact, do not delete it.

Try it

• Duplicate line remover (runs locally in the browser): https://yourutilityhub.com/text/duplicate-line-remover
• Extra spaces remover: https://yourutilityhub.com/text/remove-extra-spaces
• Full walkthrough with spreadsheet and command-line methods: https://yourutilityhub.com/blog/remove-duplicate-lines

No signup, and nothing is uploaded — the text is processed in the page itself. If you clean lists regularly, the command line still wins for batch jobs. For a one-off paste, this is faster.