Skip to content

Remove Duplicate Lines from Text

Remove duplicate lines from text online — free, instant, 100% in your browser. Ignore case or whitespace, keep first or last occurrence, and see how many duplicates were removed. Also sort and clean up. No signup.

No Tracking Runs in Browser Free
All processing happens locally in your browser. No data is sent to any server.
Sort
Deduplicate
Clean up
—
Reviewed for order-preserving dedupe, keep-first/keep-last correctness, case- and whitespace-insensitive matching, deterministic pipeline order, and accurate duplicate-count statistics — Go Tools Engineering Team · Jul 13, 2026

What Does Removing Duplicate Lines Mean?

Removing duplicate lines — deduplication, or "dedupe" — means scanning a block of newline-separated text and keeping only one copy of each distinct line. It is one of the most common cleanup tasks anyone who works with data runs into: collapsing a mailing list that accumulated repeats, reducing a log to its unique messages, finding the distinct values in a spreadsheet column, or merging several lists into one clean set.

The interesting question is what counts as a duplicate. Byte-for-byte comparison is the strict answer, but real data is messier. Two lines can look identical yet differ by a trailing space, or vary only in capitalization — `Error` versus `error`. This tool lets you decide: by default it ignores case, and you can enable whitespace-insensitive comparison so lines are matched on their trimmed, lowercased form. That way you deduplicate on the meaning of a line rather than its exact encoding, which is almost always what you actually want.

The other decision is which copy to keep. Keeping the first occurrence preserves the original order and is the default; keeping the last is useful when later entries supersede earlier ones. Either way, the result is deterministic and the live stats badge tells you exactly how many duplicates were removed, so you never have to trust the tool blindly.

Deduplication pairs naturally with sorting and other cleanup. This tool folds them into one panel: remove duplicates, then optionally sort, trim whitespace, drop blank lines, reverse, or add line numbers. When sorting is the main event, use the companion Sort Text Lines tool, which shares the same engine. To count the lines and words in your text, use the Word Counter; to see what changed between two versions of a list, use Text Diff; and to normalize casing before deduplicating, use the Case Converter.

Everything runs entirely in your browser with no uploads, so deduplicating a confidential list is exactly as private as editing it locally. Paste, and the unique result is ready to copy in a second.

Input:
  alice@example.com
  bob@example.com
  alice@example.com
  ALICE@example.com

Exact dedupe (case-sensitive):
  alice@example.com
  bob@example.com
  ALICE@example.com

Case-insensitive dedupe (default):
  alice@example.com
  bob@example.com   ← ALICE collapsed into alice

Key Features

Order-Preserving Deduplication

Keeps each unique line once in its original order by default — the same behavior as `awk '!seen[$0]++'` but without a terminal. Your list stays in the sequence you pasted it, just without the repeats.

Keep First or Last Occurrence

Choose whether the first or the last appearance of each line survives. Keep-first preserves order for general cleanup; keep-last is ideal when later entries override earlier ones, like config or environment values.

Case-Insensitive Matching

By default `Error`, `error`, and `ERROR` collapse to one line — the natural choice for most lists. Flip on Case-sensitive when casing is meaningful and duplicates must match exactly.

Whitespace-Insensitive Matching

Ignore leading/trailing spaces catches the invisible duplicates that differ only by a trailing space or tab — the number-one reason "identical" lines fail to deduplicate in data pasted from spreadsheets and PDFs.

Live Duplicate Count

A stats badge shows `12 lines → 9 lines · 3 duplicates removed` as you type. You always know exactly how many repeats were collapsed — no guessing whether the tool actually did anything.

Dedupe + Sort in One Pass

Enable a sort order to get a unique, alphabetized (or natural, or length-ordered) list in a single step. Deduplication runs first, then sorting, so nothing is lost and the output is both clean and ordered.

Clean-Up Toggles

The same panel also trims whitespace, drops blank lines, reverses order, and numbers the results. Stack any of them on top of deduplication and a chaotic pasted list becomes clean, unique, ready-to-paste output in one place — no scripting required.

100% Browser-Based Privacy

Your text never leaves the page — no uploads, no logging, no analytics. Safe for email lists, access logs, and confidential data. Verify it yourself in the Network tab: deduplicating produces zero requests.

Remove Duplicate Lines Examples

Remove exact duplicates

apple
banana
apple
cherry
banana
apple
banana
cherry

The default behavior: each line is kept once, at its first appearance, and later exact repeats are dropped. Original order is preserved. The stats badge reports `5 lines → 3 lines · 2 duplicates removed`.

Ignore case

Error
error
ERROR
Warning
Error
Warning

With Case-sensitive off (the default), `Error`, `error`, and `ERROR` are treated as the same line, so only the first is kept. Turn Case-sensitive on to keep all three as distinct entries.

Ignore leading/trailing whitespace

apple
  apple  
banana
banana
apple
banana

Enable Ignore leading/trailing spaces so `apple` and ` apple ` compare as equal. This catches the invisible duplicates that plague data copied from spreadsheets, PDFs, and terminals where trailing whitespace sneaks in.

Keep the last occurrence

v1
v2
v1
v3
v2
v1
v3

Switch Keep occurrence to Last and the final appearance of each line survives, in its last position. Here the second `v1` is kept and the first is removed. Useful when later entries override earlier ones, such as env vars or config overrides.

Deduplicate and sort together

dog
cat
dog
bird
cat
bird
cat
dog

Turn on Ascending sort to get a clean, unique, alphabetized list in one step. Deduplication runs first, then sorting, so the output is both unique and ordered — ideal for turning a messy pasted column into a tidy set.

Remove empty lines and duplicates

alpha

beta
alpha

gamma
alpha
beta
gamma

Enable Remove empty lines alongside deduplication to drop blank lines and repeats at once. The stats badge separates the two: `6 lines → 3 lines · 1 duplicates removed · 2 empty removed`.

Count how many duplicates exist

a
b
a
a
c
b
a
b
c

Even if you only care about the count, deduplication gives you the answer: the stats badge shows `6 lines → 3 lines · 3 duplicates removed`. A fast way to learn how much repetition is hiding in a list without manually tallying it.

Merge two lists into unique entries

red
green
blue
green
yellow
red
red
green
blue
yellow

Paste two lists back to back and deduplicate to get the union — every distinct line, no repeats, first-seen order preserved. The quickest way to combine mailing lists, tag sets, or ID lists without a spreadsheet formula.

How to Remove Duplicate Lines

  1. 1

    Paste or type your lines

    Drop your text into the input box. Each newline is a separate line. Duplicates are removed instantly as you type — there is no submit button. A trailing newline is ignored so it won't count as an extra blank line.

  2. 2

    Choose what counts as a duplicate

    Leave Case-sensitive off to treat `Foo` and `foo` as the same, or turn it on to keep them distinct. Enable Ignore leading/trailing spaces to catch lines that differ only by stray whitespace.

  3. 3

    Pick which occurrence to keep

    Keep occurrence defaults to First, preserving original order. Switch to Last when the final appearance of each line should win — useful for overrides where later entries supersede earlier ones.

  4. 4

    Add sorting or clean-up if needed

    Enable a sort order to alphabetize the unique result, or turn on Remove empty lines and Trim each line to tidy the list in the same pass. The stats badge shows the effect of each toggle.

  5. 5

    Copy the unique result

    Click Copy to put the deduplicated output on your clipboard. Use Reset to clear both boxes, or Sample to load an example list. Nothing is uploaded at any point in the process.

Common Deduplication Mistakes

Whitespace Hiding Duplicates

Lines that look identical but differ by a trailing space or tab are not exact duplicates, so they survive. Enable Ignore leading/trailing spaces to compare on the trimmed form. This is the single most common deduplication surprise, especially with spreadsheet and PDF data.

✗ Wrong 
Exact match ("apple " ≠ "apple"):
  apple
  apple     ← trailing space, kept as unique
  banana
✓ Correct 
Ignore whitespace:
  apple
  banana

Case Difference Treated as Unique

With Case-sensitive on, `Error` and `error` are different lines and both survive. If you meant them to be the same, turn Case-sensitive off (the default). Conversely, turning it off when case is meaningful can merge lines you wanted to keep apart.

✗ Wrong 
Case-sensitive on:
  Error
  error
  ERROR   ← all three kept
✓ Correct 
Case-insensitive (default):
  Error   ← the rest collapse

Expecting Non-Adjacent Duplicates to Stay

Unlike the command-line `uniq`, this tool removes duplicates anywhere in the list, not just adjacent ones — so you do not need to sort first. If you actually wanted to collapse only consecutive repeats, this tool is not the match; sort or restructure your data instead.

✗ Wrong 
Assuming uniq behavior (adjacent only):
  a
  b
  a   ← expected to survive, but it's removed
✓ Correct 
Full dedupe (this tool):
  a
  b   ← the later 'a' is removed anywhere it appears

Confusing Empty-Line Removal with Dedupe

Deduplication collapses repeated blank lines but keeps one; it does not delete all blanks. To drop every empty line, enable Remove empty lines. The stats badge reports the two counts separately so you can tell whether blanks or true duplicates were removed.

✗ Wrong 
Dedupe only (one blank kept):
  alpha
  (blank)
  beta
✓ Correct 
Dedupe + Remove empty lines:
  alpha
  beta

Remove Duplicate Lines: Use Cases

Clean Up Email & Contact Lists
Paste an exported mailing list that accumulated repeats and get a unique set back. Enable case- and whitespace-insensitive matching so `Bob@x.com`, `bob@x.com`, and `bob@x.com ` all collapse into one address.
Reduce Logs to Unique Messages
Deduplicate a noisy log to see the distinct error or event lines without the repetition. Keep-last is handy when the most recent occurrence of each message matters more than the first.
Find Unique Values in a Column
Copy a column out of a CSV or spreadsheet, deduplicate it, and the output is the set of distinct values — with a count of how many duplicates existed, which doubles as a quick cardinality check. It also lets you find duplicate lines and strip duplicate rows from a pasted column without a spreadsheet formula.
Merge Lists Without Repeats
Paste two or more lists back to back and deduplicate to get their union: every distinct line, no repeats, in first-seen order. Faster than a spreadsheet formula for a one-off merge of tags, IDs, or names.
Deduplicate Security Wordlists
Combine and clean password wordlists, subdomain lists, or fuzzing dictionaries entirely offline. Since every operation is local, confidential security data stays on your machine and is never sent to an external service.
Tidy Import or Dependency Lists
Remove duplicate import statements, package names, or dependency entries that crept in during copy-paste. Add a sort order to produce a canonical, unique list that keeps future diffs small.
Prepare Data for Analysis
Strip repeats before counting, charting, or loading data so duplicates don't skew your results. Pair with the Word Counter to measure the cleaned list, or Text Diff to compare it against another version.
Deduplicate Keyword Lists for SEO
Keyword research exports are full of near-duplicates. Remove exact repeats here, optionally lowercased and trimmed, to get a clean unique keyword set before grouping or mapping it to pages.

How Deduplication Works

Hash-Set Deduplication
Lines are deduplicated with a hash set (a JavaScript `Set` or `Map`) keyed on the comparison form of each line, giving linear-time performance even for large lists. Each line's key is computed once and looked up in constant time.
Comparison Key Normalization
The key used for matching is derived from the line: trimmed first if Ignore leading/trailing spaces is on, then lowercased if Case-sensitive is off. The original line — not the normalized key — is what appears in the output, so casing and spacing of survivors are preserved.
Keep-First vs Keep-Last
Keep-first pushes a line to the result the first time its key is seen and skips later repeats. Keep-last computes each key's final index and keeps only the line at that index, so the surviving copy sits at its last position while overall order still follows the input.
Line Splitting (CRLF / LF / CR)
Lines are separated on any newline convention — `\r\n` (Windows), `\n` (Unix), or `\r` (classic Mac) — so mixed or pasted input just works. One trailing newline is dropped before processing so a file that ends in a line break doesn't register a phantom empty duplicate at the bottom.
Deterministic Pipeline Order
The pipeline runs in a set sequence — whitespace trimming, then empty-line removal, then deduplication, then any sort or shuffle, then reverse, then line numbering. Because duplicates are collapsed before the optional sort step, turning on a sort order re-orders the unique set without ever dropping a distinct line.
Separate Duplicate & Empty Counts
The stats badge tracks duplicates removed and empty lines removed independently, computed as the line-count delta before and after each stage. This lets you see, for example, `1 duplicates removed · 2 empty removed` rather than a single combined number.
In-Browser, No Workers Needed
All deduplication runs on the main thread in your browser with no server round-trips. Typical lists complete instantly; because the algorithm is linear and set-based, performance scales gracefully to very large inputs without freezing the page.

Deduplication Best Practices

Trim Whitespace to Catch Hidden Duplicates
The most common reason duplicates survive is invisible trailing whitespace. When deduplicating data pasted from spreadsheets, PDFs, or terminals, enable Ignore leading/trailing spaces so lines are compared on their trimmed form. If the count of removed duplicates is lower than you expect, whitespace is usually why.
Decide on Case Sensitivity Deliberately
For emails, tags, and human-readable lists, keep Case-sensitive off so `Foo` and `foo` collapse. Only turn it on for case-significant data like identifiers or hashes where `A1B2` and `a1b2` are genuinely different values. Choosing wrong silently changes which lines survive.
Use Keep-Last for Override Semantics
When later entries should win — configuration overrides, environment variables, a log where the newest status matters — switch Keep occurrence to Last. For general list cleanup where order should be preserved from the top, keep the default First.
Deduplicate Before You Count or Analyze
Repeats skew counts, charts, and aggregates. Remove duplicates first, then measure. The stats badge doubles as a quick cardinality check: the difference between lines in and lines out tells you how much repetition the data carried.
Sort After Deduplicating for a Canonical List
If you need a stable, comparable list — for a config file, a manifest, or a diff — enable a sort order alongside deduplication. A unique, sorted list produces minimal diffs and is easy to compare against another version with Text Diff.
Verify with the Live Count Before Copying
Glance at the stats badge before you copy. `12 lines → 9 lines · 3 duplicates removed` confirms the tool did what you intended. If the numbers look off — zero removed when you expected repeats — revisit the case and whitespace settings.

Frequently Asked Questions

What does this tool do?
It removes duplicate lines from a block of text, keeping each unique line once. As you paste or type, the deduplicated result appears instantly with a live count of lines in, lines out, and how many duplicates were removed. You can control what counts as a duplicate (ignore case, ignore whitespace), choose to keep the first or last occurrence, and optionally sort, trim, remove empty lines, or add line numbers in the same pass. Everything runs 100% client-side in your browser — nothing is uploaded, logged, or stored — so it is safe for emails, logs, and any confidential list. You can confirm this in your browser's Network tab: deduplicating triggers zero network requests.
Does it keep the first or last occurrence of a duplicate?
By default it keeps the first occurrence and removes later repeats, which preserves the original order of your list. Switch Keep occurrence to Last to keep the final appearance of each line instead — the surviving line then sits at its last position. Keep-first is the right choice most of the time (it is what `sort -u` and most dedupe tools do); keep-last is handy when later entries supersede earlier ones, such as configuration overrides, environment variables, or a log where the newest status wins. The stats badge reports the same duplicate count either way.
Is my text uploaded anywhere?
No. Deduplication and every other operation run entirely in your browser using JavaScript. Your text is never transmitted, never stored on any server, never logged, and never analyzed. This makes the tool safe for email lists, access logs, internal data, and any content you would not paste into a networked service. No cookies capture your input and no third-party analytics read what you type. Open your browser's developer tools and watch the Network tab while you paste a huge list — you will see zero requests, because the work happens locally.
How do I remove duplicates but ignore case?
Leave Case-sensitive off — that is the default. With it off, `Error`, `error`, and `ERROR` are treated as the same line and collapsed to a single entry (the first one encountered, in its original casing). Turn Case-sensitive on only when the casing is meaningful and `ID` should be kept separate from `id`. Case handling applies to the comparison, not the output: the surviving line keeps whatever casing it originally had; only the decision about which lines are duplicates changes.
Why aren't my visually identical lines being deduplicated?
Almost always the culprit is invisible whitespace — a trailing space, a tab, or a non-breaking space — that makes two lines differ byte-for-byte even though they look the same. Turn on Ignore leading/trailing spaces so lines are compared after trimming, and the duplicates will collapse. This is extremely common in data pasted from spreadsheets, PDFs, and terminal output. If lines still won't merge, they may differ by an internal character (a double space vs single space, or a smart quote vs a straight quote); trim only handles the edges.
Does removing duplicates change the order of my lines?
No — with keep-first (the default), the surviving lines stay in their original order; only the repeated appearances are removed. So deduplicating `a, b, a, c` gives `a, b, c` in that order. If you also want the result sorted, enable a sort order and the tool will deduplicate first, then sort. With keep-last, order is still based on original positions, but each surviving line moves to where its last occurrence was. In every mode the output is deterministic, so the same input and settings always produce the same result.
Can I remove duplicates and sort at the same time?
Yes. Enable a sort order (Ascending or Descending) while deduplication is on, and you get a clean, unique, ordered list in a single step. The pipeline removes duplicates first, then sorts, so nothing is lost. If sorting is your primary goal and dedupe is secondary, the companion Sort Text Lines tool shares the same engine and opens with sorting enabled. Both tools expose the full option set; they differ only in which task they lead with.
What happens to blank lines?
By default, blank lines are treated like any other line: if the same blank line appears twice, the duplicate is removed, but a single blank line is preserved. If you want to drop all blank and whitespace-only lines entirely, enable Remove empty lines — the stats badge counts them separately from duplicates, so you can see `6 lines → 3 lines · 1 duplicates removed · 2 empty removed`. A single trailing newline at the end of your input is treated as a line terminator, not an extra blank line, so it won't skew the counts.
Is there a limit on list size?
There is no hard limit built into the tool; it is bounded only by your browser's memory and how much text you can paste into a textarea. Deduplication uses a hash set, so it is fast even for large lists — tens of thousands of lines are instantaneous and hundreds of thousands complete with a brief pause. Because everything runs locally, large lists are often quicker here than on server-based tools that must upload your data first. For multi-million-line files, a command-line tool like `sort -u` or `awk '!seen[$0]++'` will be more comfortable, but for everyday lists this handles them easily.
How does this compare to the command line (sort -u, awk, uniq)?
`sort -u` removes duplicates but also sorts, and `uniq` only collapses adjacent duplicates (so it requires pre-sorted input). `awk '!seen[$0]++'` removes duplicates while preserving original order — the same as this tool's keep-first mode — but requires a terminal. This tool gives you order-preserving dedupe, keep-first or keep-last, case- and whitespace-insensitive comparison, optional sorting, and a live count, all without leaving the browser or installing anything. It is the fastest option when the list is already on your clipboard rather than in a file.
How is this faster than removing duplicate lines in Excel or Notepad++?
In Excel you'd use Data → Remove Duplicates on a column, and Notepad++ needs a TextFX or sort-then-uniq plugin — both require the data to already be inside that app. This tool works the moment text is on your clipboard: paste it and duplicate lines are removed instantly, with case- and whitespace-insensitive matching, keep-first or keep-last, and a live duplicate count those tools don't show. It also preserves the original order, unlike Excel's row-based removal or a sort-first approach, and nothing is uploaded. When you just need to dedupe a chunk of text quickly, it beats switching to a spreadsheet or installing a plugin.
Can I use this to find unique values in a column?
Yes. Copy a single column out of a spreadsheet or CSV, paste it here, and the deduplicated output is the set of unique values. Enable Ignore leading/trailing spaces to catch values that differ only by stray whitespace, and turn on a sort order if you want them alphabetized. The stats badge tells you how many duplicates existed, which doubles as a quick way to gauge the cardinality of the column. Paste the unique result back into your sheet or query when you're done.

Related Tools

View all tools →

Case Converter — UPPERCASE, lowercase, camelCase & More

Text Processing

Convert text between UPPERCASE, lowercase, Title Case, Sentence case, camelCase, PascalCase, snake_case, kebab-case, CONSTANT_CASE and 6 more formats instantly. Free, browser-only, no signup.

Lorem Ipsum Generator — Free Placeholder Text Tool

Text Processing

Generate Lorem Ipsum placeholder text instantly — by paragraph, sentence, word, byte, or list. Copy or download as plain text, HTML, Markdown, or JSON. 100% free, private, in-browser. No sign-up.

Free Regex Tester — Debug & Match Patterns Online

Text Processing

Test regex patterns instantly against any text. Live match highlighting, capture groups, replace preview, split, and pattern explainer. JavaScript-flavor regex, 100% private, no signup.

URL Slug Generator — Slugify Any Text

Text Processing

Slugify any title into a clean, SEO-friendly URL slug instantly. Transliterate accents and Cyrillic, or keep Unicode letters. 100% private, in your browser.

Sort Text Lines Alphabetically & Naturally

Text Processing

Sort text lines alphabetically, numerically (natural order), or by length — ascending or descending. Free online line sorter, 100% in your browser. Also dedupe, reverse, and clean up lists. No signup, nothing uploaded.

Text Diff & Compare

Text Processing

Compare two texts instantly in your browser. Side-by-side view with inline word-level highlights, unified-diff export, ignore case / whitespace / blank lines. 100% private — your text never leaves your device.