CodeKitHubWhat Is This Tool?
Pasted lists — from spreadsheets, email exports, log files, scraped data — routinely carry the same five problems: duplicate entries, blank lines, inconsistent leading/trailing spaces, double spaces where there should be one, and no particular order. Fixing all five usually means bouncing between separate single-purpose tools.
This tool runs all five as one operation with checkboxes, in the order that actually matters (trim and collapse spaces before deduplicating, so "foo " and "foo" are correctly recognized as the same line).
Why Use It?
- Five real cleanup operations in one pass, not just deduplication.
- Checkbox control — turn off any step you don't want.
- Operations run in the order that produces correct results (whitespace normalized before dedup).
- Shows exactly how many lines were removed.
- Local and private — text never leaves your browser.
How to Use
- Paste your text or list into the input box.
- Check the operations you want (all but sorting are on by default).
- Click Clean — the result appears below with a count of removed lines.
- Copy the cleaned result.
Example
Input
apple
banana
apple
banana
cherryOutput
apple
banana
cherryWhitespace is normalized first, so "apple " and "apple" are recognized as duplicates and merged — a common miss in simpler tools.
Common uses
- Deduplicating a list exported from a spreadsheet or CRM before re-importing it.
- Cleaning up an email list merged from multiple sources, removing repeated addresses.
- Tidying log file output — stripping blank lines and duplicate entries before analysis.
- Preparing a keyword or URL list for SEO tools, one clean entry per line.
Why operation order changes the result
Running these five operations in the wrong order produces subtly wrong results, which is why this tool fixes the sequence rather than letting steps run in any combination. The clearest example: if deduplication ran before whitespace trimming, "apple" and "apple " (with a trailing space) would be treated as two different lines and both kept, silently defeating the entire point of deduplicating. Running trim-and-collapse first means every line is in its final, canonical form before the tool checks for duplicates, so lines that are meaningfully the same — regardless of stray whitespace picked up from a copy-paste — are correctly recognized as such. The same logic applies to blank-line removal happening before dedup: an accumulation of blank lines from different sources would otherwise get compared against each other in ways that don't matter to the actual content.
Frequently Asked Questions
In what order are the operations applied?
Trim → collapse spaces → remove blank lines → deduplicate → sort. This order matters: normalizing whitespace before deduplication means lines that differ only by stray spaces are correctly treated as duplicates.
Is the deduplication case-sensitive?
Yes — "Apple" and "apple" are treated as different lines. This avoids accidentally merging genuinely different entries; if you need case-insensitive matching, lowercase your text first.
What does sorting use for Chinese text?
Pinyin-based alphabetical order, so Chinese entries sort the way they would in a phone contacts list rather than by raw character code.
Will this handle a very large list?
Yes — processing happens in your browser with no size limit beyond what your device's memory can hold, which is well beyond typical spreadsheet-export sizes.
Is my data uploaded anywhere?
No. All five operations run as local JavaScript in your browser. Nothing is sent to a server, so this is safe to use on exported customer lists or other sensitive data.
Is this the same as a "duplicate line remover"?
Yes — removing duplicate lines is one of the five operations here (alongside trimming, collapsing spaces, removing blanks, and sorting). Check only "Remove duplicates" if that's all you need; the other four are optional.