Text

Text Cleaner Guide: Remove Duplicates, Sort Lines and Change Separators

·

A text cleaner turns a pasted list into a more useful list by applying the cleanup options you choose. ToolMellow can trim item edges, collapse spaces, remove empty or duplicate items, sort naturally or reverse the retained items. Set Input separator and Output separator, review the checkboxes and press Clean text. Processing runs locally in a browser worker.

The defaults trim each item, remove empty items and remove exact duplicates while keeping the original order. Case-sensitive matching stays on; Collapse repeated spaces and Ignore case start off. This guide explains the order of operations, shows reproducible examples and distinguishes literal list cleanup from CSV parsing, Unicode normalization and rewriting prose.

Text cleaner & line sorter

How to clean a list without losing the original

Paste a list or load a local text file. Choose Text language for locale-dependent comparison, then select the input and output separators. Line breaks is the default for both separators, and Keep original order is the default Line order. Text language starts with the website language and offers all eleven supported languages; it controls comparison rather than translating your text.

Check which changes you actually want before pressing Clean text. Copy result and Download operate on the completed result, while Use result as input explicitly replaces the editor contents with that result. Editing input or cleanup settings invalidates the previous result; typing alone does not run cleaning. Wrap lines changes display wrapping only.

  • Keep a separate copy of the source if whitespace, duplicate occurrences or order have meaning.
  • Start with a small sample containing the edge cases you expect, then review the result before applying the same settings to the whole list.
  • Compare the retained spelling and item count before replacing another system’s data. A cleaner does not validate that system’s identifiers or business rules.

Line breaks, commas, tabs and custom separators

Input separator determines where items begin and end. Line breaks accepts CRLF, CR and LF; its output counterpart joins items with LF. Comma, Semicolon and Tab split on the selected literal character. Custom uses the exact string you enter, with a length from 1 to 20 UTF-16 code units. Matching is case-sensitive, and a multi-character separator is matched as a whole string.

Leading, adjacent and trailing separators create empty items, which Remove empty lines can discard. A retained empty final item can make joined output end with a separator. A custom separator such as .* is ordinary text, and typing \n into the custom field supplies a backslash followed by n, not a decoded newline. To split real line endings, select Line breaks.

Line breaks, commas, tabs and custom separators
Separator choiceInput behaviorOutput behavior
Line breaksCRLF, CR or LF separates itemsJoins retained items with LF
CommaSplits on ,Joins with ,
SemicolonSplits on ;Joins with ;
TabSplits on a real tab characterJoins with a real tab character
CustomSplits on the exact 1–20-unit stringJoins with the exact 1–20-unit string

ECMAScript: String values, splitting, trimming and case mapping

The cleanup pipeline and why its order matters

Cleaning first splits the input, then applies the selected operations in the fixed order below. Trimming or collapsing whitespace can make formerly different items equal before duplicate removal. Deduplication keeps the first retained spelling, and sorting or reversing happens afterward. Changing checkbox order on the screen does not change the processing sequence.

Output separator is inserted only after these operations. It can add whitespace or increase output length, and its characters are not cleaned again. Reverse lines reverses the retained item list rather than reversing characters inside an item. None of these operations reconstructs removed occurrences or the original separators.

The cleanup pipeline and why its order matters
StageWhen enabled
SplitCreate items using the input separator
Trim each lineRemove matched whitespace at each item’s edges
Collapse repeated spacesReplace matched non-CR/LF whitespace runs with one ASCII space
Remove empty linesDiscard items whose trimmed value is empty
Remove duplicate linesKeep the first item for each exact or locale-lowercase key
Line orderKeep, sort or reverse the retained items
JoinCheck final size, then join items with the chosen output separator

ECMAScript: String values, splitting, trimming and case mappingECMAScript: Stable Array.prototype.sort

A worked example: defaults, case matching and ordering

Take the exact input apple\nbanana\napple\n\nBanana\nitem 10\nitem 2 . In these examples, \n is display notation for a real line break; enter seven physical lines, including the empty fourth line. With line-break input and output, the default cleanup produces five items. It removes outer spaces, the empty item and the second exact apple, but keeps both banana and Banana.

Each table row starts from that same original input, rather than from the preceding row’s result. For the sorting example select English as Text language. Turning on Ignore case removes the later Banana while retaining the spelling of the earlier banana. Natural sorting places the numeric run in item 2 before the one in item 10.

A worked example: defaults, case matching and ordering
Settings relative to defaultsOutput in escaped display notationOutput items
No changesapple\nbanana\nBanana\nitem 10\nitem 25
Ignore case onapple\nbanana\nitem 10\nitem 24
Ignore case on; Sort A → Z (natural numbers); Englishapple\nbanana\nitem 2\nitem 104
Reverse lines; Ignore case offitem 2\nitem 10\nBanana\nbanana\napple5

ECMAScript Internationalization: Collator and CompareStringsUnicode LDML: Collation options and numeric ordering

Trim, collapse spaces and remove empty items

Trim each line uses JavaScript’s whitespace rules at the edges of each item. It does not remove spaces between words. Collapse repeated spaces replaces a matched run with a single ASCII space, including a single tab or nonbreaking space. CR and LF are excluded from that collapse operation, but other matched whitespace such as Unicode line or paragraph separators can become spaces. This is broader than changing only repeated ordinary spaces.

Remove empty lines tests whether an item becomes empty after trimming, even when Trim each line itself is off. Thus a whitespace-only item can be removed while outer spaces on a nonempty item remain. The cleaner is not a universal invisible-character remover: zero-width spaces and other characters outside the relevant JavaScript whitespace set can remain. Do not apply whitespace changes blindly to code, signatures or identifiers.

Trim, collapse spaces and remove empty items
Input item / itemsSelected changeResult
alpha Trim onalpha
alpha\tbeta with a real tabCollapse on, within one itemalpha beta
alpha plus a spaces-only itemTrim off; Remove empty onRetains outer spaces on alpha; discards the empty item

ECMAScript: String values, splitting, trimming and case mapping

Remove duplicates while retaining the first spelling

Without Ignore case, duplicate keys are the exact transformed strings after the earlier whitespace steps. With Ignore case, each key is lowercased using Text language, but the displayed item is not lowercased. The first transformed item for a key survives. Turning off Remove duplicate lines preserves all occurrences, even if Ignore case is selected.

For input Apple\napple\nApple, default duplicate removal keeps Apple\napple; turning on Ignore case keeps only Apple. A trailing space can distinguish items when trimming is off, and collapsing internal whitespace can merge keys when it is on. This is literal item deduplication, not a search for repeated words anywhere in a paragraph or a fuzzy comparison of records.

ECMAScript: String values, splitting, trimming and case mappingECMAScript Internationalization: Collator and CompareStrings

Why visually identical lines can remain separate

Unicode can represent similar-looking text with different sequences. For example, the single character é and an e followed by a combining acute accent, written as e\u0301 in Unicode escape notation, can produce different deduplication keys. The cleaner does not apply NFC, NFD, NFKC or NFKD normalization. Canonical equivalence during sorting does not silently normalize the stored items.

Ignore case uses locale-aware lowercase keys, not full Unicode case folding, accent removal, transliteration or confusable-character detection. It does not make every visually similar spelling equivalent; for example, ß and ss are not merged merely by lowercasing. If another system requires normalized identifiers, use its explicit normalization policy and review the changes separately.

ECMAScript: String values, splitting, trimming and case mappingECMAScript Internationalization: Collator and CompareStringsUnicode Standard Annex #15: Unicode Normalization Forms

Natural sorting, selected language and stable ties

Sort A → Z and Sort Z → A use the browser’s Intl.Collator for Text language, with numeric comparison enabled. Digit runs let item2 sort before item10. This is not a numeric-data parser: it does not promise correct arithmetic ordering for signed decimals, currency, dates, IP addresses or software versions. Choose the ordering rules required by the receiving system when those values carry special meaning.

Ignore case selects accent-sensitive, case-insensitive collation for sorting; otherwise the comparison is variant-sensitive. Browser locale data and versions can affect ordering. Distinct strings may compare equal, including canonically equivalent forms, while duplicate matching still keeps them separately. Stable sorting retains the incoming order of tied retained items. Descending order therefore need not reverse the internal order of every tie, and Reverse lines is a different operation.

Natural sorting, selected language and stable ties
Line orderEffect on the retained list
Keep original orderPreserves the order surviving cleanup
Sort A → Z (natural numbers)Ascending locale comparison with digit-run ordering
Sort Z → A (natural numbers)Descending comparison; ties preserve their incoming order
Reverse linesReverses retained item positions without comparing their text

ECMAScript Internationalization: Collator and CompareStringsUnicode LDML: Collation options and numeric orderingECMAScript: Stable Array.prototype.sort

Convert plain comma lists; use a CSV parser for quoted records

For a simple list such as red, blue,red, choose Comma input, Line breaks output and the default trim/deduplicate settings. The result is red\nblue, where the display notation represents two lines. The same approach changes semicolon, tab or custom-separated plain lists into another separator style.

Quoted CSV is a different format. The input "a,b",c splits into three items on its literal commas; quotation marks do not protect the comma inside quotes. Selecting a comma output separator likewise adds no CSV quoting or escaping for commas, quotes or embedded newlines already inside items. Parse structured records with a CSV-aware workflow before treating fields as a simple list.

  • Inspect the source for quoted delimiters or multiline fields before choosing Comma.
  • Use Custom for an actual delimiter such as ||, rather than a regular-expression pattern.
  • Check the receiving format’s escaping requirements before copying the joined output into a data file.

ECMAScript: String values, splitting, trimming and case mapping

Input limits, output expansion and separator length

Input and output each allow up to 1,000,000 UTF-16 code units. A UTF-16 unit is not always one visible character: an emoji or combined grapheme can occupy multiple units. Custom separators allow 1–20 units independently for input and output. Local text files must be strictly smaller than 4,000,000 bytes, and their decoded text must separately fit the input ceiling.

Before joining, the cleaner adds the lengths of retained items and the output separators between them. If the total exceeds the output limit, it rejects the result rather than truncating it. A long custom output separator can grow a short-delimited input even after some items are removed. Select a shorter output separator or a smaller portion of the source, and keep the intended meaning of the list intact.

Input limits, output expansion and separator length
BoundaryWhat it measures
Input / output: at most 1,000,000UTF-16 code units in the respective text
Custom separator: 1–20UTF-16 units, not grapheme clusters
Text file: below 4,000,000Bytes before text decoding; exactly this size is rejected

ECMAScript: String values, splitting, trimming and case mappingW3C File API: Blob text reading

Read item counts and recognize a completed empty result

The result notice reports split input items and retained output items. With comma, tab or custom input these are not necessarily physical lines: one item can contain line breaks. The difference between the two counts includes empty-item removal and duplicate removal together, so it is not a duplicate-only statistic. The output byte badge measures the encoded UTF-8 text, not the UTF-16 limit or the item count.

Clean text is disabled for an empty or whitespace-only editor. A nonblank input can still clean to an empty result: for example, ,, with Comma input and Remove empty lines on has three empty input items and no output items. Copy result and Download are available after that successful empty result. An empty result is different from a missing result after an error or a settings change.

Text files, downloads and using the result again

File loading reads text as UTF-8 through File.text(). A leading UTF-8 BOM is removed, malformed UTF-8 can become replacement characters, and browser text editing can affect line endings. This is not an encoding detector or a byte-preserving binary-file conversion. Check imported text before cleaning, especially if your source uses a legacy encoding or meaningful whitespace.

Download writes UTF-8 plain text named toolmellow-result.txt. Copy result uses the clipboard when allowed, with a manual-copy fallback when access fails. Use result as input replaces the editor text for a deliberate second pass; it is not automatic chaining. Undo/Redo concerns editor history, rather than restoring every cleanup setting or recovering the source from an already exported cleaned file.

W3C File API: Blob text readingWHATWG Encoding: UTF-8 decoding and replacement

Clear, local processing and ordinary website requests

Clear removes editor input, the result, errors, search/replacement text and editor history. It retains Text language, separator choices, Line order, cleanup checkboxes and Wrap lines. Downloaded files and clipboard contents are outside that clear action. Closing or refreshing the page discards the workspace’s in-memory state; clearing is not a forensic-erasure guarantee.

Cleaning runs in a local worker and does not upload pasted text or local-file contents to a cleaning server. You can cancel a running job; the worker is terminated when cancelled or when the job exceeds its time limit. Loading the public website still sends ordinary hosting metadata and Google Analytics page-visit requests. Local text processing does not establish anonymity or zero network logging.

WHATWG HTML: Workers and their lifecycle

Embed text cleaner with the ToolMellow backlink

Select the website language and use the integration section beneath the cleaner workspace to copy or download the supplied HTML snippet. Paste it into a webpage or publishing platform that permits its iframe and sizing script. Keep the visible backlink to the corresponding localized ToolMellow tool page; the supplied link includes nofollow and noopener.

Check the embed on a narrow screen and verify that its host permits browser workers and the required clipboard interaction. It exposes the same interactive controls, limits and local processing as the tool page. Its height messages help size the iframe; they are not an API for sending text to an automated cleaning service.

Common questions

How do I clean a text list?

Paste or load text, choose Text language and the input/output separators, review cleanup and Line order, then press Clean text. Defaults trim edges, remove empty items and exact duplicates while retaining original order.

How do I remove duplicate lines?

Use Line breaks input and enable Remove duplicate lines. Matching uses the item after earlier whitespace steps. It keeps the first retained spelling; Ignore case changes the key to selected-locale lowercase.

How do I sort numbers naturally in text?

Choose Sort A → Z (natural numbers) and the intended Text language. Digit-run comparison places item2 before item10. It does not parse signed decimals, dates, addresses or version strings as specialized data.

How do I remove blank lines without trimming other lines?

Turn on Remove empty lines and turn off Trim each line. Whitespace-only items are still removed, while outer spaces on nonempty items remain unless another selected operation changes them.

How do I convert commas to line breaks?

For a plain list, select Comma input and Line breaks output. Review trim, empty and duplicate settings before cleaning. Quoted CSV fields need a CSV-aware parser because commas inside quotes still split here.

Does Ignore case change the output text?

It does not lowercase the displayed items. It changes duplicate keys and sorting sensitivity; the first transformed item’s spelling is retained. It is locale lowercase comparison, not full Unicode case folding.

What is the difference between trim and collapse spaces?

Trim removes matched whitespace at item edges. Collapse replaces runs of matched whitespace other than CR/LF with one ASCII space, including a single tab or nonbreaking space. They run before removal and sorting.

Why do visually identical lines remain?

Literal or lowercase keys can differ for distinct Unicode sequences. The cleaner does not normalize Unicode, strip all invisible characters, remove accents or perform fuzzy matching, even when sorting considers two strings tied.

Can a custom separator be a regular expression?

No. It is an exact literal string from 1 to 20 UTF-16 units. Characters such as .* and typed backslash-n are not regex syntax or decoded escape sequences.

Why is the cleaned output too large?

The final size includes retained item text and every inserted output separator. A long separator can expand the output past one million UTF-16 units. The tool rejects the result; choose a shorter separator or a smaller input.

Does Reverse lines reverse every character?

No. It reverses the retained item list after trimming, empty removal and deduplication as selected. Characters inside each item keep their order.

What does the result’s item count mean?

It counts input items after splitting and output items after cleanup. In comma/custom mode, an item can contain physical line breaks. The count difference combines all removals and is not a duplicate-only count.

Is my text uploaded for cleaning?

The cleaning operation processes pasted and loaded text locally in a browser worker. The website still makes ordinary hosting and analytics requests; local processing does not mean anonymity or zero logging.

Can I embed the text cleaner on my website?

Use the localized HTML snippet beneath the workspace, retain the visible ToolMellow backlink and check iframe/script/worker support. The embed retains the same controls and limits; it is an interactive workspace rather than a remote processing API.

Sources and further reading

Put it into practice.

Related guides