Remove duplicate lines from text

Keep the first occurrence of each physical line, with optional trimming, case folding, empty-row removal, and final sorting.

freeworks offlinenothing uploaded
ToolLine Deduplicator
Input
Output
—
Put this on your own site

The frame below runs the same code as this page, in the reader's own browser. Nothing is sent to us, and nothing is sent to you.

Pick a dark background and the text and panels follow it, so the frame stays readable on a dark page.

Preview

How it works

Rows become Set keys after optional trim and lowercase conversion, while the first surviving display spelling is stored separately. Disabling order preservation locale-sorts those unique rows only after duplicates are removed.

  • Exact case, trimmed rows, insertion order, and retained empty rows form the initial behavior.
  • Insertion order ensures a duplicate never replaces the spelling or position of its first occurrence.

Worked example

Remove Duplicates
Remove repeated lines from a list while preserving order
Input
											apple
banana
apple
cherry
banana
date
Output
												apple
banana
cherry
date

When to use this

Mailing exports remove repeated addresses, batch keyword files collapse duplicate rows, and log analysis retains one instance of each exact message.

Edge cases

  • Case-insensitive Apple followed by apple emits only Apple because the first display spelling wins.
  • Trimming makes a padded row equal to its unpadded form and removes that padding from output.
  • A trailing newline adds an empty source row; ignoring empties removes it but still counts it among removals.

References