Remove Duplicate Lines

Remove duplicate lines from text while keeping the first occurrence and original order. Compare lines with optional case sensitivity and whitespace trimming.

Input Text

Enter text with one item per line. Duplicate lines will be removed while preserving the original order.

0 / 100,000 characters

About Remove Duplicate Lines

Remove repeated lines from text while preserving the first occurrence and original order. Choose case-sensitive or case-insensitive comparison, trim surrounding whitespace, and optionally remove empty lines. This browser-based text utility is useful for cleaning lists, URLs, identifiers and other line-based data.

How to Use Remove Duplicate Lines

Follow these steps to clean a list or text file:

  1. Paste your text into the Input Text area, using one item per line.
  2. Choose whether matching should be case sensitive.
  3. Enable Trim whitespace if spaces and tabs around each line should be ignored.
  4. Enable Remove empty lines if blank lines should also be removed.
  5. Click Remove Duplicates to process the text.
  6. Review the result and line statistics, then select Copy to copy the cleaned text.

What Does Remove Duplicate Lines Do?

The tool compares lines in the input text and keeps only the first occurrence of each unique line. Later occurrences are removed. The remaining lines stay in their original order; the tool does not sort them alphabetically.

For example, if a list contains apple, banana, apple and orange, the result contains apple, banana and orange.

BEFORE

apple
banana
apple
orange

AFTER

apple
banana
orange

Case-Sensitive vs. Case-Insensitive Matching

Case sensitivity determines whether uppercase and lowercase letters are considered different when comparing lines.

ModeExampleResult
Case sensitiveApple / appleTreated as different lines
Case insensitiveApple / appleTreated as duplicates; the first occurrence is kept

Case-insensitive comparison uses JavaScript lowercase conversion. It is not a complete language-specific or locale-aware text comparison.

Trim Whitespace

When Trim whitespace is enabled, whitespace at the beginning and end of each line is removed before comparison. This helps identify lines that contain the same text but have different surrounding spaces or tabs.

apple

  apple

apple  

With trimming enabled, the three lines above become identical. Only the first occurrence is retained. The trimmed version is also what appears in the output.

If Trim whitespace is disabled, surrounding whitespace is preserved and is considered part of the line when checking for duplicates.

Remove Empty Lines

Enable Remove empty lines to discard blank lines from the output. If Trim whitespace is also enabled, lines containing only whitespace become empty after trimming and are removed as well.

If this option is disabled, empty lines are treated like other lines. Only the first empty line is kept when duplicate removal is applied.

Understanding the Line Statistics

After processing, the tool displays separate counts to explain what changed:

  • Original Lines: Number of lines in the input before processing.
  • Unique Lines: Number of lines remaining in the result.
  • Duplicate Lines Removed: Number of repeated lines removed during comparison.
  • Empty Lines Removed: Number of blank lines discarded by the empty-line option.
  • Total Lines Removed: Sum of duplicate lines and empty lines removed.

Line counts are based on splitting the input at line endings. A trailing newline creates an additional empty line for counting purposes.

When Should You Remove Duplicate Lines?

Removing repeated lines can help prepare text for further processing and make lists easier to review. Common examples include:

  • Cleaning lists of URLs or domains.
  • Removing repeated identifiers and reference values.
  • Cleaning copied spreadsheet or text data.
  • Removing duplicate email addresses from a list.
  • Cleaning repeated entries in text-based logs.
  • Preparing unique values for importing or analysis.

Original Order Is Preserved

This tool keeps the first occurrence of each unique line and removes subsequent occurrences. It does not alphabetically sort or rearrange the remaining lines. This is useful when the original sequence of entries matters. If you want to arrange the remaining lines alphabetically, numerically, or by length, use Sort Lines.

Supported Line Endings

The tool normalizes common line-ending formats before processing. It supports Windows-style CRLF, Unix-style LF and carriage-return-only CR line endings. The output uses standard LF line endings.

Is My Text Uploaded to a Server?

No. Duplicate removal is performed directly in your browser using JavaScript. The input text is not sent to a KitDevTools server for processing.

Input Limit and Browser Performance

The input field supports up to 100,000 characters. Processing takes place locally in your browser, and the time required can vary depending on the amount of text and your device.

For very large lists, consider processing smaller sections if your browser becomes slow.

Frequently Asked Questions

Does the tool sort my lines alphabetically?

No. It preserves the order of the first occurrence of each unique line.

Why are two lines treated as duplicates?

The comparison depends on the selected options. Case-insensitive matching ignores letter case, while Trim whitespace removes surrounding whitespace before comparison.

Can I keep empty lines?

Yes. Disable Remove empty lines. The first empty line will be retained, while later empty lines are removed as duplicates.

Does trimming change the output?

Yes. When Trim whitespace is enabled, leading and trailing whitespace is removed from every line before it is added to the result.

What happens if all lines are removed?

The tool still displays the result section and statistics, along with a message indicating that no lines remain.

Your privacy matters

We use optional analytics cookies to understand how our tools are used and improve your experience. These controls apply to Google Analytics only, not advertising preferences. Privacy Policy