Remove Duplicate Lines with Python: Beginner Script

If you have a text file full of repeated keywords, copied prompts, exported notes, or messy research lines, Python can clean it in a few seconds. In this guide, I will show you a beginner-friendly script that removes duplicate lines from a text file, keeps the first copy, and can make a backup before editing the original file.

This is a small project, but it teaches a very useful automation pattern: read a file, compare each line, keep only what matters, and write a cleaner result. If you already liked the practical style of my free keyword research with Python guide, this is another simple script you can actually reuse.

What This Python Script Does

The goal is simple: give the script a text file, and it creates a cleaned file where repeated lines appear only once. The important detail is order. Many quick duplicate-removal methods use sorting, but sorting can rearrange your list. That is a problem if your file is a prompt list, a keyword list, a study outline, or a log where the first appearance matters.

The script in this post uses a set named seen to remember lines it has already found. When it reads a new line for the first time, it keeps it. When it sees the same line again, it skips it. This gives you a clean file while preserving the original order of the first occurrence.

It also includes two practical options:

• --ignore-case treats AI notes and ai notes as the same line.
• --trim-spaces treats python script and python script as the same line.

When Should You Use It?

This is useful whenever you collect text from many places and later notice repeated entries. For example, you might copy keyword ideas from AI tools, save prompt variations, merge class notes, collect URLs, export a list from a spreadsheet, or paste research points into one file. Removing duplicates manually is boring, and online tools may not be ideal if the file contains private notes.

A local Python script is better when you want repeatable cleaning, no account signup, and no upload. It also teaches a real file-handling skill you can reuse in bigger automations later.

Method	Best For	Main Risk	Keeps Order?
Manual editing	Very small files	Easy to miss repeated lines	Yes, if careful
Online duplicate remover	Quick public text	Not ideal for private notes	Depends on the tool
Sort then remove duplicates	Alphabetical lists	Changes original order	No
Python script below	Reusable cleaning	Must run one command	Yes

The Beginner Python Script

Create a file named remove_duplicate_lines.py, then paste this code:

from argparse import ArgumentParser
from pathlib import Path
from shutil import copy2


def clean_line(line: str, ignore_case: bool, trim_spaces: bool) -> str:
    if trim_spaces:
        line = line.strip()
    else:
        line = line.rstrip("\r\n")

    if ignore_case:
        line = line.casefold()

    return line


def remove_duplicate_lines(
    input_file: Path,
    output_file: Path,
    ignore_case: bool = False,
    trim_spaces: bool = False,
) -> tuple[int, int]:
    seen = set()
    kept_lines = []
    original_lines = input_file.read_text(encoding="utf-8").splitlines()

    for line in original_lines:
        key = clean_line(line, ignore_case, trim_spaces)
        if key not in seen:
            seen.add(key)
            kept_lines.append(line.strip() if trim_spaces else line)

    output_file.write_text("\n".join(kept_lines) + "\n", encoding="utf-8")
    return len(original_lines), len(kept_lines)


def main() -> None:
    parser = ArgumentParser(
        description="Remove duplicate lines from a text file while keeping the first copy."
    )
    parser.add_argument("input_file", help="Path to the text file you want to clean.")
    parser.add_argument("-o", "--output", help="Path for the cleaned file.")
    parser.add_argument("--in-place", action="store_true", help="Clean the original file after creating a .bak backup.")
    parser.add_argument("--ignore-case", action="store_true", help="Treat Apple and apple as the same line.")
    parser.add_argument("--trim-spaces", action="store_true", help="Ignore spaces before and after each line while comparing.")
    args = parser.parse_args()

    input_path = Path(args.input_file)

    if not input_path.exists():
        raise SystemExit(f"File not found: {input_path}")

    if args.in_place:
        backup_path = input_path.with_suffix(input_path.suffix + ".bak")
        copy2(input_path, backup_path)
        output_path = input_path
    else:
        output_path = Path(args.output) if args.output else input_path.with_name("cleaned-" + input_path.name)

    total, kept = remove_duplicate_lines(
        input_path,
        output_path,
        ignore_case=args.ignore_case,
        trim_spaces=args.trim_spaces,
    )

    removed = total - kept
    print(f"Read {total} lines")
    print(f"Kept {kept} unique lines")
    print(f"Removed {removed} duplicate lines")
    print(f"Saved cleaned file to: {output_path}")

    if args.in_place:
        print(f"Backup saved to: {backup_path}")


if __name__ == "__main__":
    main()

The code uses Python's standard library only. The official pathlib documentation explains how Path reads and writes files, the argparse documentation covers command-line options, and the shutil documentation describes copy2, which the script uses for backups.

Worked Example: Clean a Prompt List

Here is a messy file named sample_duplicate_lines.txt:

chatgpt prompt
gemini prompt
chatgpt prompt
python script
  python script
AI notes
ai notes
gemini prompt

Run this command in the same folder:

python remove_duplicate_lines.py sample_duplicate_lines.txt --ignore-case --trim-spaces

In my test file, the script printed this result:

Read 8 lines
Kept 4 unique lines
Removed 4 duplicate lines
Saved cleaned file to: cleaned-sample_duplicate_lines.txt

The cleaned file becomes:

chatgpt prompt
gemini prompt
python script
AI notes

Notice what happened. The first version of each useful line stayed in place. Later repeats were removed. Because I used --ignore-case, AI notes and ai notes counted as the same idea. Because I used --trim-spaces, the indented python script line counted as a duplicate too.

How to Use the Backup Mode

By default, the script creates a new file and leaves your original file alone. That is the best beginner habit. When you trust the script, you can clean the original file directly with --in-place:

python remove_duplicate_lines.py keywords.txt --ignore-case --trim-spaces --in-place

Before editing keywords.txt, the script creates a backup named keywords.txt.bak. If something looks wrong, you still have the original text. That small backup step is the difference between useful automation and stressful automation.

Step-by-Step Workflow

1. Save your messy lines in a plain text file, one item per line.
2. Put remove_duplicate_lines.py in the same folder.
3. Run the script without --in-place first.
4. Open the cleaned file and check the result.
5. If the result is correct, use the cleaned file or rerun with --in-place.

For keyword lists, prompt collections, and study notes, I usually recommend using both --ignore-case and --trim-spaces. For code snippets or data where spacing matters, skip --trim-spaces so the script does not change the line text.

Common Mistakes to Avoid

Do not sort the file first unless alphabetical order is actually what you want. Sorting removes the natural order of your notes.

Do not use in-place mode first on an important file. Test with a new output file, then use in-place mode later.

Do not trim spaces blindly if your file contains code, indentation, or formatted text. Trimming spaces is great for keyword lists, but not for Python files or Markdown outlines where spaces may carry meaning.

Do not upload private notes to random cleaning tools if you can solve the job locally. A small script is slower the first time, but faster every time after that.

FAQ

How do I remove duplicate lines from a text file with Python?

Read each line, store a comparison version in a set, and write only the first line that has not appeared before. The script above does this while keeping the original order.

How do I keep the original order when removing duplicates?

Do not sort the file. Loop through the file from top to bottom and keep the first copy of each line. A set is used only to check whether the line was already seen.

Can Python remove duplicate lines without changing my original file?

Yes. The default command in this guide writes a new cleaned file. Use --in-place only after checking the output, and even then the script creates a .bak backup.

Should I ignore uppercase and lowercase duplicates?

For keyword lists, prompts, URLs, and notes, usually yes. Use --ignore-case. For data where capitalization changes the meaning, leave it off.

Final Takeaway

This is a small Python project, but it is a useful one. You learn file reading, file writing, command-line arguments, backups, and a simple deduplication pattern. More importantly, you get a script that can clean real lists without sending your text anywhere.

If your blog, school, or AI workflow creates messy text files, keep this script nearby. It turns a boring cleanup task into one command.