Find Duplicate Files with Python: Reclaim Gigabytes of Disk Space in One Run

Hong Kong users swap laptops, phones and external drives faster than ever, and every migration leaves behind a pile of duplicate files: the same photo lives in "Photos", "Downloads" and "Backup" at once; the same contract arrived by email, by WhatsApp and was saved again by hand. Over time the drive fills up with "one more copy of the same thing", and clearing it feels impossible.

This guide shows you how to build a duplicate-file finder in Python: run it once and it scans an entire folder (subfolders included), finds files with identical content and lists them in a report so you can decide what to delete. No extra software needed — Python's built-in tools are enough.

Why can't you judge duplicates by filename?

The obvious approach is comparing filenames, but it is unreliable: the same photo may be IMG_2031.jpg in one folder and holiday.jpg in another, while two completely different documents can both be called "quote.docx". To be accurate you must compare file content. The trick is to use hashlib to compute each file's "fingerprint" (SHA-256 hash) — identical content gives an identical hash, and a single differing bit changes it completely.

Step 1: Think before you write

Start with one scope (say your "Documents" folder) instead of scanning the whole machine at once
The script only finds and reports — it never deletes anything, so your data stays safe
Small files are numerous, so filter by file size first and confirm with hashes; it is dramatically faster

Step 2: The full script

Create a new file called dedupe.py (Notepad is fine) and paste the code below:

import os, hashlib
from collections import defaultdict
import pandas as pd

def file_hash(path, chunk=65536):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        while (block := f.read(chunk)):
            h.update(block)
    return h.hexdigest()

SCAN_DIR = "D:/Documents"   # the folder you want to scan

# 1. Group by file size first (fast)
by_size = defaultdict(list)
for root, _, files in os.walk(SCAN_DIR):
    for name in files:
        p = os.path.join(root, name)
        try:
            by_size[os.path.getsize(p)].append(p)
        except OSError:
            pass

# 2. Hash only files that share a size (accurate)
by_hash = defaultdict(list)
for size, paths in by_size.items():
    if len(paths) < 2:
        continue
    for p in paths:
        try:
            by_hash[file_hash(p)].append(p)
        except OSError:
            continue

# 3. Write the report
rows = []
total = 0
for h, paths in by_hash.items():
    if len(paths) > 1:
        total += (len(paths) - 1) * os.path.getsize(paths[0])
        for p in paths:
            rows.append({"Hash": h[:8], "File": p,
                         "Size(KB)": round(os.path.getsize(p)/1024, 1)})

pd.DataFrame(rows).to_excel("duplicates.xlsx", index=False)
print(f"Found {len(by_hash)} duplicate groups, about {total/1024/1024:.1f} MB recoverable")

Line by line

file_hash() — computes a SHA-256 fingerprint and reads the file in 64KB blocks, so even multi-gigabyte files won't blow up memory.
os.walk(SCAN_DIR) — walks the target folder and every subfolder automatically.
by_size — groups by file size first; different sizes can never be duplicates, so they are skipped and you save a lot of time.
len(paths) < 2 → continue — a unique size means no hashing needed at all.
by_hash — uses the fingerprint as the key, so every group in it holds files with byte-identical content.
h[:8] — the report shows only the first 8 characters of each hash for easy eyeballing.
to_excel() — one row per file, saved to Excel so you can filter and sort.

Step 3: Run it

python dedupe.py

When it finishes, a duplicates.xlsx appears in the same folder listing every duplicate and how much space you could recover. Sort it in Excel to see which copy is the original and which are the extras, then decide yourself what to delete.

Five tips for safe cleanup

Back up before you clean — export the file list or take a backup before removing anything.
Keep the most useful copy — usually the one in the handiest location with the clearest filename.
Never delete from the system drive — do not use this method on Windows or application files.
Watch the size column — the script already filters, but the "Size(KB)" column helps you spot the biggest space hogs first.
Run it regularly — a quarterly scan is far easier than clearing hundreds of gigabytes in one go.

Summary

Python Tutorial: a few dozen lines of code give you a duplicate-file finder that scans a whole folder in seconds and hands back gigabytes — sometimes tens of gigabytes — of space. Remember the two orders that matter: filter by size first, confirm with hashes second; and the script only finds, never deletes, so the decision to clean stays entirely with you. To go further, add a step that moves duplicates into a "To delete" folder so you can review them before clearing in one sweep.