A simple duplicate files checker

Dr. Rahul Remanan

CEO, Moad Computer

An example implementation of duplicate file detection using Python. This could be used as the backbone for a de-duplicated file system.

Open in Kaggle

Import libraries

Open in Kaggle

Compute file hashes

The file hashes are computed for a specified chunk size using either SHA256 or Blake cryptographic functions using the hashlib python library.

Open in Kaggle

Detect file duplicates

Creates a dictionary output with the cryptographic hash as the key and a list of files that share that specific cryptographic hash as the value.

Open in Kaggle

Find duplicates in a list of files

​Finding the duplicate files can be performed by simply iterating over all the keys in the file comparison dictionary, looking for values with a list size of more than 1.

Open in Kaggle

References:

1.  Explainer on Python super() by Raymond Hettinger