MyClaw
HomeProductsUse CasesPricingSkillsResourcesCompany
HomeUse CasesPricingSkills
Sign In
Home
ProductsOpenClaw HostingHermes Agent Hosting
Use CasesPricingSkills
ResourcesCompareTutorialLearnBlogUpdatesResourcesCommunity
CompanyAboutSupport
Sign In
MyClaw.ai

Your AI agent, already running.

Product

OpenClaw HostingHermes Agent HostingUse CasesPricingCompareAlibaba Cloud

Resources

BlogUpdatesSkillsModelsSupportCommunityCapybara

Legal

TermsPrivacyCookieGDPRContact

© 2023 Cubo World Inc. All rights reserved.

← Back to Skills
CLI Utilities
🗂️

File Deduplicator

Find and remove duplicate files intelligently using content hashing, size, or name comparison. Save storage space and keep your file system clean.

michael-laffinv1.0.0
Connecting to VM...
Connecting to VM...
npx clawhub@latest install file-deduplicator
7Stars
3.2kDownloads
3Current Installs
14All-time Installs
📦
v1.0.0Version
View Source

File Deduplicator Skill Overview

File Deduplicator scans your directories for identical or similar files using content hashing, size comparison, or name matching, then helps you remove or archive the duplicates safely. Install it to reclaim wasted storage space, reduce backup bloat, and keep your file system clean — without risking accidental data loss thanks to dry-run previews, archive mode, and full action logging.

How to Use It

Step 1: Run in your terminal or install this skill on MyClaw

npx clawhub@latest install file-deduplicator
or

Click the Install button at the top of this page for one-click setup

When to Use File Deduplicator

Best Fit

  • You have large document, photo, or download folders that have accumulated duplicate files over time and want to reclaim disk space.
  • You manage project directories and need to identify redundant source files or build artifacts across multiple locations.
  • You want to audit and clean up backup drives without permanently deleting anything (archive mode keeps files recoverable).
  • You need a safe, preview-first workflow — dry-run mode shows exactly what would be removed before any action is taken.

When Not to Use

  • You need to detect near-duplicate images by visual similarity (image similarity detection is listed as a future roadmap item, not currently supported).
  • You need to deduplicate files stored on cloud services like S3 or Google Drive (network and cloud drive support is not yet implemented).
  • You need automated scheduled scans (scheduling is on the roadmap but not currently available).

Key Features

Content-Based MD5 Hashing

Identifies exact duplicate files by comparing their content rather than just names or sizes. This means renamed copies of the same file are still detected, making it the most reliable method for documents and code.

Multiple Detection Methods

Choose between content hashing (most accurate), size-based comparison (fastest), or name-similarity detection (good for finding versioned files like file_v1, file_v2). Each method suits different use cases and performance needs.

Dry-Run Preview Mode

Run any scan or removal operation in dry-run mode to see exactly which files would be affected before committing to any changes. Ideal for verifying results on unfamiliar directories.

Archive Instead of Delete

Rather than permanently deleting duplicates, move them to a configurable archive folder. Files remain fully recoverable, and all moves are logged for audit and undo reference.

Configurable Safety Controls

Set a size threshold so files above a certain size (default: 10MB) are never automatically removed. Whitelist critical directories to ensure they are never scanned or modified.

Detailed Reporting

Each scan returns duplicate counts, total files scanned, bytes wasted, potential space savings, and scan duration. A log file records every action taken for recovery and audit purposes.

Use Cases

Photo and Download Folder Cleanup

Scan accumulated photo libraries and download folders to surface duplicate images, installers, and media files. Keep the newest copy and delete or archive the rest to recover gigabytes of space.

Document Version Control

Find duplicate PDFs, Word documents, and reports spread across multiple folders. Archive older versions and keep the latest, preventing version confusion and reducing backup size.

Development Project Maintenance

Identify duplicate source files and build artifacts across project directories. Exclude node_modules and .git automatically, then clean up what remains to keep repositories lean.

Backup Drive Optimization

Analyze backup drives for redundant copies of the same files. Use the space savings estimate to understand how much storage is being wasted before committing to any removals.

Requirements

  • No external API keys or third-party service accounts required.
  • Read and write filesystem permissions on all directories to be scanned.
  • Sufficient disk space at the archive destination path when using action: 'move' mode.
  • Files must be locally accessible (network drives and cloud storage are not currently supported).
Connecting to VM...
npx clawhub@latest install file-deduplicator
7Stars
3.2kDownloads
3Current Installs
14All-time Installs
📦
v1.0.0Version
View Source

Reviews

0 reviews

Log in to write a review

No reviews yet. Be the first to share your experience!