How I Built Secret-Scrub: A Zero-Dependency Pre-Commit Secret Scanner with Shannon Entropy Analysis

Every developer has had that split-second wave of cold panic:

You type git commit -m "update config" && git push, and a second later you realize:
"Wait... did I just push my .env file or hardcoded AWS secret key to a public GitHub repo?"

Even if you delete the commit or force-push, git history scanners and malicious scrapers detect leaked credentials within seconds of hitting GitHub. Once an API key is leaked, it must be rotated immediately.

To protect codebases before secrets ever leave your local machine, I built Secret-Scrub — a blazing-fast, zero-dependency Node.js CLI scanner with a 1-click Git pre-commit hook installer and Shannon entropy analysis.

Architecture: Dual-Layer Detection Engine

Regex alone is never enough for secret scanning. Vendor formats change, and raw credentials don't always carry known prefixes. Secret-Scrub combines two complementary inspection layers:

[Target File / Staged Diff]
            │
            ▼
┌──────────────────────────────────────────────┐
│ Layer 1: High-Fidelity Signature Patterns     │
│ └── AWS (AKIA...), GitHub (ghp_...), Stripe,  │
│     OpenAI (sk-...), Private Keys (RSA/SSH)  │
└──────────────────────┬───────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────┐
│ Layer 2: Shannon Entropy Analysis             │
│ └── H(X) = -Σ P(x) * log2(P(x))               │
│     Calculates character randomness to catch │
│     unstructured API keys & high-entropy pwds│
└──────────────────────┬───────────────────────┘
                       │
                       ▼
        [Exit Code 1: Block Git Commit]

1. Vendor Signature Matching (18+ Cloud Providers)

Scans for well-known token formats including:

• AWS Access Keys (AKIA...)
• GitHub Personal Access Tokens (ghp_..., gho_...)
• Stripe Secret & Restricted Keys (sk_live_..., rk_live_...)
• OpenAI API Keys (sk-...)
• Slack Webhooks, Google API Keys, JWT tokens, and PEM Private Keys.

2. Shannon Entropy Math for Unknown Secrets

Attackers and developers frequently use unstructured high-randomness tokens that have no vendor prefix. Secret-Scrub measures information entropy:

$$H(X) = -\sum_{i=1}^n P(x_i) \log_2 P(x_i)$$

High entropy strings (e.g., dGhpcy1pcy1hLXJhbmRvbS1zZWNyZXQ=) score high randomness and are flagged even without a pattern match, while normal camelCase code variables score low and are ignored.

Git Workflow Integration (--staged)

Scanning thousands of files on every commit slows down developers. Secret-Scrub solves this with --staged mode:

npx secret-scrub --staged

Instead of scanning entire directories, it queries git diff --cached and only inspects files currently staged for commit. This takes less than 40 milliseconds, making it imperceptible in day-to-day coding.

1-Click Pre-Commit Hook Installer

You don't need complex external hook runners. You can install native Git protection directly:

npx secret-scrub install-hook

This automatically configures .git/hooks/pre-commit. If anyone on your team attempts to commit a secret, the commit is automatically aborted locally before it can be pushed.

Quick Start

Test your current directory or staged commits immediately:

# Scan whole directory
npx secret-scrub .

# Scan only staged git files
npx secret-scrub --staged

# Machine-readable JSON output for CI pipelines
npx secret-scrub . --format json

Source Code & Community

Secret-Scrub is 100% open-source under the MIT license.

🔗 GitHub Repository: github.com/mahdyarmonfared/secret-scrub

I'd love to hear your thoughts:

• What provider signatures or secret formats would you like added?
• How does your team enforce secret prevention before CI/CD?

If you find this tool helpful for securing your repositories, feel free to drop a ⭐ on GitHub!