Best data anonymization tools for dataset licensing (2026)
The best open-source stack for preparing a dataset to license covers six jobs, and no single tool does all of them. For personal information in text, use Presidio with spaCy (or scrubadub for something smaller); for tabl
The best open-source stack for preparing a dataset to license covers six jobs, and no single tool does all of them. For personal information in text, use Presidio with spaCy (or scrubadub for something smaller); for tables, ARX; for faces in video, deface; for secrets in code, gitleaks or TruffleHog, then git-filter-repo to remove them from history; for third-party code, ScanCode; for near-duplicates, datasketch or text-dedup; and for documentation, a Hugging Face dataset card or a datasheet. Every one of them is free, and every detection tool misses things, so a human still reviews a sample.
What does preparing a dataset for licensing involve?
It means removing what the buyer must not receive and describing what they will. In practice that is six jobs: strip personal information, blur faces, remove credentials, separate code you don't own, collapse duplicates, and write down what the dataset is and how it was made.
Licensing raises the bar compared with internal use. Records leave your control, and the buyer's lawyers will ask how you de-identified them. Our guide to de-identifying company data before licensing covers the method. This post covers the tools.
Which tools are best for each job?
We checked each project's license and status on its official repository on 7 November 2026. Licenses and maintenance change, so re-check before you adopt one.
| Tool | Job | License | What it does |
|---|---|---|---|
| Presidio | PII in text, images, tables | MIT | Detects names, card numbers, phone numbers and more with NER, regex, rules and checksums, then masks or replaces them |
| spaCy | Named entity recognition | MIT | The NLP engine Presidio loads by default (en_core_web_lg) |
| scrubadub | PII in free text | Apache-2.0 | Small Python library: names, emails, phones, addresses (US, GB, CA), card numbers, US SSNs |
| ARX | Tables | Apache-2.0 | k-anonymity, l-diversity, t-closeness, differential privacy and re-identification risk analysis |
| deface | Faces in video and photos | MIT | Detects faces frame by frame and applies blur, solid boxes or mosaic |
| gitleaks | Secrets in git history | MIT | Scans every patch through git log -p, plus directories and stdin |
| TruffleHog | Secrets in git history | AGPL-3.0 | 700+ detectors, with optional live verification of each credential |
| detect-secrets | Stopping new secrets | Apache-2.0 | Baseline file plus pre-commit hook, so only new secrets fail a commit |
| git-filter-repo | Removing data from history | MIT | Rewrites history to delete files or replace strings in every commit |
| ScanCode Toolkit | Third-party code | Apache-2.0 | Detects licenses, copyrights and package manifests in source and binaries |
| datasketch | Near-duplicate text | MIT | MinHash and LSH building blocks for your own pipeline |
| text-dedup | Near-duplicate text | Apache-2.0 | Ready scripts for MinHash, SimHash, suffix array and Bloom filter dedup |
| dedupe | Duplicate records | MIT | Fuzzy matching and entity resolution for structured rows |
| Hugging Face dataset cards | Documentation | Format | A README with YAML metadata: license, languages, structure, collection |
| Datasheets for Datasets | Documentation | Paper | A question set on motivation, composition, collection and intended uses |
| Croissant | Machine-readable metadata | Apache-2.0 | MLCommons format built on schema.org that describes files, structure and semantics |
Which PII tool should you start with?
Start with Presidio unless your data is tiny. It handles free text, images (including DICOM medical images) and structured data, and you can add your own recognizers for internal IDs such as customer or ticket numbers, which no generic model knows about.
One change matters as of November 2026: Presidio is no longer a Microsoft project. It moved to a community-governed GitHub organization, Data Privacy Stack, stays MIT-licensed, and new Docker images now ship from GitHub Container Registry instead of Microsoft's registry. Old links redirect, but update mcr.microsoft.com/presidio-* image references.
A minimal run looks like this:
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
text = "Ticket 4411: call Dana Ruiz on 212-555-0143 about the refund."
findings = AnalyzerEngine().analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=findings).text)
The default operator replaces each finding with its entity type, so a name becomes a PERSON placeholder. The README itself warns that automated detection cannot guarantee it finds everything. Treat its output as a first pass and review a random sample by hand.
scrubadub is simpler and fine for a quick script, but its last release (v2.0.1) dates from September 2023. For tables of customers or transactions, ARX is the better fit, because the risk there is rare combinations of ordinary fields (postcode, age, job title), not names in sentences.
How do you clean secrets out of a codebase?
Scan the full history, rotate what you find, then rewrite. A key deleted in a later commit is still in the old one, and a code buyer receives the whole history.
-
gitleaks is the fast default.
gitleaks git -v --log-opts="--all" .scans every branch. -
TruffleHog adds verification: it tests each candidate against the provider's API and reports whether it is live. That is useful for triage, but it means the tool contacts those services with your credentials.
--no-verificationturns it off. The AGPL-3.0 license matters if you plan to embed it in a product rather than run it. -
git-filter-repo does the rewrite. Since version 2.47 it has a
--sensitive-data-removalflag that also reports which commits changed and what to clean up elsewhere.
How do you handle faces, duplicates and third-party code?
Use deface for faces, a MinHash tool for duplicates and ScanCode for code you don't own. Each solves one narrow problem well.
deface blurs faces only. Number plates, screens and paperwork in frame need another pass. Its output leaves out the audio track unless you pass --keep-audio, which is the right default for workplace footage.
For duplicates, DeepMind's Genie paper is a useful reminder of why curation matters: a curated set of about 30,000 hours, roughly a tenth of the raw video, trained a better model than the full set (Genie, arXiv). Near-duplicate support tickets and email threads inflate your volume count without adding new examples, so dedupe before you describe the dataset to anyone.
For codebases, ScanCode lists vendored libraries and their licenses so you can exclude or document them. Our guide to selling source code for AI training explains why ownership gaps end code deals.
What should the documentation say?
Write a dataset card even if the buyer never asks. Cover what the records are, the years they span, how they were collected, what you removed and with which tools, and what the buyer may not do with them. Datasheets for Datasets gives the questions; a Hugging Face card or Croissant file gives the format.
The card also forces an honest inventory. If you can't say where a field came from, a buyer can't license it. Records you never kept can't be documented at all, which is why a retention policy that keeps data sellable comes before any of these tools.
Common questions
Do these tools make data anonymous?
Not on their own. They remove identifiers, but under the GDPR pseudonymised data is still personal data, and rare combinations of facts can identify people after names are gone. Anonymity is a result you test, by sampling and by asking whether someone with outside knowledge could re-identify a person, not a setting you switch on.
Is gitleaks or TruffleHog better?
You can run both. gitleaks is MIT-licensed, fast and easy to put in CI. TruffleHog's verification tells you which findings are live credentials, which helps you rotate the right keys first. The two use different detector sets, so a second scan on a long history is cheap insurance.
Can I use an LLM to redact PII instead?
You can, but sending raw records to a hosted model is itself a disclosure of the data you are trying to protect. Local tools such as Presidio keep the records on your machines. If you add a model, run it locally or under a data processing agreement, and still review a sample.
Who does the clean-up, the seller or the buyer?
Usually the seller, or a processor acting on the seller's instructions, before records leave its control. The licensing agreement should say who is responsible. A buyer can describe the format and quality it needs, but raw records with personal information should not be sent out to be cleaned.
If you want to know whether your records are worth this preparation, our free estimate of what your company data is worth takes six questions and no files.
Origin Data Partners is an independent data partner that helps companies get their data bought. A buyer may pay Origin when a deal closes.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.