I audited 777 AI skills from the biggest public collections. One in three failed.
Skills are having a moment. A skill is a folder of instructions your AI reads and follows, and public collections of them now hold hundreds of thousands of stars on GitHub. People install them the way they install editor
Skills are having a moment. A skill is a folder of instructions your AI reads and follows, and public collections of them now hold hundreds of thousands of stars on GitHub. People install them the way they install editor themes: quickly, and without reading them.
That bothered me. A skill is not a theme. It is text your AI treats as instructions, sometimes including shell commands. So I wrote a checker and ran it over the biggest public collections I could find.
What I checked
SkillDoctor is a single Python file with zero dependencies. For every SKILL.md it can find, it checks four things:
- Validity: frontmatter present, name in kebab case and matching its folder.
- Security patterns: shell pipes from the network, access to credential stores, exfiltration language, prompt override phrases, long encoded blobs, destructive commands.
- Conflicts: duplicate names and near duplicate descriptions across a collection.
- Token cost: what the skill adds to your context, estimated from its size.
Each skill gets a verdict and a score. A collection gets a health score out of 100.
The audit
I fetched every SKILL.md from six collections through the GitHub API: Anthropic's skills (20), superpowers (15), mattpocock/skills (39), addyosmani/agent-skills (26), K-Dense scientific-agent-skills (177), and the first 500 of ECC. Total: 777 skills.
Results: 310 passed, 200 warned, 267 failed.
The curated collections are in great shape. Anthropic scores 97, superpowers 100, mattpocock/skills 99, addyosmani 92, K-Dense 96. Small, maintained sets stay clean.
ECC, the giant open collection, scores 76. Of its first 500 skills, 253 failed and 155 warned. The failures cluster in duplicates: many skills carry near identical descriptions, which confuses any AI trying to pick the right one.
The findings that matter
Across all 777 skills:
- 136 mention API keys, tokens or passwords.
- 97 reference credential stores.
- 9 use destructive commands.
- 8 carried the highest severity pattern: instructions to read or send credential material.
To be fair: a mention is not a crime. Plenty of legitimate skills talk about API keys because that is their job. The checker is a pattern heuristic, and a fail means read this before you trust it, not this is malware. But that is exactly the problem. Today there is no read this step at all. People install instructions into their AI with less scrutiny than a browser extension gets.
Skills also cost quietly. Added together, the 777 files would take about 1.76 million tokens if they all sat in one context. Every skill you install is rent you pay on every conversation.
The tools
SkillDoctor is free and open source. It runs as one file, as a GitHub Action, as standalone binaries, or from PyPI:
pip install skillcheckup
skilldoctor path/to/your/skills
It scores every skill, explains each failure with a fix line, and can quarantine the failures into a folder you can restore from. Nothing is ever deleted.
And because checked skills deserve somewhere good to live, I also maintain SkillPort: curated skill packs that run on Claude, Cursor, Codex and ChatGPT, each skill written to one standard and converted for every platform.
Full audit, method and tables: https://github.com/Dongdarealest/skilldoctor/blob/main/AUDIT.md
The checker: https://github.com/Dongdarealest/skilldoctor
The packs: https://github.com/Dongdarealest/skillport
If you run the checker on your own collection, I would love to see your health score in the comments.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.