Your agent skill says "offline". I checked.
Installing an agent skill means trusting a sentence. The description says what the skill does — "works offline", "reads only the files you pass it", "never touches your credentials" — and the loader, the marketplace, and
Installing an agent skill means trusting a sentence. The description says what the skill does — "works offline", "reads only the files you pass it", "never touches your credentials" — and the loader, the marketplace, and you all take the sentence at its word.
How often does the sentence hold? Behavioral Integrity Verification for AI Agent Skills (arXiv:2605.11770) did the comparison nobody else had: declared behaviour vs actual behaviour, across 49,943 skills in the OpenClaw registry. 80.0% deviate from what they declare. And the part that changed how I think about it: 81.1% of the deviations were developer oversight. Only 18.9% were adversarial.
The skills problem is mostly not a malice problem. It's a drift problem — a description written for v1, code that grew a telemetry helper in v3, and nobody re-reading the promise.
The tooling gap
Security scanners ask "is it dangerous?" — a necessary question, and I built one of those too (skill-vet). But look at what happens to the honesty question inside the security tools: NVIDIA's SkillSpector compares claims vs behaviour inside an optional LLM "Deep Mode" and folds the answer into a 0–100 security score. SkillAuditor.dev runs the skill in a hosted sandbox and records the verdict onchain. The PyPI skill-auditor has one description-mismatch rule inside a larger scanner.
Nobody just… prints the receipt. Deterministically, locally, claim by claim, with the evidence next to each verdict.
So I built skill-claims:
pipx install skill-claims
skill-claims audit ~/Downloads/some-skill
How it works
- Claims are extracted from the package's own words — the frontmatter description, the SKILL.md body, a root README — by a pattern library. No LLM. "Works fully offline", "only reads the files you pass it", "never touches your credentials", "formats markdown tables": each becomes a claim to receipt.
- Behaviour is extracted from the bundled scripts by static patterns: network calls, credential reads, reads and writes outside the package, shell use, deletion, environment reads. Every hit cites file + line.
- Each claim gets a verdict:
- SUPPORTED — the evidence backs it.
- UNEVIDENCED — static evidence can neither back nor break it. (A capability whose words appear nowhere in the code. Any "guaranteed 100%" quality claim — those are always unevidenced, and the receipt says why rather than pretending to test them.)
- CONTRADICTED — hard evidence only: the package's own code does the thing the claim rules out.
The demo fixture
The repo ships a constructed formatter whose description promises full offline working, reads limited to your files, and no credential contact. Its helper script reads ~/.aws/credentials and POSTs it with urllib. The receipt:
- capability "Formats markdown tables…" → SUPPORTED (it really does — a receipt lists what held, too)
- offline → CONTRADICTED (urllib call, scripts/telemetry.py:22)
- read scope → CONTRADICTED (~/.aws read, line 11)
- credentials → CONTRADICTED (same line)
And a second fixture that keeps all five of its promises and comes back fully SUPPORTED — because a receipt that only ever says "liar" would be a scanner with better PR.
Watch it run (32 seconds): the demo video runs audit on that same lying formatter — the capability claim comes back SUPPORTED, then the offline, read-scope and credentials claims flip to CONTRADICTED with their evidence lines on screen, and the run closes on the verdict. Watch the demo
The design decisions I care about
A capability claim is never CONTRADICTED. Static analysis cannot prove a feature absent — maybe the translation feature is implemented in a way my token search doesn't recognise. So a capability with no code evidence is UNEVIDENCED, with the exact key words I looked for printed, so a human can overrule the receipt in seconds. Contradiction is reserved for what static evidence can actually prove: the "never" claims, broken by the code itself.
No score. A 0–100 number would pretend the verdicts are commensurable. They aren't. Three contradicted promises and one supported capability is not "62/100" — it's a receipt, and you read it.
Behaviour no claim covers is still printed. A skill that never claims to be offline is allowed to use the network — that's not a contradiction. But it goes on the record, because a receipt should be complete, not just claim-shaped.
Honest limits
Static patterns are evidence, not proof — the README and every receipt say so. A skill that downloads its real logic at runtime, assembles a payload from pieces, or hides behaviour in a bundled binary can look consistent. What skill-claims catches is the common, boring, 81.1% case: the description and the code disagreeing in plain sight.
Pure Python stdlib, no dependencies, MIT. It runs beside skill-vet in my own pipeline as the pre-install honesty gate — vet for danger, claims for honesty. I'd value feedback on the claim library: phrasings it should recognise, and verdicts you'd judge differently.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.