Dev.to AI 🤖 Ai 👁 0 📖 6 min read

Your agent can rewrite its own instructions. I built a pre-run lock that proves it didn't.

Your agent reads its skills from plain markdown files. It can also edit them. Nothing in the format stops it from rewriting its own instructions — and nothing proves the instruction it is running is the instruction you a

Your agent can rewrite its own instructions. I built a pre-run lock that proves it didn't.

Your agent reads its skills from plain markdown files. It can also edit them. Nothing in the format stops it from rewriting its own instructions — and nothing proves the instruction it is running is the instruction you approved. I built skill-lock to prove exactly that, before the agent runs.

Two measured facts set the problem. First, the instruction drifts: "Don't Write Markdown Files for Your Coding Agent. Write Scripts." (dev.to, Daniel Nwaneri, 5 Nov 2026) makes the point that an agent can read a CONTRIBUTING.md, AGENTS.md, or rules file, quote from it, and still not follow it — and the article's own comment thread warns that agents modify their own scripts. Second, what the skill tells the agent to install may not exist at all: in the "Phantom Packages" study (dev.to, 1 Nov 2026), three local open-weight models wrote 1,199 Python scripts and 212 of 1,410 install suggestions (15.0%) named packages nobody has ever published — and on a full rerun from scratch, 161 of the 162 ghost names came back. A reproducible hallucination is a target, not noise. The underlying study (Spracklen et al., USENIX Security 2025) found 19.7% of recommended packages hallucinated across 576,000 samples and 16 LLMs. The attack has a name — slopsquatting — and Snyk's ToxicSkills study (5 Feb 2026) found 76 confirmed malicious payloads in 3,984 skills scanned from ClawHub, 8 of them still public when the report was published.

What it does

skill-lock pins every skill file by SHA-256 in a human-approved lockfile, diffs any drift before the agent runs, and checks that every package and URL the skill references actually exists. One local, zero-dependency CLI:

skill-lock init --by operator   # pin every skill file by SHA-256
skill-lock verify               # exit 1 on any drift
skill-lock diff                 # show exactly what moved, line by line
skill-lock approve --by operator  # human-only re-baseline, no --force
skill-lock check-refs           # verify every referenced package/URL exists

The lockfile holds hashes only, never file contents. verify never prints file contents; diff prints only the lines that changed, and marks non-printing characters (invisible Unicode, zero-width, bidi controls) as ⟦U+XXXX⟧ so a smuggled character cannot hide in a diff.

The demo below runs the real CLI against real fixture skills — init, verify clean, a one-line edit to a skill (the way an agent editing its own instructions would), verify catching the drift, diff showing the exact line, approve, then check-refs finding the package that does not exist:

skill-lock demo

And the check-refs output, captured from the real CLI:

$ skill-lock check-refs
reference                                        registry  status    source
prettier                                         npm       EXISTS    refs-demo/SKILL.md:15
requests                                         pypi      EXISTS    refs-demo/SKILL.md:19
https://example.com                              url       EXISTS    refs-demo/SKILL.md:23
skill-lock-demo-ghost-9x7q                       npm       MISSING   refs-demo/SKILL.md:30
https://skill-lock-demo-ghost-9x7q.invalid/docs  url       ERROR     refs-demo/SKILL.md:34

3 exist, 1 missing, 1 unreachable — 5 reference(s) checked.
GHOST REFERENCE FOUND: a skill tells the agent to install or call something that does not exist. Review before approving.
exit: 1

The ghost name skill-lock-demo-ghost-9x7q is a clearly-labelled fixture, verified non-existent on npm and PyPI at demo time, and never registered or published anywhere.

Use it as a gate, not just a hook

A local hook can be skipped by the agent — the dev.to article says so itself. The hook is convenience; the required CI job and merge gate are the real enforcement. The repo ships a ready-made workflow (examples/pr-review/github-actions.yml): checkout, install, skill-lock verify, skill-lock check-refs --offline. Wire it as a required check on the branch that holds your skills, and the agent has no rights to change it. Exit codes are 0 (clean), 1 (drift or ghost found), 2 (usage or config error).

Where rivals beat us

I compared against the closest tools before building, and skill-lock loses on their home turf:

  • ContextLock (contextlock.net) — the strongest rival. Real cryptography: Ed25519 + Sigstore keyless signing, DSSE envelopes, provenance-attested, 375 tests. skill-lock v0.1.0 has hash-pinning only and no signatures. If you need to sign published skills, use ContextLock. Its gap, and our wedge: ContextLock is publisher-signature-centric. A locally self-modified skill with no publisher has no signature to check — its own spec falls back to "seal it yourself: pin its bytes locally," a manual step, not a managed approve/diff flow. It does not verify that the packages, APIs, or URLs the skill references actually exist.
  • skillfold — lockfile with a sha256 content hash per remote skill; skillfold install --frozen fails the moment installed content diverges. Install-time only — not pre-run, not a CI gate. No existence verification of referenced packages or APIs, no human approve/diff UX. Its own blog states it plainly: "Skillfold verifies integrity, not intent."
  • Phantom Packages — closest on the existence half: Claude Code hook + offline detector, blocked 212/212 ghosts in replay, catches ghosts offline with AUROC 0.975. But PyPI only (no npm, no URLs/APIs), Claude Code hook only, no skill pinning at all, and it needs a trained model where a plain registry lookup suffices for the common case. If you only need to guard pip install, use Phantom Packages.
  • Tech Leads Club agent-skills — curated, hardened registry with SHA-256 hashing and Snyk scans at publish. Registry-only: it covers their skills, not yours.
  • baaderagency skills library — content pinning to an exact commit SHA with hash verification at download time. Download-time only, their library only.

The wedge: one local, zero-dependency CLI that does both halves together — a human-approved lockfile plus a pre-run diff of the skill files themselves, with an approve flow for your own skills, and existence verification of everything the skill tells the agent to install or call — runnable as a pre-run hook and as a required CI merge gate.

Honest limits

  • A hash is not safety. Pinning proves the file is the one you approved; it says nothing about whether the approved content is safe. Review the diff, every time.
  • The lockfile itself can be edited. Anyone with write access to the repo can re-baseline. Commit the lockfile and enforce it in CI; the CI gate is the enforcement, the lockfile is the record.
  • A local hook can be skipped. Run verify in CI as a required check, not only as a local hook.
  • check-refs MISSING is a finding, not a verdict of malice. Registry lag, renamed packages, and private names all produce MISSING. Review the finding; --allowlist covers private names.
  • Offline snapshots go stale. --offline checks against the last snapshot written by an online run. Re-run online regularly.
  • No signatures in v0.1.0. Ed25519 signing of the lockfile is on the roadmap, as is invisible-Unicode linting and private-registry support.

Install

Install from source for now (no published npm package yet):

git clone https://github.com/techdevfun/skill-lock.git
cd skill-lock
npm install
npm run build
npm link        # puts `skill-lock` on your PATH

Node 20 or later. Zero runtime dependencies. TypeScript, strict mode. The test suite is 26 tests (17 unit, 9 end-to-end) plus 2 live tests against the real npm and PyPI registries, and CI runs it on Node 20 and 22. The repo dogfoods itself: its own root skill-lock.json pins its fixture skills and verifies CLEAN.

Sources

Phantom Packages study (dev.to, 1 Nov 2026; repo T-RexBytes/phantom-packages): 212/1,410 ghost install suggestions (15.0%), 161/162 reproducible. Spracklen et al., USENIX Security 2025, "We Have a Package for You": 19.7% hallucinated across 576,000 samples, 16 LLMs, 205,474 fake names, 43% reproduced on every reprompt. 2026 replication: 4.62–6.10%. Snyk ToxicSkills (5 Feb 2026): 76 confirmed malicious payloads in 3,984 ClawHub skills, 8 still public. OWASP Agentic Skills Top 10 rates AST01 Malicious Skills as critical. "Don't Write Markdown Files for Your Coding Agent. Write Scripts." (dev.to, Daniel Nwaneri, 5 Nov 2026).

The code is MIT licensed at github.com/techdevfun/skill-lock. If your agent runs on skills you did not write this morning, pin them before it runs — and check what they tell it to install.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.