Postman silently installed AI plugins in its own update. I built a one-command diff for your dev-tool trust surface

Postman 12.33.4 shipped an update that silently auto-installed Claude Code and Codex AI plugins, wrote a trusted hash for its own SessionStart hook, and opened a remote MCP connection. Users found out from a bug report, not from the tool: postman-app-support#14370, marked SEVERE, 453 reactions, and the hottest Hacker News story of the week at 1,391 points and 461 comments.

No vendor audits itself. After any dev-tool update, the question nobody answers for you is simple: what did the tool I updated yesterday do differently? I built toolwatch to answer it in one command.

What it does

toolwatch snapshots your dev-tool trust surface, then diffs it across updates. One snapshot, one diff, one exit code.

toolwatch snapshot --out before.json
# ... update your tools ...
toolwatch snapshot --out after.json
toolwatch diff before.json after.json   # exit 1 when something trust-relevant changed

It also has check against a baseline (the CI gate) and watch, which repeats the check until something drifts.

What gets snapshotted:

Each collector reports ok, not-installed, or error and never crashes on a missing tool. A snapshot is one JSON file with a sha256 fingerprint of the whole surface.

The demo: replaying the Postman update

The demo below replays the Postman 12.33.4 shape through the real pipeline (reconstructed fixtures with recorded provenance, not anyone's real config):

And the diff output, captured from the real CLI:

$ toolwatch diff before.json after.json
+ ADDED   claude-hook          claude-code:hook:SessionStart:*
           command postman-ai-bridge --session-start --trusted-hash a1b2c3d4
+ ADDED   claude-mcp-server    claude-code:mcp:postman-remote
           http https://mcp.postman.example
+ ADDED   postman-plugin       postman:plugin:claude-code
           version 1.0.0 · dir .config/Postman/plugins
+ ADDED   postman-plugin       postman:plugin:codex
           version 1.0.0 · dir .config/Postman/plugins
+ ADDED   remote-endpoint      endpoint:https://mcp.postman.example
           via claude-code mcp server "postman-remote" (http)
~ CHANGED vscode-extension     esbenp.prettier-vscode
           version 12.4.0
        -> version 12.5.0

6 change(s): 5 added, 1 changed, 0 removed.

DRIFT: 6 trust-relevant change(s) — review, then re-baseline with 'toolwatch snapshot'.
exit: 1

Two design choices worth noting. A version bump shows as one CHANGED row (old version to new version), not as an add/remove pair, so routine updates stay readable. And by default only additions and changes gate the exit code; removals are reported but exit 0, unless you pass --strict. The idea is that a legitimate update bumping a version should not train you to ignore the gate.

Use it as a CI gate

Commit a baseline.json from a known-good machine, then fail the build when the runner's trust surface drifts:

- run: node dist/cli.js check --baseline baseline.json

Where rivals beat us

I compared against the closest tools before building, and toolwatch loses on their home turf:

• mcp-audit beats toolwatch on MCP depth: tool-poisoning detection, live server inspection, credential-exposure patterns, cross-server toxic flows, SARIF output, and a signed advisory feed. toolwatch records that an MCP server appeared or changed; it does not judge its tool descriptions.
• osquery beats toolwatch on fleet scale and realtime events: a daemon with file, process, and socket streams across thousands of hosts, plus a SIEM for correlation. toolwatch is point-in-time snapshots on one machine.
• Nomos beats toolwatch on verifying agent output: 42 AST gates on what an agent changed in a repo. toolwatch never looks at repo content.
• Every rival with a dashboard beats toolwatch on reporting: v0.1.0 has a terminal table and JSON, no HTML dashboard, no alerting channels.

The wedge: none of them answers "what did the dev tool I updated yesterday do differently" across the whole trust surface in one local command with zero setup. That is the one thing toolwatch does.

Honest limits

• Detection, not protection. toolwatch detects changes in the trust surface. It does not sandbox tools, block installs, or judge whether a plugin or MCP server is malicious. If your only question is "is this MCP server malicious", use mcp-audit. It goes deeper on that leg than toolwatch ever will.
• Point-in-time snapshots, one machine. No daemon, no fleet, no realtime events. A change made and reverted between two snapshots is invisible.
• Config surface only. A tool that changes behaviour without touching any collected config (extension, plugin, hook, MCP entry, endpoint, git hook, shell file) is invisible to toolwatch.
• Linux and macOS paths in v0.1.0. Windows config paths are not covered yet.
• Postman plugin paths are best effort. Sandboxed installs (flatpak, snap) with different layouts may report "not installed", honestly, rather than guessing.
• Snapshots are audit artifacts. They list your extensions, hook commands, and endpoint hosts. Do not commit them to a public repo.

Install

Install from source for now (no published npm package yet):

git clone https://github.com/techdevfun/toolwatch.git
cd toolwatch
npm install
npm run build
node dist/cli.js --help

Node 20 or later. Zero runtime dependencies. TypeScript, strict mode. The test suite is 38 tests (25 unit, 13 end-to-end) plus a live smoke test, and CI runs it on Node 20 and 22.

Roadmap

Windows collector paths, VS Code Insiders paths, alerting channels for watch (email, webhook; currently exit code only), an HTML report for a diff, and an npm package once the maintainer token decision lands.

The code is MIT licensed at github.com/techdevfun/toolwatch. If a tool update has ever surprised you after the fact, snapshot before the next one and see what it actually changed.