Set up toolsmith: an agent that researches tool swaps and can't run rm

toolsmith is the Hermes agent profile that looks at one area of my homelab each week, researches what could replace the tool I use there, tests the best candidate in a throwaway container, and asks me with ✅ / ❌ in Discord. Anything it can't back up with a measured fact gets withheld.

It needs Hermes Agent and Proxmox to do the full job, so this isn't a five-minute toy. But the safety parts, the tests, the installer and the guard, run anywhere with Python. That's where the setup guide starts, and those steps run on a fresh Ubuntu machine every week.

Writing the guide found a bug

The README told you to run the tests and then install.sh. Doing exactly that broke the install. The tests leave a scripts/__pycache__ folder, install -m 0755 scripts/* refuses a directory, and set -e turned that into a silent stop halfway through. No profile, no config, no data files.

I clearly never ran the README in that order on a clean machine. A test that runs the guide did, the first time it ran. It's fixed now: the installer copies files only.

Install and prove it

git clone https://github.com/casareanderson/toolsmith && cd toolsmith
python3 -m unittest discover -s tests       # Ran 13 tests ... OK
HERMES_HOME=$HOME/.hermes ./install.sh

Then poke the guard. It's a pre_tool_call hook that sees every tool call before it happens, and it speaks JSON on stdin:

S=$HOME/.hermes/scripts
echo '{"tool_name":"terminal","tool_input":{"command":"rm -rf /"}}' | python3 $S/toolsmith_guard.py
echo '{"tool_name":"web_search","tool_input":{"query":"caddy vs traefik"}}' | python3 $S/toolsmith_guard.py

The first comes back "decision": "block" with the list of what is allowed. The second comes back {}, allowed. It's wired fail_closed, so if the guard crashes, the call is blocked too.

Then wire it up

Discord channel and your user id (the only ✅ that counts), your Proxmox nodes, which containers it may sandbox in, and the two timers. The bot token goes in an env file, never the config. toolsmith_sandbox.sh refuse-test asks for too much memory on purpose and should be refused, and toolsmith-review.sh --no-post does a full dry run with nothing sent.

What CI can't prove

All of that wiring. It needs Discord, Proxmox and real containers, so the weekly run stops after the guard. The guide marks which is which.

Full guide: github.com/casareanderson/toolsmith/blob/main/docs/SETUP.md

Do you let your agents run shell commands at all? What's your guard?

🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.