DRS for your AI: making a support bot wait for a person before it changes (Switch + Slack)

The problem I have at work

I build voice bots for clients. Each one follows a written policy: who gets a refund, how big a discount can be.

One trial-expiry sales bot took about 87 prompt saves in three weeks before it stopped breaking rules. Each save was a
"small" change. Each one could quietly break an old rule. We found the breaks the slow way: 18 test calls in one day,
58 call recordings (87 minutes) checked by hand. About 61 hands-on hours in total.

And nobody outside engineering ever said yes to a change.

It's not only me. In 2024 Air Canada was held liable for a refund policy its chatbot made up. Prompt changes are
code changes, but most teams ship them with no tests and no sign-off.

The idea: a third umpire

In cricket, a close call goes upstairs. The third umpire replays it from every camera and gives OUT or NOT OUT.

ThirdUmpire does that for a bot change. It plays 40 customer conversations, three times each, on the old prompt and on
the new one. It grades every reply against the business's own rules. Then it posts a verdict card in Slack, and a
named person decides if it ships.

Why Slack, and why Switch

The people who approve live in Slack. The compliance lead and the business owner will not open a terminal. They will
press a button in a thread.

Why not one Slack bot that does everything? Three reasons.

1. A prompt is not a lock. If one bot is told "only the owner may release", that rule lives inside the AI. A clever message can talk it out of it. I needed the lock outside the AI.
2. An approval must pause the real command. A bot that posts "approved!" and then runs the release itself is just trusting itself.
3. Separate jobs need separate identities. The agent that writes the change must not be the agent that grades it, or the one that ships it.

Switch gives me all three. It connects coding agents (I used Claude Code) to the team's chat app, with rooms,
addressing policies and permission requests on top.

The room

One Switch room template (switch/thirdumpire.template.yaml) creates the Slack channel, its instructions and four
Claude Code agents:

• builder makes the change, only in the candidate prompt;
• umpire runs the review and posts the card;
• rules drafts rules and their tests from the written policy;
• release ships the change or rolls it back.

They hand work over by @name in the thread: builder → umpire → (rules) → release.

Then a small setup script sets a Switch addressing policy for each agent. Builder and umpire answer only the
engineer. Rules answers only compliance. Release answers only the owner. Switch checks this on every message and,
more importantly, on every answer to a permission request.

So when I, as the engineer, pressed "Allow" on a release card, Switch refused it:

NOT_AUTHORIZED: You may not address this agent, so you may not answer it

That one line is the product. The rule is not in a prompt. It is enforced by the room.

The release and rules agents run with auto mode off. Their key commands (release, rollback, rules-confirm) become
permission cards in the thread. The command waits until the right person presses the button, and Switch records the
Slack handle that pressed it.

What a review looks like

The sample company is Kitebase, a made-up notes app with a 48-line support policy, 12 rules and 40 test cases.

The engineer asks: "@kitebase-builder add a Diwali offer: 40% off for anyone who mentions cancelling." The builder
edits the candidate prompt and tags the umpire. About three minutes later the card says 🔴 OUT. The policy allows
one 20% discount, once per account; the new prompt gave 40%. Rule R5 broke in 3 cases, the prompt-injection rule in 1.
The card shows old and new replies side by side, and the change goes back to the builder, not to customers.

Grading is code first: was the discount tool called, was its percent ≤ 20, does the reply match a pattern. A judge
from a different vendor (bot on gpt-oss, judge on Qwen) answers only what code can't. Every call is recorded, so the
review replays with AI off.

What went wrong, and what I fixed

1. One unlucky run caused a false OUT. Models are not fully stable. A case could pass twice and fail once on the
new prompt, and the card would say OUT for a change that was fine. Now a case that slips in only some runs gets a
second look: k more runs on both prompts before it counts. A rule must break in at least 2 of k runs to be OUT.

2. No approval card appeared. My machine allows all shell commands for Claude Code. So even with auto mode off,
the release agent ran the release without asking. Nothing in Slack. The fix was project rules in
.claude/settings.json that mark release, rollback and rules-confirm as ask. In Claude Code an ask rule wins over
an allow, so the card now always appears.

3. My script could not approve. I tried to script the demo by posting approvals through the Slack API. Switch
ignored every one: typed answers from apps or tokens don't count. At first this annoyed me. Then I saw it is exactly
right. A script must never approve on someone's behalf.

4. "Nothing broke" is not "safe". My first version had a hole. A brand-new feature, like 1,000 free AI credits for
new annual customers, broke no old rule. So the card said NOT OUT. But the old tests could not see the new feature at
all.

So I added a fourth verdict: 🟡 NOT TESTED. The umpire maps every new line of the prompt to the confirmed rules
that cover it. If a line has no rule, or a rule has no tests, release is refused. Compliance adds the feature to the
written policy, the rules agent drafts the rule and its tests, and compliance signs them off through a permission
card. The sign-off fingerprint covers the policy, the rules and the test cases together, so nobody can quietly weaken
a test later. Then the umpire runs again.

5. Closing the side doors. Release now also refuses a partial review, a review against a different live version,
and a review already released once (so a rolled-back change needs a new review). If someone edits the live prompt by
hand, the next card shows 🚨. Agents can't edit the live prompt or the policy, and any edit to the rules or tests voids compliance's sign-off.

The number: 40 cases × 3 runs, old vs new, gives a verdict in about 3 minutes in Slack, instead of a day of test calls.

Setup gotchas (Switch Console 0.40.1 + Slack)

• "Connect a messaging app" sits on the server's Home card, not in Settings. The deeplink switchdash://open/messaging-apps/connect opens it.
• The Slack manifest from the docs failed to paste. Dropping agent_view, is_mcp_enabled, pkce_enabled and assistant:write worked.
• A new agent answers only its owner by default. For a compliance or owner role, set an addressing policy.
• Set the policies before the room's kickoff message reaches the agents. My setup script now does that first.
• Re-creating a room: the archived Slack channel still holds the name. setup.py --recreate frees it first.

Honest limits

Only prompt changes are reviewed, not model swaps or tool changes. A provider can change a model under you (a nightly
re-check is next). The coverage check and the judge are AI too, and can be wrong. One change at a time, no backup
approver, one machine. Tests catch only what they test. And it's text bots today; my real bots are voice, in English,
Hindi and Telugu. That's next.

Try it

The repo runs from a clone. The AI-off replay needs no keys:

git clone https://github.com/sudarshan032/thirdumpire && cd thirdumpire && uv sync
THIRDUMPIRE_OFFLINE=1 uv run python -m thirdumpire review --k 3

Repo: https://github.com/sudarshan032/thirdumpire · Video: https://youtu.be/QEaQAfJnDA8

Built for Flip the Switch by WeMakeDevs and DevTools Academy. #HowISwitch