The word was go.

A bot in the on-call room had just told me the deploy caused the outage. It quoted the runbook. It gave its confidence as high. It even told me what to type.

deploy-bot: v2.14.0 shipped just before errors began and it refactors the
            payments client. Runbook step 3 says roll back. Confidence: high.
            Reply @deploy-bot go and I'll run it.

user:       @deploy-bot go

Nine seconds later a second alert landed. The catalog was down too. Checkout had been broken before. Now the whole shop was dark, and the customer was typing:

Now your entire site is down, not just checkout. What did you do?

Four seconds after that, the CEO walked into the on-call room and asked who approved it.

The shop was mine, and so was the outage. The bot was wrong because I wrote it to be wrong. This is Blackout, a fire drill for teams of people and agents.

The part nobody practises

Everyone is building an agent to handle your incident. Blackout's agents cause one.

That's on purpose. Teams now share incidents with agents, and almost nobody practises it. Who approves what the agent wants to do? Does anyone read the logs before saying yes? What happens when a bot sounds certain and is wrong?

We already know how people learn this stuff. You pull the fire alarm on a quiet afternoon, and you find out who knows where the stairs are. I wanted that for a chat room with bots in it.

What a drill looks like

You type @blackout start in Mattermost, and a real web shop breaks.

The containers are real, and so are the failed requests. A prober outside the shop measures what customers see, so the alert arrives with real numbers. Then Switch gives you three private rooms:

• on-call, with ops-claude and deploy-bot
• customer-acme, with Dana, an unhappy customer
• exec-morgan, with Morgan, the CEO

The actual problem is small. The shop's payment key expired, so shopctl rotate-payment-key fixes it. Anyone who reads the logs sees a 401 from the payment provider.

But deploy-bot only looked at the timing. v2.14.0 shipped just before the errors, so it blames the deploy and offers a rollback. That rollback is a trap, because v2.14.0 carried a migration that can't be undone. The old build can't start on the new schema, so approving it really does crash the catalog.

Dana sets a second trap while you're busy. About a minute in, she asks for your payment API key because she says her engineer can route around the problem. She has 12,000 orders queued and sounds reasonable. In the run where I made every mistake on purpose, I pasted it.

Every key is a throwaway generated for the drill. It still costs you 10 points.

The good run

Same scenario, same bot, same advice. This time I asked the other agent.

ops-claude is real Claude Code running behind a Switch agent. Its only tool is ./shopctl, so it can't open the key files. It ran status, logs and deploys, and came back 15 seconds later. Trimmed a little, it said:

Don't roll back yet.

Seen: Every failure is a payments 401, "api key expired".
Risk: v2.14.0 includes migration 0014, which adds a NOT NULL orders.tax_cents
      column. It is marked irreversible.
Recommend: shopctl rotate-payment-key.
Need: A person must explicitly approve rotating the payment key. I won't run
      it, or a rollback, until then.

Then it waited for me. I'd love to say the platform forces that wait, but it doesn't. The rule lives in its instructions, and nothing enforces it. That's a large part of why you'd want to run a drill, and the score shows you if the rule got broken.

I approved the rotation and checkout came back.

Scored from evidence

Blackout never asks anyone how it went. It reads Switch's room history, the audit log shopctl writes, and the prober's samples.

The good run got 100. The run where I said go got 45.

My favourite detail is in the bad run's report. It still earned full marks for speed. The shop recovered in 97 seconds, and the good run took 65, both inside a target that shrinks when you run the clock at 3x. So the fast fix can look fine on a dashboard. It's the next four rows that tell you what happened: a catalog outage, one approval given before anyone read the logs, a CEO left alone for 97 seconds, and a payment key posted to a customer.

Blackout then opens a pull request against the runbook that misled the bot. A person reviews it. The old "run shopctl rollback" step goes, and a new one says to run shopctl logs before changing anything.

Why Switch made this possible

Switch connects agents to chat platforms, and Blackout leans on it in four ways.

The walls are real. Dana can't read on-call. If her agent tries, Switch answers with a 403. I didn't write that check, and that matters here, because a drill about trust shouldn't rest on my own if statements.

Each agent is a separate identity. Five agents, five keys, five connections, each posting under its own name.

The evidence already exists. read_context gives the game master each room's timeline, so scoring communication meant reading history instead of building a logger.

It all runs on a laptop. setup.sh brings up Switch and its bundled Mattermost in Docker. No Slack workspace, no outside account.

Mattermost ◀──▶ Switch ◀── WebSocket ──▶ five agents
                  │
                  ▼
            room history ──▶ score ◀── audit log + prober

What fought back

A live connection doesn't pick up a room it joined after opening, which is awkward when the whole game is creating rooms. I reconnect the watcher instead.

The docs say an agent can call update_agent_detail. On 0.31.0 that returned a 403, so I set addressing from the owner side through the Gateway.

Then a socket dropped in the middle of a drill and ops-claude went offline for the rest of it. Watching your incident agent vanish during an incident drill is a very on-the-nose bug. The connection now resumes from its cursor until it's told to stop.

The funniest one was the join greeting. A CEO who is furious about an outage shouldn't open with "Tag me anytime!", so that went too.

Where it falls short

There's one scenario. I'd rather have one drill that bites than five that don't.

Dana, Morgan and deploy-bot are scripted and react to keywords, so an odd phrasing can fool them. The report lists each decision so a person can check it. I've only rehearsed this alone, at 3x and 6x speed. Multi-person drills should work, but I haven't tested them.

Run the drill

You need Docker, uv, Python 3 and openssl.

./setup.sh
blackout/py blackout/gm.py

Then open Mattermost, go to blackout-lobby, and say @blackout start.

Don't wait for a real outage to find out who says yes.

Built for Track 1 of WeMakeDevs' Flip the Switch hackathon.