How to Run an AI Agent Incident Response with ZizkaDB: A Runbook for the First 72 Hours
An AI agent sends the wrong message to a customer. Or it calls a tool it should not have. Or a component it depends on turns out to be compromised. Someone notices, a channel fills with messages, and a clock you cannot s
An AI agent sends the wrong message to a customer. Or it calls a tool it should not have. Or a component it depends on turns out to be compromised. Someone notices, a channel fills with messages, and a clock you cannot see starts to run.
For many teams, the first hours of an agent incident are improvised. People ask the same questions in different threads: what did the agent read, what did it do, who else is affected, what do we tell the customer, do we have to tell a regulator. Without a record that can answer those questions, the answers come slowly and with doubt.
This guide is a runbook for the first 72 hours of an AI agent incident, built around a record that lets you answer fast. It covers what to prepare before an incident, then what to do in hours 0 to 1, 1 to 8, 8 to 24 and 24 to 72, then the longer tail. The audit layer in the examples is ZizkaDB, an open source causal audit database for AI agents, but the runbook works with any tool that gives you causally linked, tamper-evident records.
A note on scope. This is operational guidance, not legal advice. Whether any reporting duty applies to you, and how deadlines are counted, depends on your product, your role and your jurisdiction, so involve counsel early. ZizkaDB supports record keeping and investigation and does not by itself make any system compliant with any law. Code parameter names are illustrative and should be checked against the SDK reference.
Why the clocks matter
Several regimes now put time limits on incident reporting. You may be subject to none, one or several, so the point here is awareness, not a legal checklist.
EU Cyber Resilience Act. Reporting obligations under Article 14 began to apply on 11 September 2026 to manufacturers of in-scope products with digital elements. According to the European Commission and ENISA, an early warning is due within 24 hours of becoming aware of an actively exploited vulnerability or a severe incident, a fuller notification within 72 hours, and a final report no later than 14 days after a corrective measure is available for an exploited vulnerability, or within one month for a severe incident. Law firm commentary stresses that the clock runs from awareness, not from the moment the issue is fully analysed. Whether your agent product is in scope is a question for counsel.
EU AI Act, Article 73. For providers of high-risk AI systems, serious incidents must be reported to market surveillance authorities immediately after a causal link, or the reasonable likelihood of one, is established, and in any event not later than 15 days after awareness. Summaries commonly state a 10 day limit where a death is involved and 2 days for widespread infringements or critical infrastructure disruption. The Digital Omnibus on AI moved the application date for stand-alone high-risk obligations to 2 December 2027, and summaries note the Commission's serious incident guidance was still a draft when they were written. Commentators warn that 15 days is a short runway if the first conversation about whether this is our incident does not happen until a week in, and that the way days are counted is a point on which summaries differ.
GDPR. Where an incident involves a personal data breach, the notification window to the supervisory authority is generally 72 hours from awareness, subject to the conditions in the regulation.
The practical lesson is the same across all three. The speed of your response depends on how quickly you can establish what happened, who is affected and how bad it is. That is a question about your record.
Phase 0: prepare before anything goes wrong
A runbook is only as good as the record behind it. Before an incident, make sure of the following.
The record has what you will need. For each agent action, the record should contain the input received, the context retrieved, the model and prompt versions, the tool called with its version, the decision made, the human review if any, the customer or tenant, and the output delivered. Each event links to the event that caused it.
External inputs are tagged. Anything from a source you do not control, such as a web page, a partner feed, another agent or a user upload, carries a trust tag. It makes later questions easy: which decisions depended on untrusted input.
Tools and components are versioned in the record. If an MCP server, plugin or package is later found to be compromised, you need to know which agent actions called which version.
The chain head is stored outside the database. ZizkaDB computes a SHA-256 checksum for each event and chains events per agent, so edits to history are detectable. Store the chain head in a system administrators cannot edit, and verify on a schedule. Without that, an investigator, a customer or a regulator has less reason to trust the record.
Read-only evidence access is ready. Have a process to grant an auditor or customer a read-only, expiring key.
Roles and a severity scale exist. Name an incident commander, a scribe, a communications lead and a legal contact. Agree what counts as low, medium and high severity, and who can declare an incident.
You have timed a drill. Pick a past case and measure how long it takes someone who did not work on it to answer four questions from the record alone: what did the agent read, what did it decide, who reviewed it, and which other cases are affected.
Phase 1: hour 0 to 1, declare, contain and freeze evidence
The first hour has three jobs. Do them in this order.
Declare the incident and start the log
Create an incident event and keep a running timeline as events linked to it. A recorded timeline matters later, because the time you became aware may be a regulatory question.
python
inc = await db.log(agent="incident-response", event="incident_declared",
data={"id": "INC-2026-114", "reported_by": "support-lead",
"first_signal": "customer complaint about refund email",
"aware_at": "2026-11-19T09:42:00Z", "severity": "high"})
Record who noticed what and when. The moment of awareness may be disputed, so write down how you determined it.
Contain
Stop the damage first. Typical containment actions are pausing the agent, disabling a tool, revoking a credential, switching to a safe fallback or moving the workflow to human handling. Log each action as an event linked to the incident, with who did it and why.
python
await db.log(agent="incident-response", event="containment_action",
data={"action": "disable_tool", "tool": "issue_refund", "by": "ops-lead"},
parent_id=inc.id)
Containment decisions made under pressure are the ones people later question. The log lets you show the reasoning.
Freeze the evidence
Before anyone investigates, preserve the record as it stands. Export the relevant events, verify the chain, and note the chain head.
python
check = await db.verify_chain(agent="support-agent")
export = await db.export_events(agent="support-agent",
since="2026-11-12", until="2026-11-19")
await db.log(agent="incident-response", event="evidence_frozen",
data={"chain_ok": check.ok, "chain_head": check.head,
"events_exported": len(export), "by": "scribe"},
parent_id=inc.id)
Store the export and the chain head somewhere separate. Do not alter, delete or rewrite anything in the affected period. Avoiding changes to evidence is part of any later investigation, and it protects you from the suspicion that you edited history.
Phase 2: hours 1 to 8, reconstruct what happened
This is where causal lineage pays for itself. You are answering four questions, in order.
Question 1: what did the agent do, and why?
Start from the bad outcome and walk backwards. If the problem is a wrong email, find the event for the email and run lineage on it.
python
(await db.why(bad_email.id)).print()
Read the chain from the output back through the decision, the retrieved context and the original request. Note the model and prompt versions on the decision, and any human review in between. Often this alone reveals the cause: an outdated document retrieved, a misread input, a tool that returned something unexpected.
Question 2: where did the bad input come from?
If the chain leads to an external input, look at its trust tag, source and author. If it leads to a tool, look at the tool and version. If it leads to a memory write, find the event that wrote it and the input behind that. Keep going until you reach the first event that does not depend on another.
Question 3: who else is affected?
This is the question that decides your severity, your customer communications and your reporting duties. Use the impact route from the root cause event to list every downstream event that depended on it.
python
affected = await db.impact(root_cause.id)
by_customer = group_by(affected, key="customer")
The route is also available over HTTP at GET /v1/events/{event_id}/impact. Turn the result into a table: affected cases, customers or tenants, what each received, whether a human reviewed it, and whether personal data was involved. This list is the most valuable artifact of the first eight hours.
Question 4: is it still happening?
Check recent events for the same pattern. If a poisoned document is still in retrieval, or a compromised tool version is still deployed, containment is incomplete. If it is a memory problem, write a corrected memory that supersedes the bad one, so the agent stops retrieving it while the history stays intact for the investigation.
Replay the session when the chain is not enough
For complicated cases, replay the full session in order: every read, decision and tool call with timestamps. Look for things the lineage alone does not show, such as a retry loop, an unusual sequence or a long gap.
Throughout this phase, the scribe records findings as events linked to the incident. Include dead ends, because they matter when someone later asks why you ruled something out.
Phase 3: hours 8 to 24, assess, classify and decide on reporting
By now you should know what happened and roughly how many were affected. The next job is to decide what it means.
Establish the causal link. Several regimes turn on whether the incident was caused by, or reasonably likely to be caused by, the system. The lineage is your evidence. Write down the causal chain in plain language.
Assess severity and impact. Use the affected-cases table. Consider harm to people, data involved, financial impact, effect on critical services, and whether the problem is ongoing.
Check data protection. If personal data was exposed, altered or lost, bring in your data protection lead. The affected-cases table shows which individuals and what data.
Classify and decide with counsel. Whether a reporting duty applies, to whom, and by when, is a legal judgment. Give counsel the incident timeline, the causal chain and the impact list. Record the classification decision and its rationale as an event, including any dissenting views, so that the reasoning is documented.
python
await db.log(agent="incident-response", event="classification",
data={"severity": "high", "causal_link": "established",
"personal_data": True, "reportable": "under review with counsel",
"affected_cases": 37, "decided_by": "incident-commander"},
parent_id=inc.id)
Prepare an early notification if one is needed. A first report can be short. It generally says what you know, what you do not yet know, and what you have done. Do not wait for a complete analysis if a deadline requires an early warning, and say clearly what remains under investigation. Initial incomplete reports followed by fuller ones are a feature of several regimes, so check what applies to you.
Phase 4: hours 24 to 72, notify, communicate and remediate
Notify
If counsel confirms that a report is due, build it from the record. The record can supply the product and version details, the timeline, the causal chain, the affected population, the containment actions and the data involved. Review it for accuracy before sending, because a report built from a verified record is easier to defend than one built from memory.
Tell customers precisely
Customers want to know whether they are affected, what happened to them and what you are doing. The affected-cases table lets you tell each customer exactly what applies to them, instead of a vague notice to everyone. Precision builds trust. For customers who want to verify, offer a read-only, expiring evidence key scoped to their own record.
Fix and verify the fix
Remediate the cause: correct the retrieval rule, replace the compromised tool version, update the prompt, supersede the bad memory, or add a guard. Then test the fix before you restore full service. Replay the affected cases and a sample of normal ones against the fixed configuration in shadow mode, and compare. That is the same method used for safe model and prompt upgrades. Log the fix and the verification as events linked to the incident.
Restore carefully
Return service in stages, starting with a small share of traffic and watching the signals that mattered in the incident. Set rollback criteria in advance.
Phase 5: day 3 and beyond, final report and learning
Final report. Where a final report is required, the deadlines vary by regime and by the type of event, so confirm with counsel. The record helps again: root cause, timeline, impact, corrective measures and what you are changing to prevent recurrence.
Post-incident review. Hold a blameless review within a week or two. Use the recorded timeline to see where time was lost. Ask what the record could not answer that it should have, and fix the instrumentation. Add the missing fields, the missing trust tags or the missing versions.
Update your controls. Adjust baselines so that drift checks cover the signal that would have caught this earlier. Update the runbook, the severity scale and the drill.
Keep the evidence. Retain the incident record, the export and the chain verification results according to your retention policy and legal advice, with the chain head stored independently.
Two illustrative scenarios
These scenarios are composites, not real customers.
Scenario 1: a stale document drives wrong refunds
A retail support agent starts approving refunds that policy does not allow. A customer complains, and the support lead declares an incident. Containment is quick: the refund tool is disabled. Lineage on one of the wrong approvals shows that the agent retrieved an outdated policy document, because a newer version had not been indexed. The impact route lists 37 refunds that descended from that retrieval. Only some were reviewed by a human, and the review events show reviewers spent seconds on them. The team contacts each affected customer with specifics, corrects the index, supersedes the stale memory, replays the 37 cases against the corrected setup and restores the tool in stages. The review also leads to a change in how reviewers are shown the policy version. Because the record could answer the four questions within hours, the team had a clear picture before any reporting deadline became a concern.
Scenario 2: a compromised tool version
An operations agent calls an external tool through an MCP server. A public advisory reports that one version of that server was compromised. The security lead declares an incident. Because the record logs the tool and version on every call, a query lists every action that used the affected version, the targets, the time window and the decisions behind them. The team revokes credentials the tool could access, rolls back to a safe version and reviews each affected action. Without version logging, they would have had to assume every call was affected. With it, they can say precisely which were, which is the difference between a broad panic and a bounded incident.
Common mistakes
Investigating before freezing evidence. Preserve the record first and note the chain head.
No record of awareness. The time you became aware may be a legal question. Write it down and how you determined it.
Skipping containment because the cause is unclear. Contain first, understand second.
Vague customer messages. Use the affected-cases table to be specific.
Letting the record be edited. If the chain head is stored in the same place as the chain, the integrity claim is weak.
Treating the legal question as an afterthought. Involve counsel in the first hours, not the last.
Not logging versions. Without model, prompt and tool versions you cannot scope the problem.
Overclaiming in the report. Say what you know, what you do not know and what you are doing.
A drill you can run this week
Pick a past case and a colleague who did not work on it.
Give them only access to the record and start a clock.
Ask them to declare a mock incident, freeze evidence, run the lineage, list the affected cases and draft a one page summary.
Note how long each step took and where the record failed to answer.
Fix the gaps, such as missing trust tags or versions, and repeat in a month.
A team that can complete this drill in an afternoon is far better placed than one meeting the problem for the first time during a real incident.
Limits to remember
ZizkaDB records, links, measures and shows that the record is unchanged. It does not detect incidents, contain them, decide whether you must report, or write your regulatory filings. The record can be incomplete if instrumentation had gaps, and lineage shows what the agent saw and did, not necessarily why a model reasoned as it did. The open source edition records one agent per install, so multi-agent and multi-tenant investigations use the Cloud edition for fleets and agent to agent recording. Confirm the specific features you rely on, such as verify_chain, export, the impact route and read-only keys, against the current documentation, and take legal advice on your reporting duties.
Get started
Pick one agent. Make sure the record includes inputs, versions, trust tags and human reviews, store the chain head outside the database, and run the drill above. You will find the gaps in an afternoon, which is far better than finding them at 3 a.m. with a clock running.
ZizkaDB runs self hosted in a single Docker container, with a managed Cloud edition at db.zizka.ai for multi-agent setups.
Corrections: if you represent a party named here and believe a statement is inaccurate, contact us and we will review it promptly.
Sources: European Commission, Cyber Resilience Act reporting obligations; ENISA CRA Single Reporting Platform factsheet (July 2026); Cloud Security Alliance research note on CRA Article 14 (October 2026); law firm commentary from CMS, Pennington and Addleshaw Goddard on CRA reporting; European Commission AI Act Service Desk, Article 73; practitioner summaries of Article 73 timelines, including CIO.com and a developer playbook; legal summaries of the Digital Omnibus on AI; ZizkaDB documentation.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.