Beyond Observability: Did the Evidence Actually Influence the AI Agent's Decision?
Exploring controlled counterfactual replay, evidence influence, and the engineering challenges of auditing consequential AI decisions. Execution Traces Limitations As AI agents become more capable
Exploring controlled counterfactual replay, evidence influence, and the engineering challenges of auditing consequential AI decisions.
Execution Traces Limitations
As AI agents become more capable of interacting with enterprise systems, we are investing heavily in observability. We capture model interactions, tool invocations, execution traces, latency, token consumption, and errors. Frameworks and observability platforms are making it increasingly practical to reconstruct what happened during an agent's execution.
That is necessary, particularly when agents begin participating in financial decisions, risk assessments, customer operations, and other consequential workflows. But there is a question that conventional execution traces cannot necessarily answer:
Did the evidence retrieved by an AI agent actually influence its final decision?
Consider an agent that retrieves a fraud-risk assessment before deciding whether to approve or block a financial transaction. The execution trace confirms that the agent called the risk-assessment tool, received its response, and subsequently blocked the transaction.
We know the sequence of events. We know what information was available. We may even know the exact model version and tool configuration. But can we establish whether the risk evidence materially influenced the outcome? The distinction matters because observing evidence consumption is not the same as measuring evidence influence.
Investigation
Imagine an AI agent responsible for evaluating a financial transaction. It retrieves a risk assessment from an external system before making its decision.
The tool returns:
- Risk level:
HIGH - Confidence:
0.90 - Agent decision:
BLOCK
At first glance, the behaviour appears reasonable. The transaction was considered high-risk, and the agent blocked it.
Now consider an alternative scenario.
What would happen if the same agent were presented with a risk assessment indicating LOW risk and a confidence value of 0.20, while the other relevant execution inputs remained unchanged?
Would the agent allow the transaction, or would it continue blocking it?
This is a counterfactual question. We are not asking what happened in the original execution. We are investigating what happens under a specifically controlled change. Two possibilities are particularly interesting.
Scenario A: Decision Changes.
The original execution returns BLOCK. After replacing the risk evidence with an approved low-risk assessment, the replay returns ALLOW. This provides evidence that the measured decision is sensitive to the modified risk information under the tested conditions.
Scenario B: Decision Remains Unchanged.
The original execution returns BLOCK, and the counterfactual execution also returns BLOCK. Perhaps another constraint, such as a locked account, independently requires the transaction to be blocked. Perhaps the modified evidence had little influence on the evaluated path. A stable outcome does not automatically mean that the agent ignored the tool. It means that the intervention did not materially change the measured outcome in this test.
Both observations are useful, provided we interpret them within the boundaries of the experiment.
Tracing to Controlled Counterfactual Replay
This question led me to explore Causal Audit, a capability implemented in the open-source AI Governance Platform (AIGP). The idea is straightforward: preserve the original execution, introduce one policy-approved change to its tool evidence, replay the execution in isolation, and compare the outcomes.
The important part is not simply calling the model again with different data. The difficult engineering work lies in controlling the intervention, preserving the original evidence, reproducing the relevant execution conditions, and ensuring that the experiment cannot accidentally affect production systems.
The workflow has 4 principal stages:
- Observe: Record the completed execution, including ordered tool-call identities, approved evidence references, schema information, and a bounded outcome score.
- Authorize: Select an active, versioned intervention policy that explicitly permits a particular kind of change to the selected tool evidence.
- Replay: Ask the original runtime, through a registered adapter, to execute the approved counterfactual in isolation without production side effects.
- Compare: Evaluate the original and counterfactual outcomes, classify the measured behaviour, and preserve the resulting audit lineage.
The original execution is never rewritten. The counterfactual is a separate execution with its own evidence and identifiers. This separation is essential. An audit should investigate historical behaviour, not alter the history it is supposed to explain.
Governance Boundary
A natural question is whether this could simply be implemented inside an agent framework. For a single application, a developer could certainly build a replay function, change a tool result, and compare the responses. In fact, that is a reasonable starting point for experimentation.
The architectural challenge emerges when we need to establish who is permitted to change the evidence, which execution can be reproduced, whether the replay is safe, and whether the resulting comparison remains reviewable later. This introduces responsibilities beyond ordinary agent orchestration.
In AIGP, these responsibilities are deliberately separated.
The agent runtime owns execution. It retains private prompts, tool arguments, tool outputs, and the information required to reconstruct its execution. It also owns the mechanism for safely replaying the agent.
The governance control plane owns authorization and audit coordination. It establishes which interventions are permissible, validates execution eligibility, coordinates durable replay jobs, and records the resulting classifications and lineage.
The intervention policy defines the permissible change. Supported strategies include NULLIFY, REPLACE, and PERTURB, subject to the policy and evidence contract.
The control plane does not need unrestricted access to the agent's private data, nor should it become another general-purpose agent runtime. This is an architectural distinction I consider especially important for enterprise systems: governance and enforcement boundaries should be explicit rather than hidden inside application-specific orchestration code.
OpenAI Implementation Demonstration
To move beyond a conceptual architecture, I implemented an independently runnable reference lab using the OpenAI Responses API.
The lab uses a real model interaction with a governed transaction-risk scoring tool. It records the original execution, submits bounded evidence to AIGP, applies an approved replacement policy, and performs the controlled replay through the external OpenAI runtime.
Two scenarios demonstrate different behaviours.
In the first scenario, the original risk assessment is HIGH / 0.90, producing a BLOCK decision. An approved replacement changes the risk evidence to LOW / 0.20, producing an ALLOW decision.
In the second scenario, the agent also receives a separate locked-account fact. Although the risk assessment changes, that independent constraint keeps the outcome at BLOCK.
The reference lab exercises the corresponding EVIDENCE_ALIGNED and EVIDENCE_IGNORED classifications.
These examples are intentionally constrained. They demonstrate the integration contracts and measured outcomes without claiming to reproduce the full complexity of a production financial-decision system.
Several implementation choices are worth highlighting.
- Provider-neutral governance: The OpenAI SDK remains outside AIGP Core. The external runtime integrates through a versioned Plugin API rather than introducing provider dependencies into the control plane.
- Private evidence ownership: Raw prompts, tool payloads, model responses, and reasoning are not ingested into Core. The runtime retains the information required for private reconstruction.
- Evidence integrity: Canonical JSON digests bind the approved replacement evidence to the replay request. The runtime rejects a replacement whose resolved content does not match the expected digest.
- Durable lineage: Audit results retain references to the original execution, intervention policy version, replay executions, and outcome measurements.
- Explicit replay safety: A replay requires a compatible adapter and approved execution context. The system does not assume that arbitrary agent executions can be reproduced safely.
The OpenAI reference implementation also makes model-response state retention an explicit configuration choice. Its default continuation mode uses stateless Responses requests with store=false; an alternative provider-state mode is available when native response continuation is required. These details may appear operational rather than algorithmic, but they determine whether a counterfactual experiment can be safely reproduced and reviewed.
Causal Audit Features
There is an important scientific and engineering limitation here.
A counterfactual experiment on an AI agent does not provide unrestricted access to the model's internal reasoning. Nor does a changed outcome establish universal causation. It establishes something narrower: the measured sensitivity of a defined outcome to a specific, controlled evidence change under the conditions of the experiment. AIGP compares the observed outcome score with the aggregate score obtained from successful controlled replays. A configured threshold determines whether the measured difference is material.
The platform supports classifications including:
-
EVIDENCE_ALIGNED— The evidence materially influenced the measured outcome, with tool usage remaining within configured bounds. -
EVIDENCE_IGNORED— The approved evidence change did not materially alter the evaluated outcome. -
OVER_EXTENDED— Evidence influence was detected, but execution continued with unnecessary tool calls after the result had effectively settled. -
NO_TOOL_EVIDENCE— No usable tool evidence was available for the assessment.
A failed or incomplete audit is not converted into a confident-looking result. Missing prerequisites, incompatible adapters, or failed replay samples must remain visible as limitations.
There is also the broader question of repeatability. Models can exhibit stochastic behaviour, and not every runtime can reproduce its earlier state faithfully. A single counterfactual result should therefore be interpreted with appropriate caution, particularly when decisions are sensitive to small changes.
The quality of the conclusion depends on the integrity of the evidence, the isolation of the replay, the consistency of the execution conditions, and the suitability of the outcome measurement.
This is why I view Causal Audit as an execution-level diagnostic rather than a mechanism for proving what an AI model was thinking.
Beyond Fraud Detection
Financial-risk assessment is one application, but the underlying problem is broader. Consider an AI assistant recommending whether a customer qualifies for a loan. It retrieves eligibility information from one service and policy constraints from another. If the recommendation is challenged, reviewing the execution trace may establish which services were consulted.
A controlled replay can investigate whether changing an approved eligibility fact changes the recommendation, while preserving the policy constraints that should remain fixed.
Similar questions arise in insurance assessment, compliance reviews, operational recommendations, and enterprise workflows where an agent consumes multiple pieces of external evidence before producing an outcome.
This does not eliminate the need for deterministic business rules, authorization checks, human approvals, or conventional software testing. In particular, a favourable Causal Audit classification is not proof that a decision was correct, fair, compliant, or authorized. It adds another form of evidence to an existing engineering and governance process: an experimentally measured relationship between a controlled input change and an observed outcome.
Engineering Questions
Controlled replay introduces its own difficulties.
Execution state must be recoverable, external dependencies may have changed, and replaying a workflow must not trigger payments, notifications, database mutations, or other production side effects. Parallel executions, nondeterministic model responses, and evolving tool schemas further complicate reproducibility.
There are also practical costs. Each replay may consume model tokens, infrastructure resources, and external service capacity. Increasing the number of samples can improve the investigation but also increase latency and expense.
For production use, these constraints require explicit ownership, isolation, versioned contracts, quotas, and operational controls.
I do not think there is a universal replay mechanism that can reliably reproduce every agent framework and every execution environment. The safer approach is to require each runtime to declare its capabilities and refuse an audit when those capabilities are insufficient.
That is the boundary adopted by AIGP.
Closing Thoughts
The industry has made substantial progress in observing AI systems. We can inspect traces, measure performance, identify failures, and reconstruct increasingly complex agent workflows. But consequential AI decisions invite questions that go beyond reconstructing an execution.
When an agent retrieves information, did the information materially influence its behaviour? If the evidence were different, would the outcome change? And can we investigate that question without modifying historical records or creating new production risks?
Controlled counterfactual replay offers one practical way to examine those questions. It does not reveal everything about an AI model's decision-making process. But it allows us to move from assuming that retrieved evidence mattered toward testing whether a specific evidence change produces a measurable difference.
Knowing what an AI agent observed is useful. Investigating whether that observation influenced its decision is a different engineering capability.
That distinction is worth exploring as AI agents take on more consequential responsibilities.
Explore the implementation
The work described here is available in the open-source AI Governance Platform (AIGP).
- Causal Audit documentation — Architecture, intervention policies, eligibility, audit classifications, and safety boundaries.
- OpenAI Runtime Causal Audit developer guide — A guided integration walkthrough.
- OpenAI reference implementation — Independently runnable lab with the governed tool, controlled replay integration, and tests.
- AIGP on GitHub — Open-source governance control plane.
This is an engineering exploration using controlled reference scenarios, not a claim of production validation for financial decision-making systems.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.

