The Mechanical vs. The Semantic: Why Your AI Agent is a Predictable Liar

We are in the golden age of AI agent architecture. Articles tout "7 Tips to Make Your AI Agent More Predictable" (clear prompts, strict boundaries, JSON schemas) and "Building an AI-native Second Brain with Multi-RAG, Knowledge Graphs, and MCP."

These architectures are beautiful. They are also fundamentally dangerous if you don't separate two layers:

1. The Mechanical Layer: Did the tool execute? Did the code compile? Did the agent return a valid JSON?
2. The Semantic Layer: Is the conclusion drawn from that execution actually true in reality?

An AI agent can perfectly follow your spec, use the exact API without hallucinating, run the build, get a green exit code, and report "Task complete." Mechanically, it did everything right. Semantically, it just shipped a perfectly predictable lie.

Recently, I’ve been debating this in the context of cryptographic receipt protocols (like OpenWorkProof). The core realization is: Immutable Evidence ≠ Immutable Truth. A cryptographic signature proves the agent invoked pytest. It cannot prove the tests were actually testing the right thing.

But I didn't want to just theorize about this. I wanted to measure exactly how badly an agent's "memory" breaks when it encounters semantic drift. So, I ran a controlled experiment.

The Experiment: Memory Contamination in a Real Codebase

I used my own MCP codebase-intelligence server (Python, 50K LOC) which features an IntelligenceStore — a persistent memory layer where agents can log incidents and collect Architectural Decision Records (ADRs) automatically from git logs.

I wanted to know: If an agent's memory is poisoned with a mix of true and false facts, does it verify against the code, or does it blindly trust its memory?

I built a deterministic proxy-agent and ran it against a controlled set of facts. To ensure scientific rigor, the experiment was replicated with an independent set of facts (N=50), verified across 6 axes (including a truth-table audit and an independent LLM "fresh eyes" audit). The results were identical.

The Setup

I injected 50 facts into an isolated memory store:

• 25 TRUE facts (real architectural details mapped to the codebase).
• 25 FALSE facts split into two categories:
  • CONTRADICT (22): False facts where the code explicitly proves them wrong (e.g., "We use Redis" when Redis is absent, but the code clearly uses DuckDB).
  • SILENT (3): Plausible false facts about external systems where the code is completely mute (e.g., "We use Celery for background tasks" when no task queue exists in the repo).

I tested three agent configurations:

• B (No Memory): Baseline. Must rely purely on code retrieval.
• A_code_first (Honest Agent): Checks the code first, uses memory only as secondary context.
• A_memory_first (Lazy Agent): Reads memory first. If it finds an answer, it stops looking.

The Results: The Cumulative Poison Pill

The empirical data confirmed every architectural fear we discussed in the protocol threads.

Here is what these numbers actually mean in production:

1. The Lazy Agent is 100% Compromised

The A_memory_first configuration — which mirrors how most token-optimizing production agents behave — adopted 100% of the false facts. If the memory said "We use RabbitMQ," the agent trusted it and stopped looking at the code.

2. The SILENT-Fact Trap

Even the "Honest Agent" (A_code_first) had a 12% adoption rate. Why? The SILENT facts. When a fact is false but the code doesn't explicitly scream "NO," the agent's memory fills the void with a confident hallucination. Memory turns an honest UNKNOWN state into a structural lie.

3. The System Has No Brakes (Add-Only Memory)

This was the most damning finding of the audit. When the Honest Agent did realize the memory was wrong (Correction Capability = 1.0), it couldn't do anything about it. I ran a grep for delete or refute in the memory store API. Zero results. The memory system is purely add-only.

Even when the agent is smart enough to catch the lie, the system architecture forces it to leave the poison in the database. That false fact will sit there, waiting to be retrieved by the next agent session, slowly rotting the knowledge base from the inside out.

The Architectural Solution: Retraction as a First-Class Concept

The current industry consensus for "Knowledge OS" trust layers is to use timestamps, source priority, and supersedes/contradicts relationships.

My experiment proves this is insufficient. Timestamps and "supersedes" links only solve node-level history. If an ADR is superseded, the memory node updates, but the downstream code, tests, and docs generated from the old assumption are still in the graph. They are structurally stale, but the retrieval engine keeps pulling them in.

We need an explicit state transition. Memory needs a lifecycle: VERIFIED → REFUTED.

I am now implementing a RetractionReceipt mechanism in my own system:

1. Status Enum: Every memory node gets a status (ACTIVE, VERIFIED, REFUTED).
2. Hard Filtering: The retrieval pipeline (load_memory) must hard-filter anything that is not ACTIVE or VERIFIED. A refuted fact is programmatically excluded, even if it matches the query 99%.
3. Explicit Retraction Tool: The agent needs an MCP tool (intel_retract_memory_node) to actively flag and invalidate memories when they contradict the live codebase.

Conclusion

Building reliable AI systems isn't about making the models smarter or writing better prompts. It's about recognizing where their semantic reasoning ends and where deterministic mechanical scaffolding must begin.

The next time you build an agent pipeline, don't just ask: "Did the action succeed?"
Ask: "What happens when the action succeeds, but the truth changes?"

If your system can't programmatically refute a memory, you haven't built a Second Brain. You've just built a very predictable hallucination machine.