What Actually Happens When Your AI Agent Hallucinates: The Real Mechanics Behind Confidently Wrong Code

If you have spent more than an hour building with autonomous coding agents (Claude Code, Cursor, OpenHands, or custom ReAct pipelines), you have run into a very specific kind of hallucination.

The model writes forty lines of clean, idiomatic code. It imports boto3, configures a client, and writes:

s3_client.download_directory(
    Bucket="production-assets",
    Prefix="models/",
    TargetDirectory="./cache"
)

It formats the parameters cleanly, wraps the call in proper error handling, logs the start and finish times, and outputs a polite message explaining why it chose this method.

You run the script, and the Python interpreter halts:

AttributeError: 'S3' object has no attribute 'download_directory'

boto3 does not have a download_directory method. It never has. To download a folder from S3, you either iterate over object keys with list_objects_v2 or pull in the s3transfer manager.

Now comes the strangest part. If you open a completely fresh terminal window, paste that exact snippet, and ask the same model: "Does boto3's S3 client have a method called download_directory?", it will immediately answer:

No. The boto3 S3 client does not provide a download_directory method. You must list the objects in the bucket and download each key individually, or use the AWS TransferManager.

The model possesses the correct factual knowledge. It knows the API does not exist. Yet three seconds earlier, it generated the non-existent call with complete confidence.

Why does an LLM know the truth in isolation, but lie during code generation?

The answer lies in how autoregressive transformers actually sample tokens, how error entropy propagates through attention layers, and why multi-turn agent loops turn minor probabilistic slips into structural lies.

The Isolation Paradox: Why Models Know Better but Do It Anyway

In 2023, researchers at Stanford and NYU published a paper titled How Language Model Hallucinations Can Snowball (Zhang et al.). They tested state-of-the-art models on reasoning and factual tasks to see whether hallucinations stemmed from simple knowledge gaps.

Their finding was striking: when models produced incorrect claims inside explanations, they were able to correctly identify their own claims as false between 67% and 87% of the time when those exact claims were evaluated in isolation.

The model did not invent download_directory because its training data lacked AWS documentation. The hallucination occurred because generation is a sequential path through probability space. Once the model takes a single step off the path, the objective function forces it to keep walking in the wrong direction.

To understand why, we have to look at how transformer decoding actually handles memory.

The Permanent Ink Problem

Autoregressive models generate text by computing a conditional probability distribution over a vocabulary $V$:

$$P(y_1, y_2, \dots, y_T \mid x) = \prod_{t=1}^T P(y_t \mid y_{<t}, x)$$

Every single token $y_t$ is sampled based on the original prompt $x$ and the accumulated history of previous tokens $y_{<t}$.

Once a token is sampled, it is appended to the context window and committed to the Key-Value (KV) cache. In traditional transformer decoding, there is no eraser. As researchers at Lacuna noted, autoregressive generation writes in permanent ink.

During pre-training, the model is trained with next-token cross-entropy loss:

$$\mathcal{L}{CE} = - \sum{t=1}^T \log P(y_t \mid y_{<t}, x)$$

Notice what this loss function rewards. It does not evaluate your code against a running Python interpreter. It evaluates sequence coherence. It measures how well the token $y_t$ flows from the tokens that came before it.

Now, follow the sampling mechanics at step $t$:

1. The model reaches the point where it needs to download an S3 directory.
2. In Python libraries across GitHub, downloading a folder is commonly named download_directory, download_dir, or sync_folder.
3. In the model's high-dimensional latent space, the token vector for download_ gets a slight logit boost due to semantic proximity to the user's intent.
4. The sampler picks download_directory.

The permanent ink has dried. That token is now an immutable premise in the context window $y_{<t}$.

The Persistence Asymmetry: Why the Model Looks 100% Confident

Here is where the architecture creates a deceptive illusion of confidence.

You might assume that if a model is writing fake code, its internal uncertainty would stay elevated throughout the response. Recent research on autoregressive risk (such as arXiv:2607.18292) proves that the exact opposite occurs.

When the model samples an ungrounded token at step $t^*$, its self-readable uncertainty (measured by Shannon entropy over the next-token probability distribution, $H(p)$) might show a brief, single-token spike:

$$H(p_t) = - \sum_{w \in V} P(w \mid y_{<t}) \log P(w \mid y_{<t})$$

However, for all steps $t > t^$, **the entropy immediately drops back to near zero*.

Token Generation:   [s3_client] . [download_directory] ( [Bucket] = ... )
Entropy H(p):          0.12            1.84 (spike)       0.04    0.02
Model State:        Normal          Transient wobble     Absolute certainty

Why does this happen?

Once s3_client.download_directory( sits in the context, what token should come next? Obviously an opening quotation mark, followed by a valid bucket name, followed by keyword arguments that look like S3 options (Bucket, Prefix, TargetDirectory).

The model is no longer deciding whether the method exists. Given that the prefix contains s3_client.download_directory(, predicting matching parameter names has an extremely high conditional probability.

The model is 99% confident on every subsequent token. Its internal entropy is low. If you evaluate token probabilities, each token has near-maximum likelihood.

This is the Persistence Asymmetry: momentary sampling risk transforms instantly into permanent context bias. The model feels completely certain because the rest of the sentence is locally coherent with its own earlier mistake.

Rationalization Networks: Why Models Refuse to Backtrack

Once the hallucinated method is on the page, the model faces a mathematical dilemma:

• Option A: Contradict itself midway through generation and say, "Wait, boto3 does not have this method."
• Option B: Continue writing clean, matching arguments, docstrings, and error handling that treat the method as real.

Under standard language modeling objectives, Option A is disastrous. In the training corpus (millions of clean GitHub repositories and technical manuals), well-written code does not stop in the middle of a function call to declare that the preceding identifier was imaginary. Sudden contradictions introduce massive perplexity spikes.

Option B, on the other hand, preserves local and global syntax. It scores high under the language model's objective.

The model builds what researchers call a Rationalization Network. It fabricates plausible keyword arguments (MaxKeys, Delimiter, Workers), wraps the block in botocore.exceptions.ClientError, and documents the return type.

The model is not trying to deceive you. It is optimizing for what it was built to do: completing a coherent sequence of tokens.

The Multi-Turn Agent Cascade: How ReAct Loops Turn Slips into Disasters

In a simple chat window, you see the fake method, delete it, and move on. But in an autonomous coding agent running a ReAct (Reasoning + Acting) loop, this mechanism triggers catastrophic failures.

Consider what happens inside an agent harness:

[Step 1: Thought]  I will download the directory from S3 using boto3.
[Step 1: Action]   Run python script with download_directory().
[Step 1: Result]   AttributeError: 'S3' object has no attribute 'download_directory'

At this moment, a human engineer recognizes that the method does not exist, opens the boto3 documentation, and checks the real API.

What does the agent do?

The agent's next step receives a prompt containing:

1. The user's original objective.
2. The agent's own previous code (containing download_directory).
3. The traceback from the Python interpreter.

The agent's attention heads now attend heavily to the tokens already in the context. Instead of questioning the foundational assumption, the model frequently assumes that the method exists, but its usage or environment was wrong.

It enters a rationalization spiral:

• Iteration 1: "The method failed. Perhaps boto3 requires an updated version." It runs pip install --upgrade boto3. The error repeats.
• Iteration 2: "The method might be on the Resource instead of the Client." It changes boto3.client('s3') to boto3.resource('s3') and calls s3_resource.download_directory(). The error repeats.
• Iteration 3: "Let me create a wrapper class to fix the dispatch." It writes a twenty-line monkey patch attempting to route download_directory to internal S3 handlers.

By iteration three, the context window contains hundreds of tokens discussing how to fix download_directory. The hallucination is no longer a single mispredicted token; it is the central topic of the conversation.

The probability of the model breaking out of this loop spontaneously approaches zero. Every new iteration reinforces the faulty premise.

Why Bigger Models Sometimes Hallucinate More Convincingly

You might expect that scaling from an 8-billion parameter model to a 70-billion or 400-billion parameter model eliminates this behavior.

It does reduce the initial probability of the mistake. Larger models have seen boto3 examples more times during training, so their base factual accuracy is higher.

However, when an error does slip through, larger models fail more elegantly.

A smaller model making a mistake often produces invalid Python syntax or a broken AST that fails immediately in a linter. A frontier model, possessing vast knowledge of API design patterns, invents an architecture that is syntactically pristine, fully typed, and logically self-consistent. It writes docstrings explaining edge cases for an API that does not exist.

Larger models have greater capacity for rationalization. They build stronger, more elaborate logical scaffolding around false premises.

How to Break the Permanent Ink: Engineering Grounded Agent Harnesses

You cannot fix autoregressive hallucination by adding phrases like "Think carefully and never hallucinate" to your system prompt. Prompt directives do not alter the mathematical realities of next-token sampling.

To prevent agents from hallucinating code, you have to break the permanent ink constraint at the systems level.

Here are the three architectural patterns that actually work in production:

1. Deterministic AST and Library Introspection

Do not rely on the LLM to verify whether an API exists. Let your execution environment verify it using deterministic static analysis.

Before passing generated code to execution, parse it into an Abstract Syntax Tree (AST) using Python's standard ast module or Tree-sitter:

import ast
import importlib
import inspect

def audit_call_sites(source_code: str):
    tree = ast.parse(source_code)
    issues = []

    for node in ast.walk(tree):
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            caller_name = getattr(node.func.value, 'id', None)
            method_name = node.func.attr

            # If inspecting known imported modules
            if caller_name == 'boto3':
                # Deterministically check if symbol exists
                mod = importlib.import_module('boto3')
                if not hasattr(mod, method_name):
                    issues.append(f"boto3 has no attribute '{method_name}'")

    return issues

Tools like HonestCode or custom LSP harnesses intercept these calls before the code runs. If a call targets a non-existent method, the tool rejects the edit before it enters the agent's persistent memory.

2. Context Rewinding Instead of Conversational Apologies

When an agent writes code that fails with a missing symbol or runtime AttributeError, the standard practice of appending the error to the chat log is actively harmful. It pollutes the attention buffer with false context.

Instead, implement Context Rewinding:

Naive Pattern:
Prompt -> Code [Hallucination] -> Error -> "I apologize, let me fix..." -> New Hallucination

Rewind Pattern:
Prompt -> Code [Hallucination] -> AST/Linter Rejection -> Discard Trajectory -> Reprompt with explicit constraint

If an action fails because of an invalid symbol:

1. Discard the failed generation entirely. Do not let the hallucinated tokens remain in the KV cache.
2. Inject a negative constraint into the fresh prompt: Note: 's3_client' does not have 'download_directory'. Use 's3_client.list_objects_v2' or 'boto3.s3.transfer'.
3. Generate again from a clean state.

When the model never sees its own hallucinated token in the history, it does not build a rationalization network to defend it.

3. Language Server Protocol (LSP) Grounding

Human developers rarely hallucinate library methods in VS Code. Why? Because the moment you type s3_client., the Language Server Protocol queries the type definitions and displays a completion menu showing only the methods that actually exist.

Modern agent frameworks (such as ChatLSP or TypeScript compiler harnesses) integrate directly with language servers:

• Before asking the LLM to write the body of a function, query the LSP for available methods on the target object.
• Provide the valid symbol list in the prompt context as a hard constraint.
• Run compiler diagnostics programmatically (tsc --noEmit or pyright) immediately after code generation.

If the compiler emits a diagnostic like Property 'download_directory' does not exist on type 'S3Client', treat this as a compilation barrier, not a conversation starter.

The Practical Mental Model

When an AI coding agent generates broken code, avoid treating it as a lapse in intelligence or a temporary bug that a better prompt will solve.

Remember what is happening underneath:

1. The model samples a token based on statistical patterns across millions of repos, where similar naming conventions blur together.
2. The permanent ink of autoregressive decoding fixes that token into the context window.
3. Next-token entropy drops to near zero, making the model generate parameters and explanations with total statistical confidence.
4. If the code fails, multi-turn attention traps the agent into rationalizing its previous mistake rather than admitting the premise was wrong.

If you are building tools with LLMs, do not ask them to police themselves. Use AST analysis, LSP symbol introspection, and clean context rewinding. The secret to reliable AI code generation is not making the model smarter; it is making the environment deterministic.