I gave my AI agent a memory. It started seeing patterns that weren't there.

Memory is the feature everyone wants to add to their agent: remember what happened, stop repeating mistakes, accumulate context across sessions. I have one. It works. It also does something nobody warns you about — it sees patterns that don't exist.

Not occasionally. Structurally. And the reason is the same machinery that makes memory useful in the first place.

Retrieval isn't recollection

A memory store doesn't work like recollection. When an agent "remembers," what's actually happening is similarity search: embed the current situation, find the nearest stored items, treat proximity as relevance. It's fast and fuzzy and it has a property that causes real trouble — every result looks like a match. The system returns neighbors, and neighbors always look related to the query, because "nearest" is the only filter that ran.

Whether the relationship is real is a second question, and nothing inside the retrieval mechanism can answer it.

Watch what that does over time. Three failures, weeks apart, sharing a few surface tokens — a timeout, a flag, an identifier — get clustered into one "recurring pattern." The store didn't find the pattern. It manufactured it, because clusters are the shape similarity produces. Then a consolidation pass summarizes the cluster, the summary loses the provenance of the originals, and now there's an entry that reads like an established fact. The pattern was never verified. It was generated — because generating patterns is what nearest-neighbor memory does when you let it talk to itself.

The human version of this is the false pattern recognition that makes conspiracy theories feel coherent: connect enough dots and a shape emerges whether or not the world put it there. Agents get the same failure mode, except the dots are served to them as "things you already know."

Sleep-time compute amplifies it in both directions

The industry's answer to idle agents is offline processing, and it's a real answer. Letta's Sleep-time Compute paper showed agents reworking their context during downtime can cut the test-time compute needed for the same accuracy by about 5x on their stateful benchmarks, and actually improve accuracy by 13-18% while doing it — the offline pass turns raw context into learned context that's cheaper to reason over later.

That's the good direction. The catch: an offline pass amplifies whatever it finds. If the store contains manufactured clusters, the consolidation pass mints them into "facts" with extra confidence and compressed provenance. A sleep cycle that promotes unverified patterns is a confabulation engine with a persistence layer.

So the question isn't whether agents should think while idle — they should. It's what they're allowed to do with what they think up.

The missing tier

Memory systems today mostly ship two trust levels: observed and verified. What offline processing needs is a third — hypothesized. The sleep pass generates candidates: possible recurrences, suspected contradictions, associations between clusters that look suspiciously close. Those land in the store tagged as hypothesized, provenance-flagged, carrying zero authority. They only promote to trusted on explicit verification while the system is awake and can actually check.

Generation is cheap and untrusted; checking is what confers truth. Proof assistants figured this out decades ago — a theorem proposer can generate a thousand candidate steps, and none of them count until the verifier accepts one. Nobody treats a candidate proof as a theorem. Memory systems shouldn't treat a candidate pattern as a fact either.

The second half of the trick is that the noticing itself is cheap. Staleness detection, recurrence clustering, contradiction flags, promotion candidacy — these are all classification and ranking problems at volume. A small, calibrated model can scan the whole store and produce a short ranked list of things worth real attention. The expensive reasoning — the "is this pattern actually true" — only runs on the handful that survive the filter. Cheap machinery notices everything; expensive machinery judges the shortlist.

The dream proposes. The waking state approves.

Holding what's true

An agent with memory but no verification tier doesn't have memory — it has a rumor mill with a persistence layer. Every session it wakes up more confident about patterns nobody ever checked.

The systems that get this right will look less like diaries and more like science. Observation, hypothesis, verification — and just as important, the willingness to forget. Salience that decays unless re-confirmed isn't a limitation; decay is what keeps the true entries findable in a store that would otherwise fill up with confident noise.

The interesting question was never "can the agent remember." It's whether the agent can tell the difference between something it saw and something it made up — and whether the system is honest enough to write that difference down.