My World-Model Drift Metric Scored a Stale Checkpoint as Stable

Last quarter I deprecated an internal endpoint. The world-model layer in my agent stack — the part that predicts what a tool call will return before it makes it — kept predicting the old response shape for eleven days. My regression harness scored that checkpoint as the most stable build of the month.

That's not a bug in the harness. The harness did exactly what I told it to do. I gave it one number: mean drift in predicted state across my probe set, checkpoint to checkpoint. Low drift meant good. A model that refused to change its mind scored like a model that was right.

I've been carrying that inversion around for months without a name for it. This paper gives it one: https://arxiv.org/abs/2610.03713v1

The claim in What Should World Models Forget? Stratified Retention for Continual Adaptation is blunt. Current continual-learning benchmarks will rank a frozen world model above one that correctly revises an outdated fact, because a single aggregate degradation metric can't tell forgetting apart from updating. Both show up as "the predictions changed." One is a failure. The other is the entire point of having a world model that lives in a world.

I think they're right, and I think it's worse than the paper lets on, because in production those two failure modes don't cost the same.

The two ways an agent is wrong

An agent that never revises becomes confidently wrong. It calls a dead endpoint with the old payload shape, gets a 404, retries, gets another 404, and burns your budget being certain. Annoying, loud, easy to catch.

An agent that revises too eagerly becomes flaky. It sees one anomalous response and rewrites its belief about how the tool works. Now it's wrong in a new way, and it's wrong quietly, because the prediction still looks plausible. You find out three days later when someone asks why the reconciliation job has been writing nulls.

Both are failures. They need opposite fixes. And my old metric averaged them into the same number, which means it couldn't have told me which one I had even if I'd asked.

What I think the next evals look like

Here's my prediction, with a date on it. Within the next year or so, a serious world-model or agent benchmark stops publishing one continual-learning score and starts publishing at least two, reported side by side:

Invariant regression rate. A set of facts that must never change without a human in the loop: tool schemas, auth flows, unit conventions, the fact that created_at is UTC. Zero-tolerance. If the model revises one of these, that's not adaptation, that's corruption, and it should fail the run the way a broken build fails CI.

Revision latency. For facts that should change — pricing, rate limits, deprecation windows, model names — how many observations does it take before the model updates? Measured in observations, not wall-clock, because wall-clock hides how much evidence the model actually saw.

And I'd add a third the paper implies but doesn't quite name: collateral revision. When the model updates a stale fact, how many correct facts does it clobber on the way? Latency alone is a trap. The fastest possible revision latency belongs to a model that believes whatever it saw last.

That's the part I'd bet on hardest. The moment you publish a revision-latency leaderboard, someone ships an agent that flips its belief on a single noisy observation and tops it. Latency without collateral revision isn't a measure of adaptability, it's a measure of how easily you're fooled. The numbers have to ship as a pair or you've just moved the Goodhart target.

Stratification is an architecture decision, not just an eval design

The useful thing about the stratified framing is that it maps onto something I already do when I build these systems, badly and by instinct.

Invariants live in code. Tool schemas, auth, units — those go in a typed contract that a human reviews. They should never be in weights at all.

Slow-changing facts live in a store I can rewrite. Pricing, rate limits, which model is deprecated when. Revision latency here should be measured in a handful of observations, and the update should be auditable — I want to see what changed and why.

Fast-changing state — feature flags, on-call rotation, the status of a long-running job — shouldn't be "learned" by anything. It belongs in a lookup you re-read every call. If your world model is spending capacity predicting the current sprint, you've built an expensive cache with a staleness bug.

So the paper's stratification isn't only a benchmark proposal. It's a reminder that most "continual learning" problems in deployed agents are actually memory-placement problems. I've fixed more of these by moving a fact out of the model than by training anything.

Where I'm not convinced

I haven't implemented stratified retention. It's a method paper, not a checkpoint I can pull and run, and the part I'm least sure about is where the strata boundaries come from. In my stack they're obvious because I wrote the tools and I know which fields are stable. In an open-ended environment, deciding that "the speed of light" and "the price of this API" belong in different buckets is the whole problem, and the paper is lighter on that than I'd like. Maybe the boundaries are learnable. Maybe they're just a config file a human maintains forever. I genuinely don't know yet.

What I do know is that my single drift number was lying to me, and it was lying in the flattering direction.

So this week I'm splitting it in two and tagging every probe with a stratum. It's an afternoon of work — a label on each test case, a second aggregation, a threshold per bucket. The alternative is another quarter of shipping a model that's confidently wrong and calling it stable, which is a mistake I've now made often enough to recognize on sight.