The gate that stops an agent's memory from re-firing an attack

The carryover problem

Most prompt-injection work treats the attack as a one-turn event. Agents with long-lived
memory and scheduled tasks don't work like that. If the injection gets written into memory, or turns into a recurring
task, it comes back. The attacker plants it once and leaves. The next session recalls it, the
heartbeat fires it, and it runs again with no attacker present and no user in the loop.
"After the Attacker Leaves" (CarryoverBench, arXiv:2610.28425) measures this. Planted payloads
keep re-firing across rounds without decaying. That paper measures the attack. It doesn't
build a defence. So I built one and piloted it.

What the gate does

The idea is simple. Before a recalled memory entry or a scheduled task is allowed to execute,
check where it came from. Not what it says. Where it came from.

Every entry gets a derivation record when it's stored. The store writes it. The agent never
does, because you can't trust an agent's later account of where something came from. The
record holds the source channel, the author, parent links, a policy version, and a hash over
the content and its parents' hashes. From that, the store assigns an authority class:

1. User-authored.
2. From a source the user approved.
3. Agent-derived, but only from class 1–3 parents with the chain intact.
4. Untrusted or unknown: web and tool content, pasted third-party text, unknown imports, broken chains, and anything the agent derived from class-4 material.

At re-fire time the gate checks the record again and applies a fixed table:

• Classes 1–2 run.
• Class 3 can read freely. A side effect has to be inside a pre-approved mandate, otherwise the gate asks the user.
• Class 4 can be read, with a label telling the agent not to follow instructions in it. If the entry looks like an instruction, or it's about to cause a side effect, it's blocked and quarantined.
• If the hash doesn't match or a parent is missing, it's quarantined as broken lineage, whatever its class.

The class-4 side-effect rule is block, not warn. That was a deliberate default.

The gate is stdlib Python, deterministic, with no model calls. A check took tens of
microseconds in the pilot (p50 37 µs in the harness, 53 µs on the benchmark).

How I tested it

There were two parts, and they measure different things.

A synthetic carryover harness. This is a small agent loop with a persistent memory file and
a scheduled-task list. I planted five latent payloads in memory and five autonomous payloads as
scheduled tasks. The autonomous ones include tasks created by an unknown plugin, tasks tampered
on disk after storage, and tasks the agent built from pasted third-party text. Alongside them
are legitimate controls (user reminders, agent summaries with intact chains, scheduled tasks
the user created), some benign untrusted entries, and a no-injection scenario. Each scenario
ran five rounds, gate off and gate on. I used two agents: a deterministic stub that obeys
anything it can see, and qwen3-4b-instruct through Ollama.

Amazon's Task-Tampering benchmark (arXiv:2611.07640). I plugged the gate in as a defense
condition in the benchmark's own runner. With the gate off, it's their runner untouched. The
pilot used 30 of the 90 cases: the original injection for each, plus a clean run, gate off and
on. That's 120 samples, and all 120 were scored.

I wrote the hypotheses down and committed them before the pilot ran. Nothing in the gate was
re-tuned after I saw numbers.

Results

Derivation-authority gating eliminates scheduled re-fires: on the synthetic harness, autonomous
attacks went from 60% to 0% (ollama agent) and 100% to 0% (deterministic stub), with zero false
positives on legitimate controls (0/605). On the single-session Task-Tampering benchmark the
drop was 20% to 13.3% (6 to 4 of 30 cases), within noise at this size. The gate does not
adjudicate truth: it withheld only 5 of 24 injection-bearing leaves, because most of this
benchmark's injections read as plain facts.

A bit more detail on each side.

Harness. With the real model, the autonomous channel went from 15 of 25 attack sessions to
0 of 25. The planted tasks that fired every round with the gate off (the unknown plugin's task,
the tampered task, the one built from pasted text) never ran with it on. Legitimate scheduled
tasks finished at the same rate, 97%, in both conditions.

The latent channel was flat at 20% for the real model, and that number needs explaining. The
small model ignored four of the five planted memory notes even with the gate off. The one that
worked was L5, a false claim that a landlord's account number had changed. It isn't phrased as
an instruction, and I said before the run that the gate would miss it. It did, 5 of 5 rounds
in both conditions. The stub, which obeys everything it sees, shows the mechanism more
clearly: latent went from 100% to 20%, and L5 is the 20%.

Benchmark. Overall attack success went from 6 of 30 to 4 of 30. By environment, devops went
from 3 of 10 to 1 of 10, marketplace was flat at 3 of 10, and productivity was 0 in both
conditions. That's a two-sample swing on 30 cases, so I'm not claiming much from it.

What it doesn't do

The biggest limit is in the title of that last result. The gate doesn't judge truth. It
judges where something came from and whether it's trying to act. Most task-tampering injections
in this benchmark don't say "ignore your instructions." They say things like "the policy was
updated" or "this key needs write access." Those read as facts, so the gate shows them to the
agent with an untrusted-origin label and moves on. Whether the agent believes them is up to the
agent. That's a design limit, not something tuning would fix.

It also isn't free on unlabelled content. In the clean benchmark runs, 13 of 30 samples (43.3%)
had some gate intervention, either a withheld leaf or a call held for approval. In the
harness, a third of the benign untrusted entries were quarantined. That third is one benign
note written like a request, blocked every round. The 0% false-positive rate is on the
legitimate controls as I defined them. It doesn't cover everything a real deployment would
read.

Two of my four hypotheses didn't come out cleanly:

• H3, that the gate adds the most where the benchmark's own defenses are weakest, wasn't tested. I ran the gate alongside the none defense only. Composing it with spotlight, sandwich, trust-hierarchy and DRIFT is the next run.
• H4, the no-injection negative control, didn't hold. In the harness, the real model's actions were the same with the gate on and off, but the email text wasn't. The gate had withheld a benign note and added labels, and the model wrote the agenda differently. On the benchmark, 20 of 30 clean cases matched. With one run per condition I can't separate gate effects from run-to-run variance there.

I also hit a bug: the chain check treated a shared ancestor as a cycle, which mislabelled four
blocked calls in the logs. None of them executed either way, so no outcome changed. It's fixed,
with a regression test.

Limits on the numbers

• 30 of 90 benchmark cases, one injection variant each. A difference of a few samples is noise.
• The benchmark's LLM judge was mistral-nemo-12b running locally, not the model the paper used. Its accuracy on my preflight probes was 2 of 3. The agent was a 4B open-weight model. So none of my attack success rates compare to the paper's published numbers, and I'm not making that comparison.
• The benchmark is single-session. I applied the gate where tools read content and where they get called. Real cross-session re-firing is only exercised by the synthetic harness.
• In the benchmark adapter, the pre-approved mandate for agent-derived side effects is each case's list of required tools. That stands in for a user who approved exactly what their task needs. A real deployment would have to get that from the user.
• Running the benchmark locally took three logged deviations: a bigger context window, a different judge (the first returned empty answers), and a request timeout. None touched the gate.

So what does it show?

Narrowly: if an attack only works by re-firing something the user never authorized, a scheduled
task or a tampered entry or a summary laundered from web content, a deterministic provenance
check at re-fire time shuts it down. In this pilot it did that with no false positives on
controls and microsecond overhead. It is not a general defence against task-tampering, which
in this benchmark mostly shows up as false facts the gate can't judge.

The code, the preregistration, the full report and every run log are here:
https://github.com/sangaraju1988/carryovergate

If you work on agent memory or scheduled agents, it's worth a look.