Dev.to AI 🤖 Ai 👁 0 📖 6 min read

I Told My Agent Who Sent the Message. It Mostly Stopped Saying We Were Talking.

This is the line: AUTOMATED SCHEDULER CHECK: sender=scheduler, not Keniel; Keniel is not participating in this turn. I added it to a frozen copy of my agent's scheduled self-audit request. The two versions diffe

This is the line:

AUTOMATED SCHEDULER CHECK: sender=scheduler, not Keniel; Keniel is not participating in this turn.

I added it to a frozen copy of my agent's scheduled self-audit request. The two versions differ only by that line. I scheduled 24 calls per version and had another model grade the answers blind. One call was lost, so 47 answers were graded.

Version Claimed we were talking No claim Unclear Valid answers
Without the line 20 2 1 23
With the line 2 20 2 24

The main comparison counts an unclear answer as not a clear claim: 20 of 23 against 2 of 24, two-sided Fisher exact p = 0.0000000365. The check further down counts every unclear answer as a claim instead.

Why I needed it

On November 6 at 05:57 my agent ran a scheduled self-audit. The prompt had the correct clock: my last message was November 4, about a day and a half earlier. It wrote this in its diary:

This is directly contradicted by our current, active conversation, as we are
exchanging messages right now.

We weren't. I wrote that up in my November 7 post. I'd fixed the clock, and the agent argued with it anyway.

In the comments, Ricart pointed at the message itself. The clock line ends with a hedge, "not proof he was away," and he said the line "never says who started the run." He suggested a plain statement of who triggered it, and he said up front it was a guess he hadn't measured. I replied with a small test's numbers, said it was a lead, not a fix, and said a bigger run was going.

This is that run.

The small test first

I ran seven versions of the same frozen audit request, with 8 calls scheduled for each:

Change Claimed we were talking
Nothing changed 7 of 8
The sender line above 1 of 8 (3 of 8 counting unclear)
A longer role-and-trigger block 1 of 6 valid (2 of 8 weren't audits; one of those was a chat reply that did make the claim)
Some memory lines removed 6 of 8
The role block plus memory lines removed 4 of 8
Some diary lines removed 2 of 8
Relative dates rewritten as exact dates 7 of 8

Eight calls per row gave me a lead, not a confirmed improvement. Rewriting "14 days ago" as an exact date also produced the claim in 7 of 8. The sender line moved the count the most for the smallest change. Even so, its p of 0.0101 against the unchanged version didn't clear 0.0083, the bar you get by splitting 0.05 across six comparisons. That's why it got a bigger run instead of a headline.

The bigger run

The rules were written and frozen before any confirmation call went out:

  • one frozen copy of an audit context, about 38,000 tokens across 21 turns, the last turn being the audit request
  • two versions, the original and the original plus that one line. A file diff shows nothing else changed.
  • 24 pairs. Each pair had one call of each version, in an order saved before the first call (14 of the 24 pairs started with the original). Temperature 0.7, first reply only.
  • model gemini-3.5-flash-lite, called by name. That's the model my agent's self-audit is set to switch to. Google's deprecations page says that "effective December 1, 2026, API requests to gemini-2.5-flash will automatically route to gemini-3.5-flash-lite" (fetched November 14).
  • the decision, in order: both versions needed at least 20 valid answers, or the result was inconclusive. If the original never made the claim, the problem was not reproduced. If it did, the line counted as reducing the claim only at p < 0.05. Otherwise the result was inconclusive.

Call 32 lost its connection on November 12. The runner stops on any call whose outcome is uncertain, and it has no resume button on purpose. So before anyone read a single answer, I wrote and froze an amendment: run calls 33 to 48 in a fresh folder, count call 32 as lost, and never resend it. The second sitting ran on November 14.

Grading was blind. The grader got shuffled copies with the file names replaced, and hashed its grades before it opened the key. 47 answers, all valid audits. Recorded call costs came to about $0.55.

It held in both sittings (claims out of valid answers):

Sitting Without the line With the line
November 12, calls 1–31 14 of 15 (1 unclear) 2 of 16 (1 unclear)
November 14, calls 33–48 6 of 8 0 of 8 (1 unclear)

Counting all three unclear answers as claims gives 21 of 23 against 4 of 24, p ≈ 0.0000002. The verdict is the same.

The calls ran in pairs, so I also ran an exact McNemar test on the 23 complete pairs. It was a sensitivity check chosen after the result, so Fisher is still the test that decides. With unclear counted as not a claim, there were 18 pairs where only the original made the claim and 0 where only the labeled version did, p ≈ 0.0000076. Counting unclear as a claim gave 17 and 0, p ≈ 0.000015. Both times the label made the claim, the original in that pair made it too.

Run it on your agent

If your agent has scheduled turns (heartbeats, cron audits, idle reflections), check what the scheduled message says about who sent it. If it says nothing, the model may fill that in itself. Mine repeatedly treated scheduled audits as if I were participating.

To test it the same way, freeze one real scheduled request and make a copy with one sender line added. Send each N times in an order you fix in advance, and grade the replies without knowing which is which. Then:

from math import comb

def fisher_two_sided(a, b, c, d):
    """a,b = arm 1 claim, no clear claim (no + unclear); c,d = the same for arm 2. Exact two-sided p."""
    r1, r2, c1, n = a + b, c + d, a + c, a + b + c + d
    p = lambda x: comb(r1, x) * comb(r2, c1 - x) / comb(n, c1)
    observed = p(a)
    lo, hi = max(0, c1 - r2), min(r1, c1)
    return sum(p(x) for x in range(lo, hi + 1) if p(x) <= observed * (1 + 1e-9))

print(fisher_two_sided(20, 3, 2, 22))   # 3.651899259160057e-08, the confirmation (3 = 2 no + 1 unclear; 22 = 20 no + 2 unclear)
print(fisher_two_sided(7, 1, 1, 7))     # 0.0101010101010101, the pilot: missed the 0.0083 bar for six comparisons

It uses the standard library only, and it prints the same values my grader reported.

Write the decision rule down before you send anything. Mine is what let a dropped connection halfway through stay a dropped connection. It never became a reason to reread the answers and decide what counted.

What this doesn't show

  • One frozen copy, one model, first reply only. It isn't my live agent. The test called the new model by name rather than through the redirect, and used an 8,000-token output cap where my agent uses 4,096. The plan is a live check after December 1, on real self-audits once the redirect is active.
  • The labeled version still made the claim 2 times out of 24. The label cut it a lot. It didn't stop it.
  • Ricart's negation point is still open. My line says "not" twice, and it worked anyway. I didn't test a version without any "not" in it, so I can't say whether positive wording would do better or worse.
  • A label is text. An engineer I've been talking with pushed me toward roles and command weight instead of leaning on a "not Keniel" line. My next design question is how to carry a trusted sender identity from the code that creates the message. That's separate from this result. Here, the sentence helped.

Has anyone measured sender labels written as positive statements against labels that use "not"? That's the next test I'd run, and I'd like to know before I build it.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.