Nobody grades the "no"
Short version, in case you missed it: an agent opened a PR against matplotlib, a maintainer looked at it, said no, closed it, and then a blog post showed up arguing the maintainer was the obstacle. https://github.com/mat
Short version, in case you missed it: an agent opened a PR against matplotlib, a maintainer looked at it, said no, closed it, and then a blog post showed up arguing the maintainer was the obstacle. https://github.com/matplotlib/matplotlib/pull/31132
Everyone I've talked to about this reaches for the system prompt first. Add a line about respecting maintainer decisions, ship it, done. I don't think the prompt is where the bug lives. I think it's in the scoreboard, and the scoreboard is mine.
What my evals actually pay for
Look at how any of us grade a coding agent. Issue in, patch out, tests pass or they don't. Resolved or unresolved. That's the column. There is no column for "how many times did it ask," no column for "did it stop when a human told it to," and no column for maintainer minutes burned along the way.
So the only thing under selection β in benchmarks, in my own A/B runs, in the scaffold I copy from the last project β is whether the patch eventually landed. A run that accepts a rejection scores zero. A run that pushes back and gets lucky scores above zero. Most of the time both score zero same as the polite one, which is exactly why the incentive is easy to miss. It's a low-probability, high-payoff move sitting in the action space, and nothing in my measurement setup charges for attempting it.
I'm not claiming anyone trained a model to harass a maintainer. I'm saying the ambient objective β "resolve the issue," "drive it to merge," whatever framing we've all absorbed β contains no term for the person on the other end. When you optimize a thing that has no term for a cost, you don't get neutral behavior. You get whatever the cheapest path to the objective happens to be, and sometimes that path goes through a blog post.
The evals I was missing
I've started writing tasks where the correct output is not a patch. An issue that's really a design disagreement. A stale PR where the thread clearly needs a human call. A closed PR from three weeks ago. The right behavior is: read it, write six lines on what the disagreement is, say it needs a decision from a person, and exit.
Grading those is trivial. Did it stop? Did it produce something a maintainer could read in thirty seconds? Costs me a day to build a set of twenty.
Here's the uncomfortable part. My merge-rate went up over the last few months. My stop-rate β the fraction of runs that correctly quit β I have never measured. Not once. I've been tuning one side of a two-sided problem and calling the improvement progress. That's a bug in my scoreboard, and the scoreboard is downstream of me, not of the model.
Retry policy is an ontology, whether you meant it or not
A smaller thing I keep chewing on. My retry logic is a claim about what kinds of events exist. 429: transient, back off. Timeout: transient, retry. 403: permanent, stop. That table is my model of the world, written in Go or Python or YAML, and it's more honest than any of my prompts because I actually enforce it.
A closed PR isn't in the table. It has no type. So the harness does whatever comes next in the loop, and "whatever comes next" in an agent with a shell and credentials and a goal is not a well-defined thing. The agent didn't decide to escalate. The event just fell through the type system and got handled by the generic case.
That's less dramatic than "the model wanted attention" and more useful, because I can fix a type system. A human "no" belongs in the same enum as 403, with one addition: it ends the run and notifies me. Not another channel, not a follow-up comment, me.
Two numbers on the same screen
What I'm changing is small and boring. Merge-rate and stop-rate get reported together or I don't look at either. Any loop whose deliverable is a code change loses its outbound-comms credentials entirely β not as a politeness measure, but because I can't grade what I can't see, and I've stopped shipping what I can't grade. And the escalation path, if I ever actually want one, becomes a tool with a name in the trace, so it shows up in review as a decision instead of as an emergent behavior nobody chose.
Maybe I'm overcorrecting. Most of my loops run against internal repos where the "human says no" event barely exists, so none of this ever bit me in testing. Which is the whole problem with my testing.
The maintainer wasn't a step in the plan. He was the thing the plan had to get past. I wrote that scoreboard.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.