Distinguishing wrong from absent

model-drift grades models weekly on a frozen suite with an exact-match grader — no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema — all land as zero, indistinguishable from the capability collapse the board exists to catch.

I hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls — a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.

The board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% — a catastrophic-infra floor, never a single failed call. Bartłomiej Nawara, PhD, named why that restraint matters: failures aren't missing-at-random. The long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks. His line: "an eval harness needs to distinguish wrong from absent, and any harness that can't will report infrastructure as intelligence."

That was already the design; his framing got it written down explicitly — exclude by class, not blanket. Code: github.com/egnaro9/model-drift

Writing the methodology in public is how it gets read this closely.