85 Perfect Scores for Answers That Never Reached the Disk

We ran the first full evaluation of the eight AI roles that help me run my company: 192 questions, zero channel failures, and a tidy score table ranging from 70.8 to 97.2. Then an independent review read the exported results back — and found that 118 of the 192 answer fields were empty. Eighty-five of those empty answers had been scored at full marks. We voided the entire score table the same morning. A score whose answers cannot be read back is not a measurement. It is a rumor with a decimal point.

The run itself was real. That is what makes this story worth telling. Eight roles — chief executive, chief of staff, technology, product, operations, finance, marketing, and legal counsel — each answered 24 questions, 53 of them red-line questions where a wrong answer is not a style miss but a safety failure. A judge from a different model family graded every answer against a rubric. No call failed. By every dashboard convention in our industry, this was a clean, successful evaluation, and the scores looked like something you would put in a slide deck.

Nobody had read the answers. That was the problem.

A clean scoreboard, an empty evidence locker

The failure was not in the run. It was in the export — the step that moves results out of the machine that produced them and onto disk, where other people can check them. When our quality team re-read the exported records question by question, 118 of the 192 answer fields were empty. Not weak answers. Not truncated answers. Empty. And the scoring column next to them was serenely complete: 85 of those 118 records carried full marks.

Sit with that for a second, because it is the purest form of a failure our whole industry is drifting toward. Every layer of the pipeline reported success. The questions loaded. The model answered — in the container where the run executed. The judge scored. The summary table aggregated. The only layer that failed was the one nobody watches: the handoff into persistent, checkable evidence. What reached the disk could not support the conclusion printed above it. The scores may even have been roughly right about the run that happened. It does not matter. Evidence you cannot produce is not evidence, and a baseline nobody can re-read is just a number the system once believed about itself.

Score-first is how evals become theater

The standard response to a story like this is: "the export is a formatting issue; fix the exporter and keep the scores." We did the opposite, and I think the opposite is the only defensible move. The full table — all eight roles, the whole 70.8-to-97.2 range — was withdrawn as an official baseline within hours of being reported, before any outside audience ever saw it as fact, with the correction recorded in the same ledger that carried the original claim. Not because we proved the scores wrong. Because we could not prove them right. Those are different statements, and conflating them is exactly how evaluation theater starts: a number survives because nobody can be bothered to kill it, then roadmaps, budgets, and model choices get built on top of it.

The popular faith is that running an eval is the hard part and reporting is clerical. My experience now runs the other way. Generating answers is cheap and getting cheaper. The scarce, expensive discipline is the chain of custody after generation: every answer persisted, every score traceable to the text it graded, every export audited for completeness before anyone is allowed to quote the aggregate. Skip that, and you have not built an evaluation. You have built a machine for manufacturing confidence — the same failure, incidentally, as a memory store where every record arrives pre-labeled as certain. Different subsystem, identical disease: a field filled in by the pipeline, treated downstream as a measurement.

The fix is a boring gate, on purpose

So the rule we wrote is deliberately unglamorous, and it now sits in front of everything else: before any evaluation result can be reviewed, signed, or used to approve a change, someone re-reads the raw records and produces an on-disk audit — every question, answer present or answer missing, no sampling, no exceptions. The required rate is 100%. One empty answer does not dent the batch; it rejects it. Scores attached to missing answers are not "provisional." They are void. Remediation runs in a fixed order, too: first check whether the original records inside the execution environment are intact — if they are, re-export and audit; if they are not, fix the persistence path and re-run exactly the missing questions. Only then does the independent review — a sampled re-grade plus a full pass over every red-line question — even begin.

None of this will trend on social media. There is no demo in it. But if you operate agents whose scores decide what ships, steal the audit before you steal anyone's benchmark: count your exported answers before you quote your scores. If the count is not all of them, you do not have a baseline. You have a rumor — and the most dangerous rumors in an automated company are the ones with decimal points.

— Jiahui Miao, founder of vertciti, building a personal AI operating system for one.