We built a router to predict when a cheap model is enough. It does not work.

If you serve a model cascade, escalation is your cost dial. Not your model choice, not your prompt,
not your context window. The single number that moves your bill is what fraction of requests climb to
the expensive tier.

So the obvious thing to build is a router: look at the incoming request, predict whether the cheap
model will get it right, and only pay for the expensive one when the answer is no. We built that. It
does not work, and the reason turned out to be more interesting than a working router would have been.

Publishing it because a negative result with a mechanism is worth more than a positive result with a
story, and because the last two times we published numbers, readers found the flaws faster than we did.

What we built and what it scored

Our gateway already embeds every prompt, because the semantic cache needs a vector. Reusing that
vector to predict difficulty is free. That is exactly why we did it, and that turned out to be the
founding design flaw.

Trained on 539 real coding tasks labelled by actual execution (428 easy, 111 hard):

held-out AUC                       0.594
5-fold CV                          0.55 to 0.57   (one fold below chance)
best threshold                     scores exactly what "never escalate" scores

A classifier that cannot beat a constant policy is not a weak classifier. It is not a classifier.

It is a feature limit, not a tuning problem, and we made it prove that

The tempting read is underfitting or overfitting, so we swept L2 regularisation across four orders of
magnitude, 0.01 to 100, with cross-validation at every step. Train AUC stayed at 0.94 to 0.98 while
test AUC stayed at 0.58 to 0.59, at every single setting. The gap never closed. A gap that survives
four orders of magnitude of regularisation is telling you the features do not contain the signal.

Then the result that actually explains it. We threw together 11 crude surface features, things like
length and token counts, no semantics at all:

11 crude surface features          AUC 0.610
1024-dimension prompt embedding    AUC 0.552
both together                      AUC 0.609   (the embedding adds nothing)

Eleven numbers you could compute with a ruler beat a 1024-dimension semantic embedding. The reason is
simple once you see it: the cache vector encodes topic, not difficulty. It is built to answer "have
I seen a question like this before", and it is good at that. Two questions about sorting a list sit
close together in that space whether one is trivial and the other is subtle. We reused it because it
was free, and free was the whole problem.

Two measurement traps, which are worth more than the router

One. Scoring a cost-saving router on accuracy alone marks it failed by construction. A router that
saves money by sending easy work to a cheap model will, correctly, be slightly less accurate than
always escalating. Our first yardstick printed DO-NOT-SHIP on a router that actually had skill. The
control you need is random routing at the same cost, not the expensive model. If you cannot beat
a coin flip that spends what you spend, you have nothing. If you can, you have something, even if your
raw accuracy went down.

Two. The ceiling is not "always escalate". We assumed the expensive model was the upper bound and
a perfect router would approach it. It is not. On our 539 tasks, escalating everything to the top tier
rescues 39 answers the cheap model got wrong and breaks 23 it had already got right. Net +16.

always escalate      82.4%
a PERFECT router     86.6%

A perfect router beats always-escalate by 4.2 points, because it also knows when not to escalate.
That headroom is real and no gate-local heuristic captures it. If you are benchmarking a cascade
against "just use the big model", you are measuring against the wrong ceiling.

What is still alive

Showing the router the cheap model's draft rather than only the prompt does better: AUC 0.640, and
81.8% accuracy at 36% escalation against 80.5% for an equal-cost random control. That is the right
shape. But permutation testing gives p=0.0375 unadjusted and roughly p=0.30 after Bonferroni, so
the honest verdict is promising, not proven. We are not going to claim it until it survives a
pre-registered run.

Which makes sense mechanically: the draft carries evidence of difficulty that the question alone does
not. A model that is about to be wrong often looks different while being wrong.

The questions, and this is why I am posting

I would rather hear from people who have hit this than keep guessing.

1. What features actually carry difficulty before generation? We have shown that a topic embedding
   does not, and that crude surface stats beat it. That is a low bar. What clears it?

2. Is pre-generation difficulty prediction possible at all, or is the draft the earliest honest
   signal? It is entirely plausible that "will this model fail" is not a property of the question,
   only of the interaction. If someone has evidence either way I would like to see it.

3. If you run a cascade in production, what do you actually route on? I suspect the honest answer
   for most teams is a hand-written rule about request type, and I suspect those rules do better than
   our classifier did. That would itself be a finding.

Numbers, labels and the negative result are ours to share. If you want the setup in more detail, ask
and I will write it up.