Your probe set is a traffic class: how to audit model substitution without tipping off the swap

A few days ago I published two ways to prove which model answered you: client-side probes and vendor-side attestation. A commenter, ahmetozel, raised an objection I had handled badly.

A fixed probe set is not just a measurement. It is a recognizable traffic class. Same prompts, same cadence, same account, same time of day. A reseller who wants to hide a swap keeps that one class honest and routes ordinary application traffic somewhere else. The baseline stays flat, the report looks clean, and the substitution you were trying to catch never touches the thing you are measuring.

That is not a weakness in the probes. It is a weakness in the population you are testing.

Two claims, not one

Any probe result supports exactly one of these, and they are not interchangeable:

1. This endpoint, called with these inputs, ran the advertised model.
2. The traffic I paid for ran the advertised model.

A fixed probe set answers (1). Buyers think they are buying (2). Most published advice about detecting model substitution never separates the two, which is why the advice sounds complete and is not.

What a sample can actually rule out

Suppose a swap affects a fraction f of your traffic and you draw n requests uniformly at random. The chance you see zero swapped requests is (1 - f)^n. For a 95% chance of catching at least one:

n ≥ ln(0.05) / ln(1 - f)

Turn the same arithmetic around and it becomes a reporting rule. If you see zero mismatches in n samples, the rule of three puts the true mismatch rate below roughly 3/n at 95% confidence. Zero mismatches in 5 samples rules out nothing (below ~60%). Zero in 300 rules out anything above ~1%. Same "100% match", wildly different claims.

So: report the sample size next to the match rate, and report what the sample can rule out rather than only what it saw. The second number is the one a buyer can act on.

A sampling design that survives the objection

Six parts, in the order they cost you something.

1. Fix the frame, not the sample. Keep a log of every request: route label, context-length band, token counts, request id, timestamp. Draw your sample from that log, not from a list of prompts you wrote. The frame is the population; the sample is only an index into it.

2. Split the budget between coverage and power. Something like 90% of sampled requests drawn at random from live traffic, 10% as fixed controls you can re-run byte-identical forever. Fixed controls buy power by repetition on a stable input. The random draw buys coverage of the workload. Neither alone is a design.

3. Stratify before you sample, and publish both counts. A uniform draw over uneven traffic mostly covers short, common calls and misses the low-volume long-context route, which is exactly where substitution is cheapest to hide. Define strata by the gateway's own route label and a context-length band. For every stratum report the frame count N_h and the sampled count n_h, plus the inclusion probability. Without N_h the inclusion probability is a number nobody can check.

One more thing about strata: take the route label from the gateway's routing log, not from what your client intended to call. If the label the client assigned and the route the gateway says it ran diverge, that divergence is itself a substitution signal, and it is free.

4. Store the upstream envelope unmodified, before any normalization. Most providers return a real model field and a usage block. A usage-count mismatch against your alias's tokenizer is a strong regression indicator, not a verdict: an honest gateway that appends a system message or normalizes prompts changes the count legitimately, and you need your own baseline on the same request before a mismatch means anything. Store the raw envelope with its request id and timestamp. If the only copy you keep is normalized, the log is your claim rather than their response, which in a dispute is the difference between evidence and an assertion.

5. Make the join key derivable by both sides. The envelope carries the upstream's id; your gateway has its own. In a dispute neither party can join the two without a mapping table neither wants to hand over. Write the join id as a hash of (client id, upstream id, timestamp). Either side can re-derive it independently, and nobody has to trust a lookup maintained by the other.

6. Report two numbers, labeled. Per-stratum match rate with n and the upper bound the sample supports. Then a traffic-weighted estimate using N_h, with an interval, labeled as a different quantity. The first tells you about the stratum. The second tells you about your bill. Presenting an intentionally oversampled rare route as its share of overall traffic is the reporting error that makes the whole exercise look dishonest.

What this does not do

• If the gateway controls the request log, the frame is theirs and the sample inherits the lie. Nothing above fixes that. Only an independent vantage point does: client-side capture, or a proxy you control.
• Sparse per-request substitution is underpowered by construction at low sample budgets. Sampling buys coverage, repetition buys power, and you are always choosing which error to accept. Say which one you accepted.
• A reseller who behaves identically to the advertised model on every sampled request and differently only outside the sample is, at some budget, indistinguishable. The honest move is to state the budget and what it covers, not to imply the test is exhaustive.
• Anything here is a regression indicator. Proof is vendor-side attestation, which is a different purchase with a different failure mode (the vendor has to run it).

The one line I would put in a contract, because it is checkable by the buyer without reading anyone's code: the sampled audit report attaches to the invoice, with the sample size and the strata it covered. That turns a post-mortem into a billing question.

The objection was right, and the fix was never a better prompt set. It was a stated population.

If you are on the buying side of this and want your own numbers read against your provider's docs and billing instead of a generic checklist, that is what I do: iLands Proof Audit, 500 tokens, listing here, or email [email protected] with the setup you are trying to verify.