How to evaluate an AI router's 'none of these' branch

An AI router has to recognize requests outside its supported categories while still accepting useful work. Those are two different things to measure.

Between 23 and 28 October 2026, I compared four decision models and five small language models. This article focuses on one task from that comparison: choosing a domain for a request, including an explicit "out of scope" option.

The decision models received typed questions and returned probabilities over the options. The language models received the same questions and guidance as text, with their answers constrained to a JSON schema. No models were fine-tuned.

Give the router somewhere to decline

I used 1,000 sampled requests from the CLINC150 test split, mapped to ten domains, with an eleventh option for requests outside those domains.

Of those requests:

• 190 were genuinely out of scope.
• 810 belonged to a supported domain.

Every model received the same guidance for the out-of-scope option. The resulting behavior varied substantially.

Here are three examples from the nine-model comparison:

The false-alarm counts come from subtracting correct rejections from total out-of-scope answers. Mammoth-1 selected that option 252 times: 152 correct rejections and 100 false alarms.

Kev-4B selected it 375 times, yet caught fewer genuinely out-of-scope requests. GPT-5.4 nano rejected only 18 valid requests, but missed 144 requests it should have declined.

A rejection count alone hides both kinds of error.

Score the branch separately

For a router with an out-of-scope option, I would report three measures alongside overall accuracy:

1. Out-of-scope recall: correctly declined out-of-scope requests divided by all truly out-of-scope requests.
2. False-rejection rate: in-scope requests wrongly declined divided by all truly in-scope requests.
3. In-scope routing accuracy: requests assigned to their correct domain divided by all truly in-scope requests, including those wrongly declined.

In this sample, the first denominator is 190 and the other two are 810.

For Mammoth-1, out-of-scope recall is 152/190, or 80%. Its false-rejection rate is 100/810. Its reported in-scope routing accuracy is 83.7%.

Keeping domain accuracy separate matters. A router can accept a valid request and still send it to the wrong destination.

I would also track malformed answers and request failures separately. In my benchmark, format failures counted as wrong in overall accuracy; they were not dropped to make the successful responses look better.

Apply the test to your own router

I would start by writing down exactly which requests each destination supports. Then I would build a labeled evaluation set containing both supported requests and plausible requests outside that scope.

For an initial comparison, keep the requests, category definitions and option guidance consistent across models. Record the model versions and settings so that a later run has a useful reference point.

After scoring, inspect examples from each error category. A false rejection may point to narrow option guidance. A missed out-of-scope request may expose an unclear boundary between supported and unsupported work. These are possibilities to investigate in the examples.

The choice depends on what those mistakes cost in the application. Sending unsupported work into a workflow and declining a valid customer request can have very different consequences.

If I added a confidence threshold, I would evaluate it on separate validation data and check calibration. The counts above describe the options the models selected; they do not establish an optimal threshold.

Inspect the experiment

The public repository contains the harness, task definitions, raw requests and responses, and reports.

Start with the saved results and reports to examine the existing experiment. The repository's README explains how to run it again.

These results describe the tested model versions on public data in October 2026. Model-specific prompting or fine-tuning could change the outcome. All nine models ran this CLINC task, and a production router still needs evaluation on its own traffic.

The full comparison on my site also covers calibration, format failures, latency, cost and repeatability across the other tasks.