Cost per successful AI task: a practical way to compare model routes

When teams compare LLM providers or gateway routes, the first spreadsheet is usually token price.

That is useful, but it is rarely enough.

A cheaper request that needs retries, produces unusable output, or breaks a workflow can cost more than a higher-priced request that finishes the task once.

A more practical metric is cost per successful task.

Disclosure: I work with ModelRouter, an independent third-party OpenAI-compatible AI API gateway. It is not an official model-provider service. The checklist below is provider-neutral, and the ModelRouter link at the end is included as one possible place to run this kind of controlled route evaluation.

The metric

Use this formula:

cost per successful task = total spend / number of tasks that meet the acceptance criteria

The important part is the acceptance criteria.

A successful task is not just a 200 response. It should mean the output was usable for the workflow.

Examples:

• a support reply that can be sent with light editing
• a JSON extraction that passes schema validation
• a code suggestion that passes tests
• a summary that includes the required fields
• a workflow step that completes without a manual rerun

Step 1: Pick one real workflow

Do not start with a vague benchmark.

Pick one workflow your team already understands:

• support triage
• lead enrichment
• invoice extraction
• n8n automation step
• Dify agent task
• Open WebUI internal assistant
• code review helper
• sales email classification

The workflow should have enough examples to show failures, but it should be small enough to review manually.

Step 2: Define success before running the test

Write down what counts as success.

For example:

Task: classify inbound support tickets
Success: correct category, priority, and summary in valid JSON
Failure: wrong category, invalid JSON, missing priority, hallucinated customer facts, timeout, or manual rerun required

If the success definition is fuzzy, the result will be fuzzy too.

Step 3: Run a small representative set

For a first pass, 20 to 50 tasks is often enough to avoid obvious mistakes.

For a team or client workflow, 50 to 200 tasks gives a better signal.

Track these fields:

route
model
input type
task id
status code
latency
retry count
raw cost
usable output: yes/no
failure reason
notes

The failure reason is where the learning happens.

Step 4: Include retries and manual fixes

If route A costs less per request but needs more retries, include that.

If route B produces valid JSON more consistently, include that.

If route C is fast but needs manual review every time, include that too.

A simple scoring table can look like this:

route | tasks | successful | spend | retries | cost per successful task
A     | 50    | 41         | $3.20 | 9       | $0.078
B     | 50    | 47         | $4.10 | 2       | $0.087
C     | 50    | 32         | $2.40 | 18      | $0.075

The cheapest per successful task may still not be the best route if latency, reliability, or review cost matters. This metric is a starting point, not a final verdict.

Step 5: Keep the first call boring

Before comparing routes, make sure one minimal request works.

A smoke test should answer:

• is the base URL correct?
• is the API key accepted?
• is the model route valid?
• does usage/spend show up where expected?
• can the client library send one chat completion?

Only after that should you evaluate real tasks.

Step 6: Decide stop, continue, or expand

At the end of a small run, choose one of three decisions:

1. stop, because the route does not improve the workflow
2. continue, because the route is promising but needs a larger sample
3. expand, because the route is clearly useful for this workflow

This prevents accidental production migration based on one good demo.

A minimal evaluation checklist

[ ] workflow selected
[ ] success criteria written
[ ] 20-50 representative tasks collected
[ ] first API call verified
[ ] route/model names recorded
[ ] spend tracked
[ ] retries counted
[ ] failures categorized
[ ] cost per successful task calculated
[ ] stop/continue/expand decision written

Optional controlled pilot

If you want to run this through an OpenAI-compatible gateway, the same pattern applies: start with one first call, then one workflow, then representative tasks.

ModelRouter is one independent third-party OpenAI-compatible gateway where this kind of route evaluation can be tested:

https://modelrouter.site/business-pilot/?utm_source=devto&utm_medium=tutorial&utm_campaign=cost_per_successful_task

Pricing and trust notes:

https://modelrouter.site/pricing-and-trust/?utm_source=devto&utm_medium=tutorial&utm_campaign=cost_per_successful_task

The safest evaluation is still small and boring: make the first call work, measure real tasks, and decide based on successful outcomes rather than token price alone.