Choosing the right AI model for your app: measure task success, then compare cost

Disclaimer: The evaluation approach in this post applies to any programming language or application stack. The code examples use C# and .NET because that's what I'm most familiar with.

Say you want to add an AI chat to a website. Users should be able to ask questions about their documents, look up information, and generate downloadable files.

The model with the lowest token price is an obvious place to start. Until it tells a user that a report is ready, and the report contains the wrong numbers.

Consider this request:

Find my Q1 expenses in connected storage. Create a downloadable JSON report with one row per month, in USD, and a total. Keep your reply to one sentence and include the download link.

Two models return exactly the same sentence:

Your Q1 expense report is ready: [download](./report.json).

Both files open. Both contain the right monthly amounts. One has a total of USD 360. The other has a total of USD 370.

Comparing the chat messages would give these models a tie. Inspecting the files gives us a failure we can actually act on.

That's the starting point for choosing a model: define what a completed task looks like, measure how often the application achieves it, and compare the cost of doing so.

Use public benchmarks to build a shortlist

Public benchmarks are useful when deciding which models deserve a trial. They're less useful as the final acceptance test for a feature.

A general knowledge score doesn't tell me whether a model can select the right storage tool, recover from an expired link, and produce a correct spreadsheet through my application's tool loop.

Stanford's HELM research takes a multi-scenario, multi-metric approach to model evaluation. That's a useful principle to borrow: look at several capabilities and their trade-offs, rather than treating one score as a complete description of a model.

For this website, I'd shortlist a few candidates with the capabilities I need:

• Tool calling supported by the provider and SDK I use.
• Enough context for realistic conversations and retrieved documents.
• The required input modalities, if users upload images or scanned PDFs.
• A deployment option that meets the project's operational requirements.
• Plausible pricing and latency for the expected traffic.

Then I'd run those candidates through the website's actual chat workflow.

The thing under evaluation is:

model + instructions + tool definitions + retrieval + application code

A model doesn't inherently have access to my users' storage. My application provides that access through tools. Likewise, file generation might mean calling a deterministic report builder, executing code in a sandbox, or returning structured data that my backend renders into a file. Those are different implementations, with different failure modes and costs.

Anthropic's guide to agent evaluations makes this distinction explicit: an agent evaluation measures the model and its surrounding harness working together.

Turn feature requirements into cases you can grade

For the expense example, the fixture contains three monthly amounts: January USD 120, February USD 80, and March USD 160. The expected total is USD 360.

I would define success before running any candidate:

1. The application reads the correct user's expense document.
2. A downloadable file exists and the current user can retrieve it.
3. The JSON contains exactly the expected months and amounts, the correct currency, and the correct total.
4. The reply contains one sentence and a working link to that artifact.
5. The workflow finishes within the agreed time and tool-call budget.

The expected amounts belong in the evaluation fixture. They should not be included in the candidate's prompt as an answer key.

Now expand that into a small dataset:

Testing when a tool should not run matters too. A model that searches the web for every question can score well on research tasks while adding unnecessary latency and cost.

I'd start with roughly 20 to 50 carefully reviewed cases, drawn from the feature's requirements and things I already check manually. That's a starting point for finding substantial problems, not proof of production-level reliability. Anthropic recommends a similar initial scale.

Keep two groups as the dataset grows:

• Development cases: inspect these while adjusting prompts, tools, and implementation.
• Held-out cases: use these for the final comparison. Don't repeatedly tune against them.

Include realistic languages, typos, document sizes, and conversation histories. An evaluation consisting entirely of short English prompts won't tell me much about a multilingual site with long-running chats.

Compare outcomes first, wording second

There are three useful grading methods. I would combine them rather than ask one metric to do every job.

1. Use code for facts you can check

For a file-generation task, download the actual artifact through the test user's application path and inspect its contents. A message saying "I've created the file" isn't evidence that the file exists.

Here's a complete console example using only the .NET libraries. Paste it into Program.cs in a .NET 8 or later console project and run it. The two byte arrays stand in for downloaded artifacts so that the example needs no model credentials.

using System.Text;
using System.Text.Json;

var correctFile = Encoding.UTF8.GetBytes("""
    {
      "currency": "USD",
      "rows": [
        { "month": "2026-01", "amount": 120 },
        { "month": "2026-02", "amount": 80 },
        { "month": "2026-03", "amount": 160 }
      ],
      "total": 360
    }
    """);

var wrongFile = Encoding.UTF8.GetBytes(
    Encoding.UTF8.GetString(correctFile)
        .Replace("\"total\": 360", "\"total\": 370"));

foreach (var (name, bytes) in new[]
{
    ("Candidate A", correctFile),
    ("Candidate B", wrongFile)
})
{
    var failures = GradeReport(bytes);
    var verdict = failures.Count == 0
        ? "PASS"
        : $"FAIL: {string.Join("; ", failures)}";

    Console.WriteLine($"{name}: {verdict}");
}

static List<string> GradeReport(byte[]? downloadedFile)
{
    if (downloadedFile is null || downloadedFile.Length == 0)
        return ["No downloadable artifact"];

    ExpenseReport? report;

    try
    {
        report = JsonSerializer.Deserialize<ExpenseReport>(
            downloadedFile,
            new JsonSerializerOptions(JsonSerializerDefaults.Web));
    }
    catch (JsonException)
    {
        return ["Artifact is not a valid expense report"];
    }

    if (report is null)
        return ["Artifact contains null"];

    ExpenseRow[] expectedRows =
    [
        new("2026-01", 120m),
        new("2026-02", 80m),
        new("2026-03", 160m)
    ];

    var failures = new List<string>();

    if (report.Currency != "USD")
        failures.Add("Wrong or missing currency");

    if (report.Rows is null || report.Rows.Any(row => row is null))
    {
        failures.Add("Missing or invalid rows");
    }
    else if (!report.Rows
        .OrderBy(row => row!.Month, StringComparer.Ordinal)
        .SequenceEqual(expectedRows))
    {
        failures.Add("Months or amounts differ from the fixture");
    }

    if (report.Total != expectedRows.Sum(row => row.Amount))
        failures.Add("Total differs from the fixture");

    return failures;
}

record ExpenseRow(string? Month, decimal? Amount);
record ExpenseReport(
    string? Currency,
    ExpenseRow?[]? Rows,
    decimal? Total);

Output:

Candidate A: PASS
Candidate B: FAIL: Total differs from the fixture

This grader accepts a different row order. It rejects missing months, duplicate rows, incorrect amounts, and a missing total. Nullable amounts prevent a missing numeric field from silently becoming zero.

It's deliberately a content grader. The surrounding runner still needs to check the download, permissions, response format, and tool trace. Unknown JSON properties aren't rejected here; if your contract forbids them, enforce that separately with schema validation.

For CSV, use a CSV parser. For an Excel workbook, inspect the workbook's cells and formulas. For a PDF, checking that its bytes start with a PDF header only proves you've received something resembling a PDF, not that the report is correct.

Tool traces are useful for checking selected tools, arguments, failed calls, and unnecessary calls. Prefer final-state checks where several legitimate tool sequences could achieve the same result. Don't require one exact sequence just because that was the first implementation you imagined.

2. Use a rubric for answers with several valid forms

Suppose the user asks:

Summarize our refund policy in three bullets and explain whether a purchase from 40 days ago qualifies.

Exact string matching would reject perfectly good paraphrases. Text-similarity metrics such as BLEU or ROUGE can be useful for specific tasks, but they aren't sufficient acceptance criteria for this one. An answer can reuse most of a reference's words and still reverse the eligibility decision.

I would check the bullet count in code, then grade factual claims against the retrieved policy:

A model-based judge can apply that rubric repeatedly. Give it the user request, relevant instructions, retrieved evidence, candidate answer, and the rubric. Ask for a structured verdict per criterion, with a short explanation pointing to the supporting evidence. Allow an unknown verdict when the evidence isn't sufficient, and send that result to review.

The judge should not simply grade whether an answer "sounds good." For retrieval, distinguish two questions:

• Did retrieval find the evidence needed to answer the question?
• Does the answer accurately use that evidence?

A grounded answer can still be incomplete if the application retrieved the wrong documents. A fluent answer can still be unsupported.

3. Calibrate automated grades with human review

An LLM judge is another fallible component.

The MT-Bench research documents position bias, verbosity bias, self-enhancement bias, and limitations in reasoning. Its results support using strong model judges in the studied settings, not assuming that any judge will agree with your users on any task.

Before trusting one, have people familiar with the domain label a sample. Compare their labels with the judge's labels and investigate disagreements, especially cases where the judge passes an answer that people reject.

For side-by-side comparison:

1. Hide model names from reviewers and the judge.
2. Use the same request and evidence for both candidates.
3. Randomize answer order. For close comparisons, repeat with the order reversed.
4. Allow a tie and an insufficient-evidence verdict.
5. Keep the judge model and rubric version fixed during the experiment.

Treat candidate answers and retrieved documents as untrusted content in the judging step too. Neither should be allowed to redefine the rubric.

I'd use code to establish that a report is correct, a rubric to assess whether its explanation is useful, and human review to check that the rubric measures what we intended.

Run a fair experiment

For an initial comparison, keep the application, prompts, tools, fixtures, and budgets fixed. Change the model configuration.

Record the resolved model version where the provider exposes it, plus provider, application commit, prompt version, tool-schema version, dataset version, and settings. Include reasoning effort and output limits where applicable. Some models don't support the same sampling controls, so document differences rather than inventing equivalent settings.

Run each case several times with fresh state. Five trials per case is a reasonable starting experiment, not a universal sample-size recommendation. Temperature zero doesn't make an entire tool-using workflow deterministic.

Interleave candidate runs to reduce timing effects. Reset storage fixtures, conversation history, generated artifacts, and application caches that could leak answers between trials. If you're measuring variation, don't let an evaluation response cache replay the same answer five times.

After the first comparison, model-specific prompt tuning may be worthwhile. Give candidates a comparable tuning budget on the development cases, then evaluate the resulting configurations on held-out cases. Label that comparison as optimized configurations, rather than a pure model swap.

Measure success per complete user task. Keep partial scores for debugging, but don't let excellent tone compensate for incorrect financial data.

Repeated attempts also need careful interpretation. "At least one of five attempts passed" is not the same as "a user succeeds on the first attempt." Only credit retries in your product result if the application actually detects failure and performs those retries. Include their cost and latency.

For close results, show the underlying counts and uncertainty. A difference of one pass in a small dataset isn't a convincing ranking. Because the candidates run the same cases, compare paired outcomes; if you use bootstrap confidence intervals, resample at the case level so repeated trials of one case aren't treated as unrelated tasks.

Compare cost per successful task

Token prices are inputs to the calculation. They're not the result.

For each complete workflow, include:

• Input and output usage across all model calls, including tool-result messages.
• Cache read/write and reasoning-token charges where applicable, following the provider's billing rules without double-counting.
• Search, storage, sandbox, and other metered tool charges.
• Retries and fallback calls.

Use actual provider usage where available and a dated price schedule. Keep evaluation-judge spending separate from the user workflow's cost. If you judge production responses online, account for that as an operational cost too.

For a representative workload:

cost per successful task = total workflow spend / successful tasks

Here's an illustrative report. These are invented numbers, not measurements of named models or current provider prices. Each candidate ran the same 1,000 trials, with any configured retries included within each trial.

If I decided before the experiment that I need at least 95% observed task success and p95 completion under 10 seconds, Medium is the lowest-cost candidate that meets both requirements in this example. I'd still inspect uncertainty and failures before making the deployment decision.

Small remains cheapest per successful task, but doesn't meet the quality requirement. Large has the highest observed success rate, but exceeds the latency limit. That's why choosing a model starts with requirements, rather than a weighted score invented after seeing the results.

Break results down by task type as well. A good overall score can hide poor export reliability behind a large number of easy FAQ answers. Keep critical failures separate from the average, and treat zero observed failures as evidence from a sample, not proof they can never occur.

If the dataset deliberately overrepresents difficult cases, use production task frequencies to estimate production cost and success. Report the difficult-case results separately.

For streaming chat, measure time to first token and time to usable completion. A fast opening sentence doesn't help much if the file takes another minute to arrive. Report timeout rates alongside latency so that excluding failed requests doesn't make a candidate look faster.

What can I automate?

Start with a runner around the existing workflow

A script, command-line app, or separate evaluation project in your existing stack is enough to begin. The runner should:

1. Load a case and reset its isolated environment.
2. Call the same chat service and tool loop the website uses, with the candidate configuration.
3. Continue through the full interaction, including tool results and any scripted follow-up turns.
4. Download artifacts as the fixture's user and capture final application state.
5. Run deterministic graders, then model judges where needed.
6. Store one result per case, candidate, and trial, including failures, usage, latency, and a trace reference.
7. Aggregate results and exit unsuccessfully when a declared release gate is violated.

The important integration point is step two. Calling a raw model endpoint with the initial user message would skip much of the behavior we're trying to evaluate.

Use scripted user turns for known conversations. A simulated user model can expand coverage of less predictable interactions, but version and review that simulator too. Otherwise, a change in the simulated user can look like a change in your chat's quality.

Separate provider or runner errors from grading failures in the report. Keep real workflow failures in the product reliability numbers. If the evaluation infrastructure itself breaks, mark the comparison incomplete rather than quietly deleting the affected cases.

For .NET: use the evaluation libraries when you need more infrastructure

Microsoft.Extensions.AI.Evaluation provides .NET evaluation abstractions and integrates with existing test frameworks.

The relevant pieces include:

• Microsoft.Extensions.AI.Evaluation.Quality for model-based evaluators such as relevance, completeness, groundedness, and tool-call accuracy.
• Custom evaluators through IEvaluator for application-specific checks.
• Microsoft.Extensions.AI.Evaluation.Reporting for stored results, caching, and reports.
• Microsoft.Extensions.AI.Evaluation.Console for report generation through dotnet aieval.

Microsoft's quickstart shows the API and required packages. I'd reuse that plumbing while retaining my own artifact and state checks. A built-in quality evaluator can't independently establish that my download endpoint returned the correct user's file.

Use an external runner or platform if it fits the workflow

Promptfoo's HTTP provider can call an application endpoint, and its webhook provider can trigger a complete application flow. That lets you evaluate a backend written in any language that can expose the required HTTP endpoint. You still need to expose the right evaluation adapter and return the evidence your graders require.

LangSmith supports datasets, offline experiments, comparison, and online evaluation. Langfuse also supports datasets, experiments, human annotations, and automated scoring. These can help when a team needs shared trace review and experiment history. Check their current SDK or API integration for your application stack before adopting one.

The framework can run cases and display scores. Your team still has to decide what constitutes a correct report, a supported answer, or an acceptable failure response.

Keep the comparison useful after deployment

I'd run a small regression set on relevant changes, and a broader repeated evaluation before switching models or changing the tool loop. Nightly or scheduled runs can cover the expensive matrix without making every pull request wait for it.

Keep the results and traces as CI artifacts. A failure should be explainable as something like:

Case: export-storage-timeout
Expected: explain that the source could not be read; no success claim
Observed: "Your report is ready", followed by a broken link
Tool trace: read_expenses timed out; no artifact was created

After release, sample real interactions, collect user feedback, and add reviewed failures to the dataset. Use sanitized fixtures and restrict access to traces containing private documents. Offline results won't cover every change in live web content, storage behavior, or user traffic.

Routing simple tasks to a smaller model and difficult tasks to a larger one can reduce cost. Evaluate that as a complete configuration, including routing errors and fallback overhead. "Use a better model when the first one is wrong" only works if the application has a reliable way to detect that failure. A confident-sounding reply isn't such a detector.

Where I'd start

For the website in this example, my first pass would be:

1. Write 30 cases covering document answers, file exports, web lookups, and failure responses.
2. Define expected artifacts, facts, and behavior for each case.
3. Run three candidate models through the real application, five trials per case.
4. Grade checkable outcomes in code, and use a human-calibrated rubric for the remaining answer quality.
5. Review failures, then compare task success, full workflow cost, and latency against requirements set in advance.

That gives me a defensible answer to "which model should I use?"

It also tells me when changing the model isn't the right fix. If every candidate fails because retrieval returns the wrong document, spending more on the model isn't addressing the problem.

Sources

Research and documentation checked on October 29, 2026. The examples, thresholds, and cost table in this post are illustrative.

• Stanford CRFM: Holistic Evaluation of Language Models, published in TMLR, 2023. Multi-scenario evaluation and explicit trade-offs between metrics.
• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks track. Judge agreement and documented biases.
• Anthropic Engineering: Demystifying evals for AI agents. Outcome checks, trial isolation, repeated runs, grading, and human calibration.
• OpenAI: Evaluation best practices. Task-specific datasets, deterministic and model-based grading, and continuous evaluation.
• Microsoft Learn: The Microsoft.Extensions.AI.Evaluation libraries and response-quality quickstart. .NET evaluators and reporting.
• Promptfoo: HTTP provider and webhook provider. Evaluating an existing application through an endpoint.
• LangSmith evaluation documentation and Langfuse evaluation documentation. Dataset, experiment, and review capabilities.