Dev.to AI ๐Ÿค– Ai ๐Ÿ‘ 0 ๐Ÿ“– 16 min read

Ranking 60 LLM APIs by Quality per Dollar: What a Value Score Can and Cannot Tell You

Dividing a benchmark score by a blended token price gives you a useful daily shortlist, as long as you treat it as a cost filter, keep guessed scores out of the picks and never mistake it for a model selection strategy.

Dividing a benchmark score by a blended token price gives you a useful daily shortlist, as long as you treat it as a cost filter, keep guessed scores out of the picks and never mistake it for a model selection strategy.

The Real Problem / Why Naive Approaches Fail

Model pricing pages change constantly. A provider cuts output prices on a Tuesday, a new open-weights model appears on three hosts at three different prices by Thursday and the "cheap model" your team settled on last quarter is now neither the cheapest nor the best in its price band. Nobody on a small team has time to recheck sixty pricing pages every morning.

That is the problem I built the ranking job in cloud-finops-rag to solve. Every day it pulls current prices from OpenRouter for 60 models across 8 providers. It blends each model's price as 75% input and 25% output. Then it divides the model's MMLU score by that blended price and produces a value ranking. Models whose MMLU score is an estimate rather than a published number stay out of the daily picks.

The naive version of this idea fails in predictable ways and most of them come from treating "quality per dollar" as a single objective truth.

The first failure is the division itself. MMLU scores for current production models cluster in a fairly narrow band, while prices span two or three orders of magnitude. When you divide a number that varies by maybe 1.3x by a number that varies by 100x or more, the result is mostly a price ranking with a small quality adjustment. The cheapest acceptable model nearly always wins.

The second failure is trusting every number in the numerator equally. Some models have published, reproducible MMLU results. Others have numbers someone estimated from a model card, a parameter count or a sibling model. A cheap model with an optimistic estimated score will sit at the top of a value ranking and that is the worst possible place for a guess to land.

The third failure is the assumption that a per-token price is comparable across providers. It isn't, quite. Tokenizers differ, reasoning models emit hidden tokens you pay for and cached input is often billed at a discount that a list price ignores.

None of this makes the score useless. It makes it a specific tool with a specific job. Getting the job description right is most of the work.

Architecture & Inner Mechanics

The pipeline is small on purpose. One HTTP call, one join, one division, one filter.

sequenceDiagram
    participant Cron as Daily scheduler
    participant Job as Ranking job
    participant OR as OpenRouter /api/v1/models
    participant Scores as Benchmark score file
    participant Store as Snapshot store (JSONL + DuckDB)
    participant Out as Daily picks report

    Cron->>Job: trigger once per day
    Job->>OR: GET model list with pricing
    OR->>Job: per-token prompt and completion prices
    Job->>Job: normalise to USD per 1M tokens, drop invalid prices
    Job->>Scores: load MMLU score plus estimated flag per model
    Job->>Job: blended = 0.75 * input + 0.25 * output
    Job->>Job: value = MMLU / blended
    Job->>Store: append full ranked snapshot (estimated rows included)
    Job->>Job: filter out estimated scores and unpriced models
    Job->>Out: publish top picks with price deltas vs yesterday

Why OpenRouter as the price source

OpenRouter exposes a model listing endpoint that returns, for each model, a pricing object with prompt and completion costs per token, plus metadata such as context length. That gives one normalized source instead of eight scraped pricing pages with eight HTML layouts. The tradeoff is that you are ranking OpenRouter's prices, which include its routing to underlying hosts. For most providers those track the provider's own list price closely, but if you buy directly from a provider under a negotiated or committed-use rate, your real price differs from what this job sees.

Why 75% input and 25% output

A blended price needs a weighting and any weighting encodes an assumption about workload shape. The project is RAG-oriented. RAG requests are input-heavy because you stuff retrieved chunks into the prompt and ask for a comparatively short answer. Consider a typical request shape of 3,000 input tokens and 500 output tokens. That is a 6:1 ratio, which works out to roughly 86% input by token count. A 75/25 blend is a deliberate round number that leans input-heavy without assuming every request is a pure retrieval lookup.

The weighting matters more than people expect, because output tokens are usually priced several times higher than input tokens. Take a hypothetical model priced at $1 per million input tokens and $4 per million output:

| Blend (input/output) | Blended price per 1M tokens |
|, -|, -|
| 90 / 10 | $1.30 |
| 75 / 25 | $1.75 |
| 50 / 50 | $2.50 |
| 25 / 75 | $3.25 |

Same model, a 2.5x swing in "price" depending on an assumption. If your workload is generation-heavy (long-form drafting, code generation, agents that write a lot), a 75/25 blend will make output-expensive models look cheaper than they are for you. The blend is a parameter, not a fact.

Why MMLU

MMLU (Hendrycks et al.) covers 57 subjects of multiple-choice questions and has been reported for nearly every model released in the last few years. That coverage is the reason to use it. You need a quality number that exists for as many of the 60 models as possible and MMLU has the widest coverage of any single benchmark.

Coverage is not the same as relevance and I will get to that. But for a daily ranking across many providers, a benchmark that half the models lack is worse than a broadly reported one with known flaws.

The estimated-score rule

Every score in the benchmark file carries a flag: published or estimated. Estimated rows are still ranked and stored, so you can see where they would land. They are excluded from the daily picks.

The reasoning is about where error does damage. In a ratio, an error in the numerator propagates directly into the score. If a model is estimated at 80 when its true score would be 72, its value score is inflated by about 11%. That alone might not matter. What matters is that the models most likely to have estimated scores are often new, cheap or niche and cheap models already dominate the ranking because of the denominator. Put an optimistic guess on a cheap model and you get a confident-looking top pick built on nothing. The picks are meant to be something a person can act on without rechecking the sources, so they only include numbers someone actually measured.

Production Implementation

The job below is a cleaned-up version of the pattern. It fetches prices with retries, validates them, joins against a score file, computes value, writes a snapshot and produces picks. It also computes a Pareto frontier, which I find more honest than a single ratio (more on that later).

The score file is plain YAML. Keeping it in version control means every score change has a diff and an author.

# scores/mmlu.yaml
# source: where the number came from; estimated: true if not a published result
models:
  "provider-a/model-large":
    mmlu: 88.0
    source: "vendor technical report"
    estimated: false
  "provider-b/model-small":
    mmlu: 79.5
    source: "inferred from sibling model, no published result"
    estimated: true
"""Daily LLM value ranking: MMLU per blended USD price, sourced from OpenRouter."""

from __future__ import annotations

import datetime as dt
import json
import logging
import math
import sys
import time
from dataclasses import asdict, dataclass
from pathlib import Path

import httpx
import yaml

OPENROUTER_MODELS_URL = "https://openrouter.ai/api/v1/models"
INPUT_WEIGHT = 0.75
OUTPUT_WEIGHT = 0.25
TOP_N = 10
SNAPSHOT_DIR = Path("data/snapshots")
SCORES_PATH = Path("scores/mmlu.yaml")

log = logging.getLogger("value_rank")


@dataclass(frozen=True)
class Price:
    model_id: str
    input_per_m: float
    output_per_m: float
    context_length: int | None

    @property
    def blended_per_m(self) -> float:
        return INPUT_WEIGHT * self.input_per_m + OUTPUT_WEIGHT * self.output_per_m


@dataclass(frozen=True)
class Ranked:
    model_id: str
    mmlu: float
    estimated: bool
    input_per_m: float
    output_per_m: float
    blended_per_m: float
    value: float
    context_length: int | None


def fetch_models(client: httpx.Client, attempts: int = 4) -> list[dict]:
    """GET the model list with exponential backoff on transient failures."""
    delay = 2.0
    for attempt in range(1, attempts + 1):
        try:
            resp = client.get(OPENROUTER_MODELS_URL, timeout=30.0)
            if resp.status_code in (429, 500, 502, 503, 504):
                raise httpx.HTTPStatusError(
                    f"retryable status {resp.status_code}",
                    request=resp.request,
                    response=resp,
                )
            resp.raise_for_status()
            data = resp.json().get("data")
            if not isinstance(data, list) or not data:
                raise ValueError("model list missing or empty")
            return data
        except (httpx.TransportError, httpx.HTTPStatusError, ValueError) as exc:
            if attempt == attempts:
                raise
            log.warning("fetch attempt %d failed: %s, retrying in %.0fs", attempt, exc, delay)
            time.sleep(delay)
            delay *= 2
    raise RuntimeError("unreachable")


def parse_price(raw: dict) -> Price | None:
    """Convert per-token string prices to USD per 1M tokens. Reject anything odd."""
    pricing = raw.get("pricing") or {}
    try:
        inp = float(pricing["prompt"]) * 1_000_000
        out = float(pricing["completion"]) * 1_000_000
    except (KeyError, TypeError, ValueError):
        return None
    # Negative values signal variable or router pricing; zero means a free tier.
    # Neither belongs in a per-dollar ranking.
    if not (math.isfinite(inp) and math.isfinite(out)) or inp <= 0 or out <= 0:
        return None
    return Price(
        model_id=raw["id"],
        input_per_m=inp,
        output_per_m=out,
        context_length=raw.get("context_length"),
    )


def load_scores(path: Path) -> dict[str, dict]:
    doc = yaml.safe_load(path.read_text())
    scores = doc.get("models", {})
    for model_id, entry in scores.items():
        if not 0 < float(entry["mmlu"]) <= 100:
            raise ValueError(f"implausible MMLU for {model_id}: {entry['mmlu']}")
        if "estimated" not in entry:
            raise ValueError(f"{model_id} missing explicit estimated flag")
    return scores


def rank(prices: dict[str, Price], scores: dict[str, dict]) -> list[Ranked]:
    rows: list[Ranked] = []
    for model_id, s in scores.items():
        p = prices.get(model_id)
        if p is None:
            log.info("no valid price today for %s", model_id)
            continue
        blended = p.blended_per_m
        rows.append(
            Ranked(
                model_id=model_id,
                mmlu=float(s["mmlu"]),
                estimated=bool(s["estimated"]),
                input_per_m=round(p.input_per_m, 4),
                output_per_m=round(p.output_per_m, 4),
                blended_per_m=round(blended, 4),
                value=round(float(s["mmlu"]) / blended, 2),
                context_length=p.context_length,
            )
        )
    return sorted(rows, key=lambda r: r.value, reverse=True)


def pareto_frontier(rows: list[Ranked]) -> list[Ranked]:
    """Models where nothing else is both cheaper and higher scoring."""
    frontier: list[Ranked] = []
    best_mmlu = -1.0
    for r in sorted(rows, key=lambda r: (r.blended_per_m, -r.mmlu)):
        if r.mmlu > best_mmlu:
            frontier.append(r)
            best_mmlu = r.mmlu
    return frontier


def main() -> int:
    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
    today = dt.date.today().isoformat()

    with httpx.Client(headers={"User-Agent": "cloud-finops-rag/value-rank"}) as client:
        raw_models = fetch_models(client)

    prices = {p.model_id: p for p in map(parse_price, raw_models) if p is not None}
    scores = load_scores(SCORES_PATH)
    ranked = rank(prices, scores)

    if len(ranked) < len(scores) * 0.8:
        log.error("only %d of %d scored models priced, refusing to publish", len(ranked), len(scores))
        return 2

    SNAPSHOT_DIR.mkdir(parents=True, exist_ok=True)
    snap_path = SNAPSHOT_DIR / f"{today}.jsonl"
    with snap_path.open("w") as fh:
        for r in ranked:
            fh.write(json.dumps({"date": today, **asdict(r)}) + "\n")

    eligible = [r for r in ranked if not r.estimated]
    picks = eligible[:TOP_N]
    frontier = pareto_frontier(eligible)

    print(f"Value picks for {today} (published MMLU only)")
    for i, r in enumerate(picks, 1):
        print(f"{i:>2}. {r.model_id:<45} value={r.value:>8.2f} "
              f"mmlu={r.mmlu:>5.1f} blended=${r.blended_per_m:.3f}/M")
    print("\nPareto frontier (cheapest to most capable):")
    for r in frontier:
        print(f"    {r.model_id:<45} mmlu={r.mmlu:>5.1f} blended=${r.blended_per_m:.3f}/M")
    return 0


if __name__ == "__main__":
    sys.exit(main())

A few choices in there deserve explanation.

The coverage guard (len(ranked) < len(scores) * 0.8) exists because the most likely way for this job to silently produce garbage is an upstream schema change or a wave of renamed model IDs. If a fifth of your scored models suddenly have no price, publishing a ranking of the survivors is worse than publishing nothing.

The explicit estimated flag check in load_scores refuses to default. A missing flag is a data entry mistake and defaulting it to false would quietly promote a guess into the picks.

Snapshots are written with estimated rows included. Hiding them from the picks is a presentation decision. Hiding them from the history would destroy the record of where they would have ranked, which is exactly what you want when a real score is eventually published and you check how wrong the estimate was.

Daily snapshots as JSONL make history queries trivial with DuckDB:

import duckdb

con = duckdb.connect()
deltas = con.sql("""
    WITH s AS (
        SELECT date, model_id, blended_per_m, value
        FROM read_json_auto('data/snapshots/*.jsonl')
        WHERE NOT estimated
    )
    SELECT model_id,
           min(blended_per_m) AS min_price,
           max(blended_per_m) AS max_price,
           round(max(blended_per_m) / min(blended_per_m), 2) AS price_swing
    FROM s
    WHERE date >= current_date - INTERVAL 30 DAY
    GROUP BY model_id
    HAVING max(blended_per_m) / min(blended_per_m) > 1.2
    ORDER BY price_swing DESC
""")
print(deltas)

That query surfaces any model whose blended price moved more than 20% in the last month. Those moves are often the most useful output of the whole system, more useful than the ranking itself.

Production Gotchas, Edge Cases & Failure Modes

Free and variable pricing. OpenRouter lists free variants of some models and uses sentinel values for routers whose cost depends on what they route to. A zero price makes the value score infinite. A negative price makes it negative. Both must be filtered before the division, which is why parse_price rejects anything not strictly positive.

Model ID drift. Providers rename models, add date suffixes and retire versions. When the score file key no longer matches a priced ID, the model silently drops out of the ranking. The coverage guard catches mass drift. Single-model drift needs a periodic diff between scored IDs and listed IDs.

The ratio is mostly a price ranking. Take logs: log(value) = log(MMLU) - log(price). Among models that clear a reasonable quality bar, MMLU might range across 15 or 20 points while prices span 100x. The variance of the second term swamps the first. A hypothetical illustration:

| Model (hypothetical) | MMLU | Blended $/1M | Value score |
|, -|, -|, -|, -|
| Frontier-large | 88 | $6.00 | 14.7 |
| Mid-tier | 82 | $0.60 | 136.7 |
| Small-cheap | 70 | $0.10 | 700.0 |

Small-cheap wins by a factor of five despite being 18 points lower. Whether a 70 is good enough for your task is a question the ratio cannot answer. This is why I print a Pareto frontier next to the picks: it shows the price you pay for each step up in quality instead of collapsing that tradeoff into one number.

Benchmark saturation and errors. Top models score close to the ceiling, which compresses differences further. Independent re-annotation work (the "Are We Done with MMLU?" paper) found a meaningful fraction of MMLU questions contain errors, which puts a noise floor under small score differences. A two-point MMLU gap between two models should not drive a decision.

Contamination. MMLU has been public for years. Some of its questions are likely in training data for many models. Scores measure some mix of knowledge and memorization and that mix varies by model in ways you cannot see.

Tokenizer differences. "Per million tokens" assumes a token is a fixed unit of text. It isn't. Different tokenizers split the same paragraph into different token counts, so two models with identical per-token prices can cost different amounts for the same prompt. Teams that care about precision often run a fixed reference corpus through each tokenizer and convert prices to cost per thousand words or per reference document.

Reasoning tokens. Models that think before answering bill for those hidden tokens, usually at the output rate. A 75/25 blend built on visible input and output dramatically understates their real cost per request. For those models, the effective output share of a request can be far higher than 25%.

Caching discounts. Many providers discount repeated prompt prefixes. In a RAG system with a large, stable system prompt, cached input can cut the real input cost substantially. List prices ignore this entirely, so the ranking can penalize the providers whose caching would actually save you the most.

Pick churn. When two models have nearly identical value scores, small price fluctuations swap their order daily. If anything downstream reacts to rank changes, add hysteresis: only report a new top pick when it beats the incumbent by some margin for several consecutive days.

When NOT to use this score

The value score ignores latency, context length and task fit. Those omissions are disqualifying in several situations.

If your product is latency-sensitive (chat interfaces, autocomplete, voice), time to first token and output throughput matter more than a few cents per million tokens. The ranking contains no latency data and OpenRouter routes some models across multiple hosts with different speeds.

If you need long context, a model's context window is a hard constraint, not a preference. The code captures context_length in the snapshot, but the value score does not use it. Filter on it before ranking, not after.

If your task is coding, tool calling, structured extraction or anything agentic, MMLU tells you little. Multiple-choice knowledge recall is a weak proxy for following a JSON schema or editing a file correctly. Use a task-specific eval set built from your own data.

If you buy through committed-use contracts, self-host open-weights models or have data residency requirements that limit you to specific regions and providers, the OpenRouter price is not your price and the ranking does not describe your options.

And if a single model call is a small fraction of your unit cost (say, one cheap call wrapped around an expensive human review step), optimizing quality per dollar on the model is optimizing the wrong line item.

Benchmark Comparison & Cost Tradeoffs

The job itself costs effectively nothing to run. It makes one GET request per day, reads a YAML file and writes a few kilobytes of JSONL. It fits comfortably in a free-tier scheduled function or a cron entry on any machine already running.

The interesting cost tradeoffs are in what the score drives. Here is how different blend weights reshape the effective price of a hypothetical model priced at $0.50 input and $2.00 output per million tokens, applied to realistic request shapes:

| Workload shape | Input tokens | Output tokens | Actual input share | Cost per 1,000 requests |
|, -|, -|, -|, -|, -|
| RAG lookup | 3,000 | 500 | 86% | $2.50 |
| Summarization | 8,000 | 800 | 91% | $5.60 |
| Chat turn | 1,200 | 400 | 75% | $1.40 |
| Drafting | 500 | 2,000 | 20% | $4.25 |
| Reasoning model, RAG | 3,000 | 500 visible + 3,000 hidden | 46% | $8.50 |

The arithmetic: RAG lookup is 3,000 ร— $0.50/M + 500 ร— $2.00/M = $0.0015 + $0.001 = $0.0025 per request, so $2.50 per thousand. The reasoning row is the one to stare at. Same visible request shape, 3.4x the cost and a 75/25 blend would never show it.

Compare that against the alternative ranking methods teams commonly use:

| Method | What it captures | What it misses | Effort |
|, -|, -|, -|, -|
| MMLU / blended price | Broad knowledge per dollar, daily | Latency, context, task fit, hidden tokens | Low, fully automated |
| Pareto frontier | Price of each quality step | Still single-benchmark | Low, same data |
| Quality floor, then cheapest | "Good enough" at minimum cost | Nuance above the floor | Low, needs a chosen floor |
| Task-specific eval per dollar | Real fit for your workload | Breadth, freshness | High, needs eval set and runs |
| Third-party latency/quality dashboards | Speed and throughput | Your specific prompts | Low, external dependency |

The pattern I would recommend is layered. Use the daily value ranking and price-delta query as a cheap trigger: it tells you when the market moved. When a new model enters the picks or a price drops significantly, that is the signal to run your own task-specific eval against it. The ranking decides what to test. Your eval decides what to ship.

Pragmatic Checklist for Monday Morning

  1. Pull the OpenRouter model list once and diff its IDs against your scored models. Fix every mismatch before trusting any ranking.
  2. Add an explicit estimated: true|false flag to every benchmark score you hold, with a source field. Make the loader fail on a missing flag.
  3. Exclude estimated scores from any list a human will act on. Keep them in the stored history.
  4. Measure your actual input/output token ratio from production logs and set the blend to match. Do not copy 75/25 unless your workload looks like RAG.
  5. Filter on hard constraints (context length, region, tool calling support) before computing value, not after.
  6. Reject zero, negative and non-finite prices before dividing.
  7. Add a coverage guard that refuses to publish when too many scored models lack prices.
  8. Print a Pareto frontier next to the top picks so readers see the quality-price tradeoff instead of a single ratio.
  9. For reasoning models, estimate hidden token volume from a sample of real requests and fold it into their effective output share.
  10. Set up the 30-day price swing query and route its output to wherever your team actually reads alerts.
  11. Build a small task-specific eval set (even 50 to 100 real examples) and run it whenever a new model enters the picks.

References & Further Reading

๐Ÿ“ฐ Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ€” full credit and traffic to the original publisher.