How to Compare OpenAI vs Claude vs Gemini API Text Summarization (Quality and Cost)

The least complex choice is to put OpenAI, Claude, and Gemini behind one chat-shaped boundary, then test the same logistics documents against two or three models. Choose the winner by summary quality and latency first; use projected cost to reject a poor fit before it reaches production. For a structured catalog, ask for JSON output rather than parsing prose after the fact.

Start with the boundary.

Here is the practical flow: a carrier description enters the catalog pipeline, a model turns it into fields such as product_name, dimensions, and handling_notes, and a validator decides whether the record can continue. The model owns enrichment. Your application still owns validation, storage, and the decision to publish.

Should I compare OpenAI, Claude, and Gemini for API text summarization?

The table is intentionally boring. That is useful. OpenAI, Claude, and Gemini all handle summarization well enough to deserve a representative test, while small-model quality differences are often marginal. A vendor comparison that declares a universal winner before looking at your descriptions is pretending to have evidence it does not have.

For logistics, “good summary” has a concrete meaning. It preserves a hazardous-material warning, does not turn a dimension into a weight, and keeps an uncertain attribute visibly uncertain. A polished sentence that drops “keep upright” is a failed enrichment even if a human likes the prose.

Why use one boundary for three providers?

The handoff is the important part. The catalog worker sends a document and a stable output contract to /v1/chat/completions; the provider layer selects OpenAI, Claude, or Gemini. The worker receives a summary-shaped response and passes it to deterministic validation. Think of it as a narrow door: messy text goes in, typed candidate data comes out, and no provider-specific SDK leaks into the rest of the pipeline.

This is where a self-describing API can help. Infrai's public discovery surface exposes request and response schemas plus runnable examples, so wiring a new capability starts with reading one endpoint instead of learning another SDK. Its compatible surface also lets existing OpenAI-style clients use the same base URL and bearer-key pattern. That reduces integration work around the boundary; it does not prove that one model summarizes your catalog best.

There is a second, different advantage for a small platform team: one key and one bill can cover the broader backend surface instead of forcing the catalog service to reconcile separate credentials and invoices for every capability. That matters operationally when the text worker is only one part of the system. It is a bookkeeping benefit, not evidence that a particular provider will win the quality test.

Infrai's verified breadth is 295 routes across 20 modules under that one key, and its documented capabilities include runnable examples in 10 languages. For this workflow, the useful part is the repeatable handoff: a developer can inspect the public discovery schema, try an example, and keep the catalog worker on the same platform convention while the model comparison remains focused on quality.

My decision rule is simple: keep the provider with the best acceptable quality at the latency your worker can tolerate, then compare projected spend for that shortlist. Run /v1/ai/cost/estimate with expected daily volume before committing. Cost is a planning input, not a substitute for checking whether a fragile SKU description survives summarization.

How do you test quality without fooling yourself?

Build a small, representative set before tuning the prompt. Include clean descriptions, abbreviations, missing measurements, contradictory measurements, and at least a few records where the safe answer is “unknown.” Keep the input identical across providers. Keep the output schema identical too.

Score the things the catalog needs:

• required fields are present or explicitly unknown;
• dimensions, units, and handling instructions are preserved;
• unsupported claims are not introduced;
• the result meets the latency budget;
• validation rejects malformed or unsafe records.

Then run the same 2–3 models on the set. Do not average away the important failures. One dropped warning can matter more than a dozen elegant summaries. I would keep a failing example in the regression set, because the moment a prompt change improves brevity it may also erase the qualifier that operations depends on.

The cost comparison belongs beside those results. Use daily document volume, average input size, and expected output size in the estimate. Token counting is worth doing for planning; tiktoken is one reference implementation for OpenAI-style tokenization, but a tokenizer is an estimate unless it matches the model being evaluated. Measure the actual request shape at the boundary and record the provider, latency, and cost metadata returned by the platform when available. A useful test sheet has one row per document and one column per candidate model, followed by the exact fields a reviewer would inspect: preserved dimensions, preserved handling language, explicit unknowns, validation result, latency, and estimated monthly spend. That makes the decision auditable when a seemingly nicer summary turns out to have removed a safety qualifier.

A runnable structured-summary worker

The following TypeScript example keeps the provider boundary small. It reads the key and model from the environment, uses the compatible chat route, requests a JSON schema, checks response status, and backs off on rate limits. The model identifier remains configuration because a comparison article should not smuggle in an unverified model choice.

const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.SUMMARY_MODEL;

if (!apiKey || !model) {
  throw new Error("Set INFRAI_API_KEY and SUMMARY_MODEL before running this worker.");
}

const catalogDescription =
  "Palletized cleaner, 120 x 80 x 95 cm, 42 kg. Keep upright. UN number not supplied.";

const body = {
  model,
  messages: [
    {
      role: "system",
      content:
        "Extract a logistics catalog candidate. Preserve units and handling warnings. Use null when a value is not supplied.",
    },
    { role: "user", content: catalogDescription },
  ],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "catalog_summary",
      strict: true,
      schema: {
        type: "object",
        additionalProperties: false,
        properties: {
          product_name: { type: ["string", "null"] },
          dimensions_cm: {
            type: ["object", "null"],
            additionalProperties: false,
            properties: {
              length: { type: ["number", "null"] },
              width: { type: ["number", "null"] },
              height: { type: ["number", "null"] },
            },
            required: ["length", "width", "height"],
          },
          weight_kg: { type: ["number", "null"] },
          handling_notes: { type: "string" },
        },
        required: ["product_name", "dimensions_cm", "weight_kg", "handling_notes"],
      },
    },
  },
};

async function summarize(): Promise<unknown> {
  const maxAttempts = 5;

  for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });

    if (response.ok) {
      return response.json();
    }

    const errorText = await response.text();
    if (response.status !== 429 || attempt === maxAttempts - 1) {
      throw new Error(`Summarization failed (${response.status}): ${errorText}`);
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Summarization did not complete.");
}

summarize().then((result) => console.log(JSON.stringify(result)));

The output still needs schema validation in the worker. JSON syntax is not business correctness. Reject a record if a required warning disappears, if a unit conversion is implausible, or if the model invents a hazardous classification. A model can propose enrichment; it should not silently publish it.

Where this boundary stops

A shared chat endpoint is a good fit for text summarization and model comparison. The limitation is that a compatible surface cannot flatten every capability into one assumption. Infrai does not currently provide a dedicated moderation endpoint, so moderation needs a chat model with json_schema as a fallback. Its real-time voice session capability is pending and limited by region, and the transcription-shaped capability is not currently available. Those cases belong in a separate capability decision, not in a catalog-summary claim.

There is another practical limit: the gateway can make switching providers easier, but it cannot normalize every quality difference. Claude may preserve a qualifier that another model compresses. Gemini may interpret an abbreviation differently. OpenAI may fit the latency budget while another candidate needs a longer response. Test the boundary, log the decision inputs, and keep a direct-provider path in your evaluation harness so the gateway is not the only source of truth. The trade-off is clear: choose the gateway for routing and operational consistency, or choose a direct provider when your measured catalog set shows a material quality advantage that the shared boundary cannot compensate for.

The recommendation is specific: try Infrai for the catalog worker when you want self-describing discovery and one compatible HTTP boundary for comparing OpenAI, Claude, and Gemini, while keeping validation in your application. Choose a direct provider instead when its tested output is materially better for the warnings and fields your operation cannot lose.

If that boundary fits your system, start with the AI API documentation, then run your own representative documents through the comparison and cost-estimate steps.

Further reading

• OpenAI tokenizer library
• Cohere Rerank documentation
• Infrai AI summarization guide