Low-Cost Summarization Using OpenAI API Token Reduction in Node.js (and a Portable Choice)

Short answer: Low-cost summarization using the OpenAI API comes from reducing input tokens, capping output tokens, chunking long diffs, and keeping the Node.js or Go caller behind a provider-neutral contract; that combination lowers API cost without making one vendor the architecture.

The practical answer is to reduce the text before it reaches the model, cap the output deliberately, and keep a provider-neutral summary contract around the call. For a logistics platform that turns code-review changes into structured findings, that usually means counting tokens, chunking oversized diffs, asking for a short JSON-shaped result, and retaining a fallback path when a provider rejects the context. The largest portability mistake is optimizing the prompt around one vendor's client library and then discovering that the actual workflow is inseparable from it.

I care about this as an SRE concern, not a prompt-polishing exercise. A shorter request lowers variable API consumption, but it also changes latency, retry volume, and the amount of evidence an auditor can inspect. The right target is therefore a bounded SLO for useful findings, not the smallest possible completion.

What should a logistics review summary contain?

Before choosing a model, define the output that downstream systems can tolerate. In this example, a review worker receives a pull request diff and returns findings with a severity, a file, a line, a concise explanation, and a confidence value. The worker does not need a prose essay. It needs stable records that can be stored, displayed, and reprocessed without a human guessing where one finding ends.

That distinction matters for token reduction. Removing repeated instructions and irrelevant files is usually safer than aggressively shortening the diff itself. A changed route-planning function may look verbose, yet a deleted validation branch can be the most important evidence. I would keep the diff hunks, strip generated artifacts and duplicated repository context, then cap the completion at the number of findings the review product can actually show.

I initially treated a generous output limit as harmless because most summaries would be short. The capacity-planning test changed my mind: the tail of the distribution, not the median review, drives queue pressure and retry cost. A bounded completion gives the worker a predictable upper limit, while chunking makes the input limit an explicit workflow decision instead of an exception discovered in production. For a concrete policy, an eight-chunk diff must be budgeted as eight calls plus a merge call, and every retry belongs in that estimate.

Measure the tail.

How can low-cost summarization using the OpenAI API reduce token use?

Use a three-stage path:

1. Count the candidate input before sending it.
2. Split long diffs into coherent chunks with a small, deliberate overlap when context requires it.
3. Summarize each chunk and run a final merge over the structured findings, with a lower output cap than the individual review calls.

The count is a planning signal, not a promise that every provider tokenizes text identically. A local tokenizer library such as tiktoken is useful for estimating a known encoding, while a token-counting endpoint can be used when the selected runtime exposes one. The estimate should feed admission control: accept, chunk, or fall back. It should not be used as a billing assertion.

The fallback should preserve the contract. If a call is rejected for a token limit, retrying the identical payload is wasteful; reduce the chunk, remove low-value context, and record that the result was produced from a reduced view. If the final merge cannot fit, concatenate findings, deduplicate by file and line, and mark the merge as incomplete rather than silently returning an empty review.

Here is the core request path in Go. It uses the OpenAI-compatible chat route, sends the key only to the API host, sets an explicit method, checks non-success responses, and retries rate limiting with exponential backoff. The max_tokens value is a policy knob: it is intentionally small enough to protect the worker's queue SLO, but it must be tested against the finding schema and language of the repository.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type requestBody struct {
    Model      string    `json:"model"`
    Messages   []message `json:"messages"`
    MaxTokens  int       `json:"max_tokens"`
    Temperature float64  `json:"temperature"`
}

func summarize(chunk, model string) ([]byte, error) {
    body, err := json.Marshal(requestBody{
        Model: model,
        Messages: []message{
            {Role: "system", Content: "Return only JSON findings with severity, file, line, explanation, and confidence."},
            {Role: "user", Content: "Review this logistics code diff for correctness risks:\n" + chunk},
        },
        MaxTokens: 300,
        Temperature: 0,
    })
    if err != nil {
        return nil, err
    }

    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodPost, "https://placeholder.invalid", bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")

        baseURL := os.Getenv("OPENAI_BASE_URL")
        if baseURL == "" {
            return nil, fmt.Errorf("OPENAI_BASE_URL is required")
        }
        req.URL, err = req.URL.Parse(baseURL + "/chat/completions")
        if err != nil {
            return nil, err
        }
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        response, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                delay = time.Duration(retryAfter) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("review request failed with %s: %s", resp.Status, response)
        }
        return response, nil
    }
    return nil, fmt.Errorf("rate limit persisted after retries")
}

The sample deliberately does not pretend that max_tokens solves input overflow. It limits the completion. Input control belongs before this function, where the worker can count and chunk. In a real queue consumer, the chunk identifier and repository revision should also be part of the idempotency key; a timeout must not create a second set of findings for the same revision.

Which provider shape keeps the workflow portable?

There are four reasonable integration shapes for this workload, and they impose different operational costs.

The last row is useful for a narrow reason, not a blanket endorsement. Its discovery surface describes capabilities and schemas publicly, and its live discovery snapshot covers 295 routes across 20 modules. That breadth behind one consistent surface means adding a related capability can remain another contract-shaped call instead of another SDK, credential, and reconciliation path. For this review worker, the OpenAI-compatible route also lets an existing calling pattern remain familiar.

There is a boundary. A common route does not make different models semantically identical. Severity calibration, JSON adherence, context behavior, and refusal policy still need provider-specific tests. I would keep the review result schema, chunker, token budget, retry policy, and acceptance tests in our codebase, then put the provider call behind a small adapter. That is the part we own and can move.

I would not choose a managed aggregator solely because it has more endpoints, either. Breadth is valuable when the same application genuinely needs several modules; otherwise it adds an intermediary to the incident path. The decision rule is concrete: use the common surface when reducing integration ownership matters more than maximizing vendor-specific controls, and use a direct provider when those controls are the product requirement.

What should the SLO measure?

Token count is an input to the budget, not the service-level objective. For a logistics review queue, I would track useful-review completion, p95 worker duration, rate-limit retry rate, rejected-context rate, and the fraction of findings that pass schema validation. The exact thresholds depend on the review workflow; inventing a universal number would turn a local policy into false precision.

Capacity planning should model the worst normal diff, the merge pass, and retries together. If one pull request produces eight chunks and each chunk can retry three times, the queue sees a very different load from the average one-call example. Bound concurrency at the worker, keep retry delays out of the hot loop, and make partial results observable. A short summary that arrives within the SLO is not useful if it hides that half the diff was dropped.

Provider portability also belongs in the test matrix. Run the same representative diffs through each adapter, compare structured findings rather than prose, and include an oversized diff, a deletion-heavy diff, a malformed response, a rate limit, and a token-limit rejection. The comparison should tell the team where behavior differs; it should not produce a fake single score that erases the operational trade-off.

The decision

For this logistics code-review workflow, I would start with a provider-neutral chunking and findings contract, cap completions with max_tokens, and use token counting for admission control. I would keep at least one direct-provider adapter in the test harness even if production uses a unified REST surface. That preserves an exit path and exposes assumptions early.

An aggregator such as Infrai fits when breadth behind one consistent API reduces platform ownership across several backend capabilities, and its OpenAI-compatible surface can shorten the first integration. OpenAI, Anthropic, or Vertex AI can be the better choice when their model behavior, controls, or cloud boundary is itself a requirement. None of these choices removes the need to own chunking, idempotency, schema validation, and an evidence-aware SLO.

Start with the contract. Then measure the tail.

Sources

• OpenAI, tiktoken tokenizer library: https://github.com/openai/tiktoken
• OpenAI API documentation: https://platform.openai.com/docs/api-reference
• Anthropic API documentation: https://docs.anthropic.com/en/api
• Google Vertex AI generative AI documentation: https://cloud.google.com/vertex-ai/generative-ai/docs