48-Hour Field Notes: My Prompt Budget Was in Characters, and the Overflow Never Raised

Silent truncation is the worst kind of bug in a batch job, because the run exits zero and the output still looks like prose. I spent two days chasing one that never raised an exception, never logged a warning, and only showed up as answers that stopped mid-thought. What would you blame first — the model, the network, or your own length math? In my case it was the third one, and the fix was smaller than the investigation.

The budget I thought I had

Most of us do the same thing when we write our first batch harness: we count characters, divide by some number we half-remember, and call it a token budget. That number usually comes from English prose, where a rough four-characters-per-token rule feels close enough for a while. Then the corpus changes, the ratio quietly stops holding, and the guard you trusted keeps saying yes.

Here is the shape of the guard I started with. It is short, it is honest about being an estimate, and it was wrong in a way I could not see from the logs:

# budget_guard.py
from dataclasses import dataclass


class BudgetError(RuntimeError):
    """Raised before a request is sent, never after."""


@dataclass(frozen=True)
class PromptBudget:
    max_tokens: int = 800
    chars_per_token: float = 3.6  # tuned on English prose, which was the mistake

    def estimated_tokens(self, prompt: str) -> int:
        return int(len(prompt) / self.chars_per_token) + 1

    def check(self, prompt: str, label: str = "prompt") -> None:
        est = self.estimated_tokens(prompt)
        if est > self.max_tokens:
            raise BudgetError(
                f"{label}: {len(prompt)} chars, ~{est} est tokens, budget {self.max_tokens}"
            )

Calling it is deliberately boring, so that a failure points at the prompt and not at the plumbing:

budget = PromptBudget(max_tokens=800)
for item in batch:
    budget.check(item.prompt, label=item.id)
    send(item.prompt)

Hour 0-6: build the fixture corpus before the pipeline

The first thing I did right was refuse to test with one prompt. I wrote four fixture families — English prose, CJK prose, a base64 blob, and an emoji-heavy log line — because those are the four shapes that actually showed up in my data. Each family is generated by repetition so the test is fully reproducible and needs no network access at all.

# tests/test_budget_guard.py
import pytest
from budget_guard import PromptBudget, BudgetError

BUDGET = PromptBudget(max_tokens=800)

FIXTURES = {
    "english_prose": "the quick brown fox " * 120,      # 2,400 chars
    "cjk_prose": "这是一个用于测试的句子。" * 150,          # 1,950 chars
    "base64_blob": "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=" * 60,
    "emoji_log": "✅ shipped 🚀 " * 200,
}


def test_english_passes_and_that_is_correct():
    BUDGET.check(FIXTURES["english_prose"], label="english")


def test_cjk_passes_the_guard_and_that_is_the_bug():
    prompt = FIXTURES["cjk_prose"]
    BUDGET.check(prompt, label="cjk")        # does not raise
    assert BUDGET.estimated_tokens(prompt) < 800

Run it with python -m pytest tests/test_budget_guard.py -q and both tests pass on day one. The second one is a test that documents the defect instead of catching it, and I wrote it that way on purpose so the next person sees the blind spot in the suite rather than in production.

Hour 6-20: what actually broke

On many tokenizers, CJK text costs closer to one token per character than to one token per four characters, so a 1,950-character prompt is nearer 1,950 tokens than the 543 my guard estimated. Base64 and other dense alphabets behave in a similar direction, because they carry almost no redundancy for the tokenizer to compress. English prose was the only fixture family where my ratio was roughly right, and it was also the only family I had calibrated against.

• The overflow never raised, because the guard only ever saw characters.
• The request went out, the endpoint accepted it, and generation simply stopped early for some items.
• My downstream validator rejected those outputs as malformed rather than truncated, which sent me looking at schema code for hours.
• Because retries looked identical to first attempts in the logs, I could not tell which failures were repeats and which were new.

I was running these batches on free model access through MonkeyCode, so a wasted call cost me wall-clock time rather than money, which is exactly why the bug survived a full day of observation. Free tiers make waste cheap to ignore and expensive to find later. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Hour 20-34: fail loudly, then truncate on purpose

The fix was not a better ratio. It was admitting that a character count is a guess, and that a guess needs a worst-case bound whenever the corpus is mixed. For anything I cannot classify ahead of time, I set chars_per_token=1.0, which over-blocks English and under-blocks nothing I have seen. For known-English corpora I keep 3.6 and accept the risk, because the alternative blocks half my legitimate work.

When the guard fires, I want one of two paths, never a third silent one:

1. Split the input at a sentence or record boundary and send two calls.
2. Truncate deliberately with a visible marker, and log the byte offsets you dropped.

BUDGET_STRICT = PromptBudget(max_tokens=800, chars_per_token=1.0)

for item in batch:
    try:
        BUDGET_STRICT.check(item.prompt, label=item.id)
    except BudgetError as exc:
        log.warning("split required: %s", exc)
        for chunk in split_on_boundary(item.prompt, limit=700):
            send(chunk)
        continue
    send(item.prompt)

If your endpoint returns usage counts, calibrate the ratio from them instead of trusting mine, and recalibrate whenever the corpus changes shape. If it does not return counts, treat the ratio as a safety margin and never as a measurement.

A decision table I actually use

Hour 34-48: what I would repeat and what I would skip

What I would repeat: writing the fixture corpus before the pipeline, and keeping one test that documents a known false negative instead of pretending the guard is exact. What I would skip: trying to find the perfect global ratio, which does not exist for a mixed corpus and cost me most of day two.

Limitations and who should not use this

If your SDK exposes the real tokenizer, use it and skip the estimate entirely, because nothing here beats an exact count. If your prompts are homogeneous and short, the guard adds ceremony without much benefit. And if your job can tolerate partial outputs with a downstream repair step, a strict pre-flight block may cost more throughput than it saves in debugging.

A character budget is a guess wearing a number's clothing, and guesses should fail loudly at the boundary rather than quietly inside the response. If you want to run these fixtures without paying per call while you calibrate, the free model access and free server option in MonkeyCode are enough for a harness this small — start with the pytest file above and change one ratio at a time.