A 1.6 MB screenshot costs fewer than 1,000 tokens — my trimmer was deleting it anyway

Ankita is a personal AI companion for the desktop — "a little companion, a lot more possible" — and companions live in long sessions. That is where context trimming fails quietly: nothing crashes, and the model just answers a question about a screenshot it never saw.

The bug is a unit error. A pasted screenshot travels as a base64 data URL, and the naive way to budget history is to measure that string. A 1.6 MB PNG then looks like 1.6 MB of context, so the trimmer deletes it first — before the model ever sees it. But vision attachments are billed by pixels, not by base64 length. That same image costs fewer than a thousand tokens. Measuring it as text guarantees the trimmer throws away the cheapest, most informative part of the turn.

Budget the image the way the provider bills it

The fix lives in src/core/history.mjs, in trimMessages(). Instead of weighing the data URL, Ankita reads just the image header and prices the pixels:

• PNG dimensions come from the IHDR chunk, JPEG from its start-of-frame markers, GIF from its 10-byte header, WebP from its VP8 / VP8L / VP8X chunks. Nothing is fully decoded.
• The estimate follows the common high-detail recipe: fit the longest side inside 2048 px, resize the short side to 768 px, then count 512 px tiles at 85 + 170 tokens per tile. It is deliberately an upper bound across OpenAI-compatible vision endpoints, converted back to bytes at 4 bytes per token.
• An image that cannot be decoded falls back to the old behaviour — its URL length — so an unreadable image never gets a free pass.

The repo's own test makes the difference concrete: a 1700 × 2200 PNG padded with 100,000 filler bytes has a data URL longer than 100 KB, but its vision estimate is under 5,000 bytes. Worked through: 2048 / 2200 scales it to about 1583 × 2048, the short side resizes to 768 px giving roughly 768 × 994, which is 2 × 2 = 4 tiles, so 85 + (170 × 4) = 765 tokens, and 765 × 4 = 3,060 bytes. The trimmer now keeps that image; the old one deleted it.

Keep the goal, keep the pair, drop whole groups

Pixel pricing alone does not fix trimming — the order of what survives matters just as much. Three rules govern trimMessages():

Anchor the request. The newest user message is always kept, plus up to two earlier user messages (USER_REQUEST_ANCHORS = 3), with those earlier requests capped at 4,096 bytes combined. The point is that a long tool loop should not erase the goal or a recent correction.

Never split a tool exchange. An assistant tool call and its tool replies are one atomic group (TOOL_EXCHANGE_MESSAGES = 2 minimum). Dropping the reply but keeping the call — or vice versa — hands the model a conversation that never happened. Old groups are dropped whole, newest first, and chronology around the anchored request is preserved.

Shed in a fixed order, and fail loudly at the bottom. Under byte pressure, the biggest text units shrink first, and an attached file body is droppable only as a last resort — losing the file body beats failing the whole turn. Images are dropped only when their vision tokens genuinely do not fit. But if even the current request plus the core tool schemas exceed the window, trimming throws instead of sending a lobotomised turn: "The current request/tool schema exceeds the model context budget." The agent side (Agent.trimHistory() in src/core/agent.mjs) reserves output tokens, 1,024 bytes of overhead, the tool schemas, and memory context first, and refuses to run at all below 512 bytes of history.

Restoring a saved session gets the same treatment. Orphan tool messages from an interrupted session are discarded, malformed tool calls are replaced with a visible "[Incomplete tool calls removed from restored session.]" note, and — the bug that motivated part of this — user turns stored as an array of content parts are no longer silently deleted for not being strings. That one used to drop the attachment and leave the model re-answering stale context.

What this does not solve

The estimate is an upper bound, not any provider's exact count, so a tight window may still drop an image that would have fit. Worse, when an image is dropped to save the turn, nothing tells the user the model never saw it — the turn succeeds, blind. And the 4,096-byte prior-request cap means an older correction eventually vanishes no matter how important it was. Those are real trade-offs, not footnotes.

So here is the design question I keep coming back to: when the budget runs out, is silently dropping the image to save the turn the right call, or should the turn fail loudly so you know the model never saw it? I chose "save the turn" — a companion that refuses to answer is worse than one that answers from text — but I am not sure that holds when the whole question is about the image. Which failure mode would you ship?

Ankita is open-source (MIT) at https://github.com/akyourowngames/A.N.K.I.T.A — the trimming code is src/core/history.mjs, and I would genuinely like to hear how you budget multimodal history, or where you think this ordering is wrong.