Serving Groot-19B-A3B at 256K on one RTX 5090: the memory budget and the thinking trap

GrootTeam says its Groot-19B-A3B holds a 256K-token context on one RTX 5090, and labels that figure an estimate from memory sizes. A token is a word piece, and the context is how much text the model can see at once. We wanted a measurement, so we deployed the 20.9 GB FP8 build (weights stored in 8 bits to halve their size) with vLLM 0.32.0, the open-source server that runs the model, on one RTX 5090 on our cloud, using the authors' own settings.

The estimate held: all 262,144 tokens of context started, and a 205,525-token prompt came back right. The lesson is how thin the margin is. The full Groot-19B-A3B test on one RTX 5090 has every number; below is what matters when you pick the card.

Two lines in the budget: weights, then the notepad

GPU memory pays for two things. The weights are a fixed cost. The KV cache is the model's notepad for the conversation so far, and it grows with every token of context you allow.

Groot is a mixture of experts (MoE) cut down from the Qwen team's Qwen3.6-35B-A3B: about 3 billion of its 19 billion parameters work on each token, so it writes fast, yet the card must still hold all 19 billion. It inherits Qwen3.6's hybrid layout, where only 10 of its 40 layers write full notes for every token and the other 30 keep a short running summary. So the room for 256K comes from the weights dropping from 35B to 19B.

On the 5090 the weights took 19.7 GiB. With --gpu-memory-utilization 0.92, vLLM had 6.8 GiB left for the cache: room for 307,024 tokens, the full window plus about 45,000 spare. vLLM reports that as a maximum concurrency of 1.17x, meaning 1.17 full-length requests at once.

Trust the line vLLM prints, not the spec sheet

No smaller cards were available, so we capped how much of the 5090's memory vLLM could use. That emulates a smaller budget. It is not a real smaller card.

With a 24 GB budget and 65,536 tokens requested, the weights took 86% of the budget and vLLM refused to start:

ValueError: ... (1.66 GiB KV cache is needed, which is larger than the
available KV cache memory (0.48 GiB). Based on the available memory,
the estimated maximum model length is 9648.

That error answers "will it fit" with one figure: 9,648 tokens, against the 128K minimum the authors recommend. For this build, budget 32 GB. On a 24 GB card the authors suggest their 11.7 GB GGUF build (a format for llama.cpp), which we did not test.

The 13.5 GB NVFP4 build (mostly 4-bit weights) at a 16 GB budget failed too, for another reason: a GPU kernel failed to compile at startup, a software fault and not a memory limit. Whether that build fits remains an open question.

The trap at runtime: it thinks before every answer

Groot reasons before every reply, and the reasoning counts as output tokens. "The capital of Australia, in one word" cost 111 to 131 tokens before "Canberra" appeared. A 300-word essay produced 2,383 to 7,156 tokens over four runs; in the two runs we counted, 2,043 and 6,785 of them were thinking. If your max_tokens is tight, replies get cut off before the answer starts.

Per request, the authors switch thinking off with chat_template_kwargs. Our API drops that field today, so thinking stays on. That gap is on our side and still open. Until it closes, start the server with thinking off by default; the article and the SDK example's README give the one-line flag.

What to measure on your own card

• Cache tokens available once weights load (vLLM's startup log).
• Peak memory on your longest prompt. Ours reached 32,016 of 32,607 MiB on the 205K prompt. Run nothing else on that GPU.
• Time to first token on a long prompt. 34.6 seconds for 205,525 tokens, complete answer at 40.3.
• Writing speed with and without the speed-up layer. The authors ship a small extra layer that drafts two tokens ahead for the main model to check. With it: 262 to 304 tokens per second. Without: 170 to 211. The two measurements took different request paths, so read the roughly 25% gain as approximate.
• Tool call shape. Asked whether Lisbon or Oslo was warmer, it issued two parallel get_weather calls in a single turn, then answered, 1.81 seconds end to end.

A cold start took 611.5 seconds, mostly vLLM warming up the engine; a restart with that work cached on disk warmed up in 87 seconds. Our API showed no measurable overhead in the earlier Sage-A1.5 single-GPU test; it was not re-measured here.

What this does not prove

One GPU, one request at a time, our own prompts, a few runs each. A single correct answer from a hidden sentence shows the window works, not that the model reasons well over long documents. The agent scores are the authors' (52.1 on Terminal-Bench 2.1 against 43.8 for Qwen3.6-35B-A3B); apart from the two-call tool test, we ran no multi-step agent session. The Groot-19B-A3B memory ladder and serve command in the appendix lists every rung we skipped.

Credit to GrootTeam for the model, its four builds, the speed-up layer and the vLLM recipe, to the Qwen team for the base model, to the vLLM and FlashInfer projects for the server, and to AI Search, whose news video is where we found the model. We did the deployment, the tests and the measurements. The commands, tables and an SDK example that runs it on your own GPU are in the full write-up of Groot-19B-A3B on one RTX 5090.