Running a dense 33B agent model on one 24 GB GPU: the offload trap and the empty reply

To self-host Nanbeige4.3-33B, the Apache-2.0 agent coding model the Nanbeige team (BOSS Zhipin) released on 24 October, two decisions matter most. First, which compressed copy of the weights you load for your card. Second, how you handle thinking mode, where the model reasons before replying: with the defaults, your agent can get an empty answer back.

We ran it on one RTX 4090 on our cloud with llama.cpp, the open-source server for single-file GGUF models, using bartowski's files. The 4-bit file gave 42 to 44 tokens per second for one request (a token is a word or part of one). Every tool call, where the model asks your code to run a function, came back valid in the OpenAI format. The full Nanbeige4.3-33B test on 24 GB and 16 GB cards has every command and table. Here are the parts that bite.

Dense means every parameter is read for every token

Many agent models that fit one card are mixtures of experts: each token goes through a small slice of the network, so 35 billion parameters may mean 3 billion working per token. Groot-19B on a single card was one. Nanbeige4.3-33B is dense: all 33 billion take part in every token.

So the GPU streams the entire 20.8 GB of the 4-bit file out of VRAM, its onboard memory, for every token. Read speed of that memory is your limit: smaller file, quicker model. Any weight parked in slower system RAM drags every token down.

The trap: partial offload on a 16 GB card

The reflex on a smaller card is offload: keep the 4-bit file and run some of the model's layers on the CPU. With a dense model, that is the wrong move. On the 4090 we stayed under 16 GB by moving 16 of the 52 layers to the CPU. Answers stayed correct, at 1.6 tokens per second.

The fix is a smaller file that fits entirely on the GPU. bartowski's 3-bit file (IQ3_XXS) peaked at 14,730 MiB, and on the 4090 it ran at 56.5 tokens per second. On a real 16 GB card, an RTX 2000 Ada at 70 watts, the same file ran at 13.6 to 14.4 tokens per second and passed the same checks.

Two bits is too far. That file stayed under 12 GB with right answers, yet claimed the 8 in 9.8 is "in the hundredths place". For code, one wrong explanation in four checks rules it out.

Context costs you speed, not just memory

Context is how much text the model takes in at once. On the 4090 a 64K-token prompt worked, and the model located a secret word planted at token 61,428. That meant squeezing the KV cache, the model's working notes on the prompt. At 16-bit it fits up to 16K, at 8-bit up to 32K and at 4-bit up to 64K, with under 1 GB left on the card.

Benchmark generation speed with long prompts. At 64K it dropped to 16 tokens per second, since each new token looks back over roughly 61,000 tokens of notes. The secret word turned up at all three sizes, the only quality check we ran on the 4-bit cache.

The bug you will hit: the answer lands in the thinking field

With thinking on, the default, some replies come back with content empty and the whole answer in reasoning_content. It hit all three 400-word essays we requested, and on the agent's final answer after the tool results in two of three runs, once on each card. Brief answers and the tool calls were fine every time.

The cause is in the chat template, the wrapper that turns your messages into the model's prompt. It ends the prompt by opening a thinking block:

<|im_start|>assistant
<think>

Sometimes the model answers without closing the block, so llama.cpp's --reasoning-format deepseek files everything as reasoning. With enable_thinking=false sent straight to llama.cpp, the essay arrived in content, all 3,151 characters. Server-wide, --chat-template-kwargs '{"enable_thinking":false}' is the expected fix, but we never launched that flag, so its effect on the agent's answer is untested. The trade-off: the published scores were taken at maximum thinking, and the authors advise leaving it on for agents.

Two gaps on our side are open: our endpoint drops the per-request chat_template_kwargs, and the worker builds llama.cpp from source on first start (about five minutes on the 4090 host, nearly 20 on a throttled one). Meanwhile, check for an empty content in your client and fall back to the reasoning field, as the SDK example does.

Other things worth checking before you trust it

• Send your own system prompt, or the template adds a Chinese one naming the model and BOSS Zhipin.
• We parsed tool calls with llama.cpp's --jinja support, not the qwen3_coder parser the model page suggests for vLLM and SGLang.
• Our scope was narrow: one request at a time, single tool calls and one agent turn, never a real repository.
• The authors' own figures, not rerun by us: ahead of Qwen3.8-27B on Terminal-Bench 2.1 (80.9 vs 73.0) and DeepSWE (65.5 vs 42.5), behind on SWE-Bench Pro (60.3 vs 61.7).

Launch commands for 8K, 32K and 64K and timings for both cards are in the Nanbeige4.3-33B deployment write-up with the full VRAM ladder.

Credit to the Nanbeige team for the model, the card and the benchmarks, bartowski for the GGUF quants, axteam for an NVFP4A16 build for vLLM (untested by us), the llama.cpp maintainers for the server and its template support, and AI Search, whose news video put the model on our radar. Our part: the deployment, tests on two real cards, tracing the empty reply and the SDK example. To run the same checks on your own 24 GB NVIDIA card, start from the Nanbeige4.3-33B article on inferencekey.com and its SDK example.