Clover-35B-A3B on llama.cpp: what "AR mode only" means, and a GGUF that answers in Gs

Four words in a pull request decide how fast DiffuSpace's Clover-35B-A3B-Preview runs on your card: "in AR mode only." It added Clover to llama.cpp, the free program many people use to run models locally. AR (autoregressive) means the model emits one token, a word piece, then starts the next. Clover was built to work differently.

We loaded the official Q4_0 GGUF on one RTX 4090 on our cloud, with stock llama.cpp v0.9.0. GGUF is the single-file format llama.cpp reads; Q4_0 is a compressed build. After a small file fix, one request got 202 to 211 tokens per second, with the first token back in 0.13 to 0.31 seconds. The full 128K context window (prompt plus answer in memory) peaked at 21,924 MiB. Every timing is in the full Clover-35B-A3B test on one RTX 4090.

"Supported" can mean the slow path

Clover is a diffusion language model. A normal model is a typist: token, token, token. Clover drafts a block of 32 slots, commits in one pass every slot where its confidence reaches 95%, and revisits the others. That parallel filling is the speed story, and the engine has to implement it.

PR #31306 by Maphist0 landed on 2 November, and the v0.9.0 release carried it on 4 November. It runs Clover one token per step. A faster self-speculative mode (draft a block in parallel, then verify it the normal way) was left for a later pull request, and we found none open. So 208 tokens per second measures the typist, not the diffusion trick.

DiffuSpace quotes 515 tokens per second on one H200 and 680 on a GB300, uncompressed, naming neither engine nor mode. The parallel modes live elsewhere. DiffuSpace's own Python code wants about 69 GB for the weights alone. SGLang PR #46927 from Zirui Wu brings draft-and-verify to SGLang, another open serving engine. It awaits review, and its own test logged 453.59 tokens per second on a single H200, one request at a time.

The trap: a file that loads is not a file that works

Out of the box, v0.9.0 rejects the official file:

key not found in model: clover_moe.expert_shared_count

The tempting fix is to add that key. We did. It loaded, we asked for Australia's capital in one word, and got the letter G 600 times.

A load error stops you. Garbage from a model that loads can slip into a pipeline. The root cause: DiffuSpace exported its files from its own llama.cpp build, which drifted from upstream in three places. One key is absent. There are 120 extra tensors (the named arrays of numbers that make up a model) only the diffusion mode reads. And 30 layers each keep a short vector of values written in another mathematical form.

Our patch_gguf.py handles all three and leaves the other weights untouched, so nothing gets compressed twice. On the 19.6 GB file it took 17 seconds, and Clover then answered Canberra. The notes on what patch_gguf.py changes name each group of tensors it touches. Status: fixed in our SDK example, open upstream in both DiffuSpace's files and llama.cpp. We checked the patched file only on v0.9.0.

The habit: before benchmarking a new architecture, ask it one question whose answer you know. A model that replies in Gs still produces a tokens-per-second figure.

What to measure on your own card

Clover has 35 billion parameters, yet each token uses about 3 billion. It is a mixture of experts: 256 specialists, and each token consults 8 plus one generalist. All sit on the GPU unless llama.cpp's --n-cpu-moe moves some layers of experts to system RAM, trading speed for card memory. With the 4090's memory capped:

All three budgets gave correct answers. Read the table for memory, not speed: a real 16 GB desktop runs at whatever its processor and RAM allow. Our real 16 GB attempt hit a broken host, so that number is missing.

Three more checks:

• A long prompt with a needle. We buried a code in an 89,367-token prompt with the 128K window. Clover returned it in 13.45 seconds.
• Thinking tokens. Clover reasons before every answer. A 400-word essay cost 4,665 tokens and 22.7 seconds, mostly reasoning. Our gateway drops chat_template_kwargs, the field that carries the off switch. That gap is ours, still open.
• The base model. Clover comes from Qwen3.6-35B-A3B by Alibaba's Qwen team. That base never ran on our 4090, so there is no head-to-head. Run both on your own workload.

Pin the version, not the latest

On InferenceKey, the worker is the agent process on your GPU machine; it takes instructions from the platform and builds llama.cpp itself. Its default tag, b9748, predates Clover. Pinning this one workload (a model deployment) to v0.9.0 solved it. The first compile for the GPU took 4 min 50 s. From then on, the endpoint (the URL your code calls) was ready in 11.7 seconds. The worker also needed three CUDA build variables it does not set yet; the example lists them.

Credit: DiffuSpace built Clover and published the files, Alibaba's Qwen team made the base, Maphist0 wrote the llama.cpp support the maintainers released in v0.9.0, and Zirui Wu is building the SGLang path. We first saw the model on the AI Search channel. Our part was the patch and the measurements.

The patch, the exact llama-server command, the timings and the full memory ladder are in the Clover-35B-A3B write-up on inferencekey.com.