A 763B MoE on a 12 GB GPU: Our Speed Ceiling Versus What Strata Reports, and What Closes the Gap

TL;DR

Two days ago we wrote that a 763B mixture-of-experts model squeezed to 2 bits per weight could, on paper, run on a 12 GB GPU, and we put illustrative speed ceilings of 3.4 to 12 tokens per second on it by assuming every expert had to be fetched cold over PCIe or from an SSD. That post had one weak spot: the card did not say how the Strata engine places tensors, so we guessed. The Strata documentation does say, and it is different from our guess. The experts that do not fit in GPU memory are not shipped to the GPU. The CPU computes them where they sit in system RAM, while the GPU works on a cache of the most popular experts.

We did not run Strata or these files. What we did is take the speeds its README and DeepSeek page report, recompute how many expert bytes one token touches for each build, and ask what the reported numbers imply. Three findings. A token touches 1.7 to 2.1 GB of routed experts on the 2.00-bit build, depending on whether the engine uses 5 or 6 experts per layer. The 4.7 tokens per second reported on a 12 GB GPU with 64 GB of RAM is above what a 3.6 GB/s SSD could deliver if experts were hit uniformly, so the routing must be skewed enough for a hot set to carry most of the traffic. And the published speed table is not a clean hardware ranking, because the rows differ in RAM, drive, engine version and settings.

What does Strata report for this 763B model?

Strata is an MIT-licensed local inference engine. When we checked on November 4, its repository had about 39,000 stars, had been created on September 24, and the latest release was v0.2.10, published at 23:52 UTC on November 2. Its README now lists DeepSeek-V4.1-Flash-1017 next to the Qwen models, using the three ISTA-DASLab GGUF sets we looked at on November 2. For the 2.00 bits-per-weight set it gives this table, measured on Strata 0.2.5 (the Strix Halo row on 0.2.7):

These are the project's own numbers, one report per row, and they are not independent measurements. The DeepSeek page of the same repository adds details that do not line up exactly with the README. It gives 4.7 tokens per second for the RTX 5070 box with a 3.6 GB/s NVMe drive, 8 to 10 for the RTX 3060 (9.9 on code, 7.8 on agent work, drafter off), and about 48 tokens per second for four R9700 cards on the 2.00 set where the README says 74 to 77. A user-built sm_89 binary is reported at 40.2 tokens per second on an RTX 4090 48 GB with 128 GB of RAM. The honest reading is a range, not a point.

Where does the engine actually put the experts?

This is the part we had wrong. The DeepSeek page describes three tiers. The dense weights (attention and the other tensors every token needs) go to graphics memory. The routed experts go to graphics memory "as many as fit", and the rest are computed on the CPU, read in place from the model files. The n-gram lookup table, called Engram, stays on the SSD, and only a few rows are read per generated token. The HOW_IT_WORKS page for the Qwen models states the same principle: the card keeps the most-asked experts and learns which ones those are while you chat, the CPU works on the others at the same time as the GPU, and more VRAM matters more than a faster GPU.

For us the consequence is simple. Our November 2 table priced every token as 2.06 GB crossing a link. In this design only the experts that miss the GPU cache cost anything, and the cost is CPU memory traffic rather than PCIe transfer. The SSD enters the picture only when the experts do not fit in RAM either, which is the 64 GB case, because the 2.00 set has about 132 GB of routed experts.

How many bytes does one token touch?

The model has 40 layers with 384 routed experts each, so 15,360 experts, and the router picks 6 per layer per token. Strata also offers a "Faster" mode that uses 5, and it is the default on PCs whose graphics cards total 16 GiB or less and on integrated GPUs. Dividing the routed-expert bytes of each build by 15,360 gives the average expert size, and multiplying by 40 layers and by k gives the traffic per token. The expert totals for the three ISTA-DASLab builds come from the GGUF headers we parsed on November 2, and the Q8 figure is the 288.8 GB that Strata's page lists for Unsloth's UD-Q8_K_XL file.

LAYERS, EXPERTS = 40, 384
N = LAYERS * EXPERTS                      # 15,360 routed experts
EXP_GB = {"IQ1_M": 114.56, "IQ2_XXS": 132.07, "IQ2_XS": 149.06, "UD-Q8_K_XL": 288.8}

def per_token_gb(set_gb, k):              # routed-expert bytes touched per token
    return LAYERS * k * set_gb / N

for name, gb in EXP_GB.items():
    print(name, round(gb * 1000 / N, 2), "MB/expert",
          round(per_token_gb(gb, 6), 2), "GB/token (k=6)",
          round(per_token_gb(gb, 5), 2), "GB/token (k=5)")

Two caveats. The searched allocation gives tensors different types, so the real size of an expert varies by layer and these are averages. And the number is bytes the router asks for, not bytes the machine must move: every expert already in the GPU cache costs nothing on the memory side of the problem.

Is the SSD the limit on a 64 GB machine?

Take the RTX 5070 row. The experts of the 2.00 build are about 132 GB. Assume, and this is our assumption, that after the 6.16 GB dense path and a cache reserve the GPU holds 3 to 5 GB of experts and that 48 to 56 GB of RAM is left for the experts. That is 51 to 61 GB, or 39 to 46% of the expert bytes in fast memory. If routing were uniform, every token would then read the remaining 54 to 61% from the SSD.

With a 3.6 GB/s drive and the Faster mode (5 experts, 1.72 GB per token), that caps decoding at about 3.4 to 3.9 tokens per second. With 6 experts it is 2.9 to 3.2. The reported 4.7 sits above both. The chart shows the SSD-bound ceiling against the share of expert bytes held in fast memory, and where the 4.7 line crosses it.

Turn it around. To sustain 4.7 tokens per second on a 3.6 GB/s drive, the machine can afford at most 0.77 GB of SSD reads per token. That is 44.5% of the expert bytes at 5 experts, or 37.1% at 6. So the cache must serve at least 55.5% (or 62.9%) of the expert lookups while holding only about 42% of the bytes. This is a mild skew, and it is exactly what a hot-expert cache is for. The DeepSeek page says the project ships a "hot-expert profile" built from routing statistics (a separate one for the Coder variant came from 519,422 calibration tokens) and copies it into the pack. A small speculative-decoding drafter, which the page measures at plus 2 to 13% on the RTX 3060 box, can also lower the cost per token, because several drafted tokens share one verification pass.

What this does not prove is the cache hit rate itself. We have a lower bound on skew implied by one reported number and our guess at memory capacity. Running strata launch --plan, which the page says works out expert placement for your cards and RAM, would replace our guess with the engine's own count.

Why is the 128 GB Ryzen box about four times faster than the RTX 5070 box?

On the Ryzen AI Max+ 395 the GPU and CPU share one memory pool. The page says a 128 GB machine keeps about 109 GB of the 2.00 set's experts in that pool and reads the other 23 GB from the SSD, which agrees with the 132 GB total we computed. That puts 82.5% of the expert bytes in fast memory against about 42% on the 64 GB box.

The pool is also fast. The chip's 256-bit LPDDR5X-8000 interface is rated at about 256 GB/s, and community llama.cpp benchmarks report roughly 215 GB/s sustained. Even if every byte of a token's experts came from the pool, 1.72 GB per token at 215 GB/s would allow about 125 tokens per second, so the reported 19 to 21 on code is only 15 to 17% of that bandwidth ceiling. Memory bandwidth is not what limits it. The SSD may be: with uniform routing, 17.5% of 1.72 GB is 0.30 GB per token, which at 19 to 21 tokens per second is 5.7 to 6.3 GB/s of reads, near the limit of a fast PCIe 4.0 drive, and the page does not say which drive was used. Chat at 11 tokens per second would need 3.3 GB/s. Code being faster than chat fits the idea that code routes into the profiled hot experts more often, but that is our hypothesis, and the drafter's acceptance rate on repetitive code would explain it just as well.

Why does a 12 GB RTX 3060 beat the newer RTX 5070 in the table?

Read as a ranking of GPUs, the table says an RTX 3060 is about twice as fast as an RTX 5070. We do not believe that, and the rows give reasons not to. The 3060 row has 46 GB of RAM and the 5070 row 64 GB, the 3060's drafter was off, the 5070 row names its 3.6 GB/s drive and the 3060 row does not. The README also says the Strix Halo row was measured on a newer engine version than the others. A cleaner experiment holds everything fixed except the card, with cold-cache and warm-cache runs reported separately. Until someone publishes that, treat each row as "this machine, this day".

What does the quantization cost in quality?

Strata reports its own comparison against the original model on 4,995 positions. For the 2.00 set it gives KL divergence 0.170 and perplexity up 15.5% (code up 8.6%, agent work up 5.3%). For 2.25 it gives KL 0.107 and perplexity up 7.3%. For IQ1_M perplexity is up 21%, with plain English prose the weakest area at plus 53%. The ISTA-DASLab card, as we noted on November 2, reports 98.7% of the base model's task average for the smallest build on three benchmarks. These measure different things, and neither tells you how the Faster mode with 5 experts changes your own task, which Strata describes only as a small change that is usually not noticeable.

The same page also quotes a perplexity of 3.405 against a reference of 3.140 for the 2.25 set, which is plus 8.4% where the text says plus 7.3%. Likely different runs, but treat single-run figures here as good to about a point.

How should you check this on your own machine?

Start with the plan, then measure, and keep the cold and warm cases apart.

strata launch --family deepseek41 --model ist-das-2.00bpw --plan

Write down the engine's experts-in-VRAM and experts-in-RAM counts from the plan. Divide by 15,360 to get your fast-memory share, then compare your decode speed with the SSD-bound line for that share. If you are above it, your routing is skewed in your favour. If you are well below it, something other than the SSD is the limit, usually the CPU lane or a drafter that is switched off. Log drive throughput while generating, measure with your own prompts and not a fixed one, and compare 5-expert and 6-expert runs at the same settings. Remember the first answer after loading is slower because the cache is cold.

FAQ

Can a 763B-parameter model really run on a 12 GB GPU?
It runs, according to the Strata README, but only because the GPU holds the dense path plus a cache of popular experts while system RAM and the SSD hold the rest. Speeds reported for 12 GB cards are 4 to 11 tokens per second on the 2.00 set, with the exact figure depending heavily on RAM and drive.

How much system RAM do you need?
The experts of the 2.00 build are about 132 GB. The Strata page says that under about 40 GB of RAM the setup is untested and slow, and that the Engram table, 111 GB shared by all three builds, belongs on a fast NVMe drive. The reported 12 GB GPU rows have 46 and 64 GB.

What does the Faster mode change?
It uses 5 experts per layer instead of 6, which cuts the bytes touched per token by 17%. It is the default on PCs with 16 GiB of graphics memory or less and on integrated GPUs such as Strix Halo.

Are these speeds independent measurements?
No. They are the project's reported numbers, and we did not run the engine. Our contribution is the byte arithmetic and the bounds it implies, and the cache-capacity figures for the 64 GB box are our assumptions.

Further reading

• A 763B MoE at "2 Bits": Where the Bytes Really Go in the GSQ-RCO GGUFs, and Why 100% Recovery Sits Inside the Noise
• Why a Sparse 35B MoE Runs Faster on a CPU Than a Dense 27B Model

Sources: Strata repository and README, Strata DeepSeek-V4.1-Flash page, Strata HOW_IT_WORKS, ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF, Strix Halo memory bandwidth, 256 GB/s rated versus about 215 GB/s measured. Speeds, quality figures and memory placement are as reported by the Strata project on its pages as of November 4, 2026. Bytes per token, SSD ceilings and implied hit rates are our own illustrative arithmetic, not measurements.