Hot experts, cold RAM: running a big MoE on a machine with more memory than VRAM
Two budgets, and only one of them is expensive Every local deployment spends against two budgets at once: the size of the model and the price of the memory holding it. VRAM is the one people reach for first, and it is
Two budgets, and only one of them is expensive
Every local deployment spends against two budgets at once: the size of
the model and the price of the memory holding it. VRAM is the one people
reach for first, and it is the one that costs the most per gigabyte.
System DDR is fast, vast and cheap; GPU memory is fast, small and
expensive. An inference engine that can put weights in the second place
changes the bill.
That is what Strata is for. It is forked from llama.cpp and built for
memory efficiency: it keeps resident only the experts that are hot – the
ones actually in use – and leaves the rest parked. That is how big MoE
models run on small RAM.
Where llama.cpp ends and Strata starts
llama.cpp is the base of everything here: an inference engine in plain
C/C++ that runs GGUF models on hardware you already own – Metal, CUDA,
Vulkan, SYCL or CPU only – with 1.5-8 bit quantisation and hybrid
offload for models too big for the GPU. Hybrid offload moves layers
between CPU and GPU. An MoE is a different shape of problem: it is not
one network that is too big, it is a fleet of experts of which only a
few matter to a given query.
So the efficient move is not to place layers. It is to place experts,
and to decide which of them deserve to be resident right now. Strata
does exactly that, and Avalon forks Strata to tune the same hot set even
tighter, which buys longer usable context and larger models on the same
hardware.
DDR over VRAM
The second half of the picture is where the weights sit while they wait.
The engine streams model weights through ordinary system memory – fast,
vast DDR instead of expensive GPU VRAM. One machine with 256 GB of DDR
and an RTX 5080 runs small and medium private AI comfortably.
For a small organisation the consequence is concrete: the machine you
can afford is the machine that fits the model you want. A 5080-class
card with a lot of memory you already bought does more work than a
bigger card with less, and the upgrade path is RAM, not a new GPU.
Why the hot set holds
Expert selection is not random. Queries cluster: a support inbox about
one product, a codebase, a corpus of legal text. The gate keeps coming
back to the same experts, so the hot set stabilises after a while,
page-in traffic drops away, and the parked experts stay parked – which
is exactly the efficiency argument for Mixture of Experts in the first
place: each query activates only the experts it needs, and the rest
never load into memory at all.
What it asks of you
Two things. First, an NVMe that is actually fast: the cold tier is only
cheap if the first read of an expert is quick. Second, a model worth the
complexity – a dense model that already fits in VRAM is usually the
simpler answer. And measure rather than trust: one model file, weights
and prompt length held identical across every engine that can load it on
that box, reading and writing speed measured apart, versions recorded
with the number (benchmark
index).
The part that stays with you
Because inference runs on your own hardware inside the building,
personal data never leaves the premises. No Chapter V transfer to
arrange, no standard contractual clauses to keep valid, no US cloud
disclosure to answer for – the accountability question never comes up,
because the violation never happens. Ask about an on-site assessment:
[[email protected]](mailto:[email protected]).
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.