Mixture-of-Depths vs Uniform Compute: The Routing Tax on Every Token

Every token in a standard transformer pays for every layer, whether the token is a difficult reasoning step or a comma. Per-token depth routing promises to charge each token only for the layers it needs. On a spreadsheet the saving looks like a straight percentage of FLOPs. On a real accelerator the saving is mediated by a router, by memory traffic, and by the awkward fact that a batch can only run as fast as its slowest member.

With {a}, a small learned gate assigns each token a capacity for a layer and the excess tokens bypass it, so the compute graph becomes ragged. With {b}, every token traverses every layer and the kernel sees a dense, predictable tensor. Dense execution is boring in the best sense: it maps cleanly onto matrix-multiply units, needs no gather or scatter, and keeps the whole batch aligned. The ragged version exchanges arithmetic for control flow, and control flow is exactly what a GPU is least eager to do.

A FLOP reduction is not a latency reduction when the workload is bound by memory traffic rather than arithmetic.

Part One: What is actually being compared

The two options are often described as competing implementations of the same thing, which is misleading. per-token depth routing and uniform full-depth execution make different promises about where the variability sits: one puts it in the capital plan, the other in the operating tail. A comparison that only measures the mean therefore misses the entire reason a team would choose one over the other in large-model inference scheduling.

There is a quiet accounting asymmetry here. per-token depth routing moves spending into capital and spreads it over the life of the installation; uniform full-depth execution keeps spending in operations where it is easy to approve and hard to cap. Finance teams prefer the second, engineering teams prefer the first, and the argument is usually settled by the budgeting process rather than by measurement in large-model inference scheduling.

Part Two: The arithmetic that decides it

Suppose a 32-layer model routes half the tokens through half the layers. Arithmetic falls by roughly 25 percent, but the useful throughput gain is smaller. A router must run before each routed layer; its top-k selection produces indices; the activations must be gathered into a compact buffer and scattered back. Those memory operations are bandwidth-bound, not compute-bound, and on current accelerators a byte moved costs more than an unused multiply. In a serving stack where the model is already memory-bandwidth limited, cutting FLOPs can leave the wall clock almost unchanged while adding latency variance.

Deployment also changes the measurement. After per-token depth routing goes in, the metric people optimise shifts from throughput to predictability, and after uniform full-depth execution it shifts from predictability to unit price. That shift is why two teams can look at the same production data and reach opposite conclusions about which option won.

Part Three: Where the simple model breaks

The harder problem is the batch. If tokens in the same request take different paths, the batch no longer has a single shape, and the scheduler has to group tokens by capacity bucket or pad the shorter paths. Grouping adds a synchronisation point; padding gives back the arithmetic you just saved. This is why the reported gain is largest in offline, throughput-oriented workloads with large batches and much smaller in interactive serving with small batches and a tight tail-latency target. The same model can be a clear win in one deployment and a net loss in the other.

Ragged per-token paths turn a dense matrix multiply into a router, a gather and a scatter, and those three steps can consume the entire 25 percent arithmetic saving.

Part Four: The cost that arrives later

There is also a quality accounting issue that does not show up in a benchmark. A router trained to minimise a language-modelling loss will learn to spend depth on tokens that are locally surprising, which is not the same as tokens that matter for the final answer. Early tokens that set up a long chain of reasoning can look easy and get skipped, while punctuation after a rare proper noun can look hard and get full depth. Any evaluation has to be end-to-end on the downstream task, because per-token loss is a poor proxy for how the routing decision propagates through the rest of the sequence.

Adoption will not be a single flip. Early movers tend to be the cases with the most extreme value of the deciding variable, then the middle of the market follows once tooling and reference designs exist, and the long tail stays with uniform full-depth execution until the cost of switching falls below the cost of an exception. A reasonable reading of the next few years is coexistence with a slowly moving boundary rather than a replacement event.

Part Five: What this changes outside the engineering team

For anyone running a service, the everyday consequence is billing predictability. Uniform execution gives a nearly linear relationship between tokens served and accelerator-seconds, which makes capacity planning and per-request cost accounting simple. Ragged execution breaks that linearity: two requests with the same token count can consume noticeably different resources, which complicates rate limits, quotas and the price quoted to a customer. That is a product decision as much as an engineering one, and it tends to be discovered after the model is deployed rather than before.

A fair trial measures end-to-end tokens per second at a fixed tail-latency target, with router and gather costs included, and checks downstream task quality rather than language-modelling loss. Anything less will overstate the win by a wide margin on interactive traffic.

A market sizing sanity check

If inference continues to dominate accelerator demand, a conservative estimate puts the serving-side optimisation market, including routing runtimes, schedulers and the compilers that make ragged graphs efficient, in the range of roughly $2B to $6B annually by 2030. The lower bound assumes optimisation stays bundled into a few large frameworks; the upper bound assumes a distinct layer of independent serving infrastructure emerges. The driver is not model size but the share of total cost that is memory traffic and tail-latency headroom, both of which grow as batch sizes shrink for interactive use.

Treat the numbers as a scale model. They are built to show where the curves cross and which term dominates, and they deliberately round the inputs that vary most between deployments. The ranking is robust to those rounds; the exact threshold is not.

The comparison that matters

The comparison that matters is between arithmetic saved and irregularity introduced. Per-token routing wins in large-batch, throughput-bound settings where bandwidth is not already saturated; uniform execution wins in small-batch interactive serving where predictability is worth more than FLOPs. The same checkpoint can legitimately be deployed both ways.

A change in regulation or in the supply of a single component could invalidate the ranking overnight. The ranking is a snapshot of prices and constraints in large-model inference scheduling, not a property of the technologies, and anyone reusing it in a different year should re-run the threshold rather than reuse the conclusion.

Judge depth routing by tokens per second at a fixed tail-latency target and by downstream task quality, never by per-token loss alone.

SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.