Why Continuous Batching Made My LLM Inference Engine Slower

I had been adding optimizations to mini-vllm-rs, my Rust LLM inference engine for macOS, including prefix caching and speculative decoding. Continuous batching looked like the next easy win. Process requests together, reuse model weights across more tokens, and watch throughput climb. What could go wrong?

I tested two concurrent requests on my M2 Mac mini with 16 GB of unified memory, running a 4-bit quantized Qwen2.5 7B model. Each request generated 512 tokens, with speculative decoding disabled.

Guess what? Boom. Aggregate throughput dropped from 16.8 tokens/s unbatched to 8.7 tokens/s batched, almost cut in half.

Fixing that slowdown led to another surprise: the same kernel-selection threshold could also change generated text, even with greedy decoding.

Two rows, a different kernel

The clue was in Candle, the framework I use for backend execution. Its quantized Metal path contained this tiny decision:

// Candle 0.11.0: candle-core/src/quantized/metal.rs, line 372
if src_shape.dim(D::Minus2)? == 1 {
    return self.fwd_mv(self_shape, storage, layout);
}
// Line 414 calls into call_quantized_matmul_mm_t

Source. The condition checks the number of input rows. One row takes the GEMV path, multiplying the weights by one vector. More than one row continues into GEMM, which handles multiple input vectors together.

Packing two requests changed the input from one row to two. That was enough to switch kernels.

GEMM can reuse weights across rows, but this implementation's overhead outweighed that benefit at such a small batch. My scheduler was doing what I wanted. The backend was choosing a slower way to execute it.

Getting the speed back

I added a fallback: split small inputs into rows, run GEMV for each, then concatenate the results.

At a batch size of two, the fallback brought throughput back to 16.25 tokens/s, compared with an unbatched baseline of 16.15 tokens/s. Relief! But would GEMV still win with larger batches?

Aggregate output throughput in tokens/s.

At five requests, the ratio fell below 100%, so GEMM pulled ahead. Separate dispatches and concatenation may have become more expensive than one GEMM call. I chose four rows as the default switching threshold for this workload. See the benchmark report for details.

Problem solved? Not quite.

Same prompt, different story

Later, I tested speculative decoding with two requests, using Qwen2.5 0.5B to propose four tokens at a time. Their overall draft acceptance rate was 32.9% when processed one at a time, versus 37.1% when allowed to run concurrently. Same requests. Same models. Same proposal policy. Greedy decoding in both runs. Why?

A separate check made the numerical issue easier to see. I asked Qwen to write a science-fiction scene about a mechanic and an archivist investigating a distress signal in an abandoned greenhouse on the Moon. Both runs also used greedy decoding. I changed the threshold so that prefill used GEMM in one run and row-wise GEMV in the other.

The first 597 characters matched exactly. Then the continuations split:

Both responses were coherent and followed the same characters and setting. Nothing looked like a completely broken model. That made small numerical differences a plausible suspect.

Tracing the speculative test gave me stronger evidence. At the first differing token, the drafter proposed the same token IDs in both runs, but two leading target scores had swapped order:

That tiny change was enough to flip the greedy choice. Every later token then followed a different prefix, which could change draft acceptance too.

The other request had already finished

Here was the part that caught me off guard: when the token choice flipped, the other request was already gone.

So why did its earlier presence still matter?

Here, rows count token positions, not requests. An earlier batch had combined 120 prefill tokens from the new request with four verification tokens from the existing request. Those 124 rows crossed my threshold and selected GEMM instead of GEMV, introducing small numerical differences into the cached state.

By the time the output diverged, the other request had finished, and the remaining request's four-token verification was back on the GEMV path. But switching back did not undo the differences already stored in its KV cache.

Looking only at the iteration where the text diverged would have missed the cause.

I forced all relevant inputs through GEMV in both execution modes. In the shortened comparison, output hashes and acceptance totals matched. This did not establish which kernel was more accurate, but it supported the explanation. The numerical report contains the investigation.

A batching decision changed the kernel selection, and its effects survived in cached state long after the batch changed again.

My longer article follows how a generation loop grew into a serving engine with clear boundaries between scheduling, model execution, and request state. It also explores how AI helped with architecture, coding, and performance optimization, and where I had to challenge its decisions.