Calibration‑free top‑k MoE selector saves tuning

MoE decoding now runs at full speed without any offline threshold tuning. The new top‑k channel selector discards calibrated thresholds entirely, yet keeps the same accuracy and latency advantages that experts have come to expect.

Previously, practitioners limited routed‑expert reads by calibrating a score threshold on held‑out data; Prox + thresholds were the de facto recipe for interactive decoding at batch size one. This calibration step was required to hit a target read budget but introduced variability across inputs and even caused repetition loops in models such as Gemma‑4.

Top‑k selection removes the need for any calibrated threshold. “Second, top‑ keeps each token’s highest-scoring channels and needs no calibration data.” [1] By simply retaining each token’s k highest‑scoring channels, the selector selects a consistent number of channels per token without requiring an external tuning phase.

Across a wide range of benchmarks, the calibration‑free top‑k version matches or exceeds its threshold counterpart. “At the calibrated budget, a selector’s top‑ version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks.” [1] The superiority holds for both classification‑style and generative workloads.

The accompanying reader preserves almost full model performance while slashing weight reads. “Across three post‑trained MoE models, our reader retains 98.1% of full‑model accuracy at 17.8% of routed‑expert reads, with Triton kernels running MoE blocks 1.60 to 1.90 faster than optimized vLLM at batch one and up to 2.69 at larger batches on the same three models.” [1] The ternary sketch plus low‑rank recovery map delivers exact top‑k channels directly from BF16 weights.

The paper’s evaluation is limited to three post‑trained MoE checkpoints and relies on Triton kernels, leaving open whether the same gains hold for training‑time experts or alternative backends. Moreover, the ternary sketch introduces a small compute overhead that could dominate in ultra‑low‑latency settings. This suggests further work is needed to confirm calibration‑free selection across broader model families and hardware stacks.

If the results generalize, deployment pipelines can drop per‑model threshold calibration entirely and adopt the provided vLLM‑compatible reader as the default inference path for MoE models. The simplification cuts CI/CD latency, reduces engineering effort, and makes MoE acceleration a plug‑and‑play component.

References

1. Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data