Mixture of Experts (MoE) is the dominant scaling strategy for modern large language models that want to increase parameter count without a linear increase in inference cost. Instead of routing every token through a single dense feed-forward network, an MoE layer spreads computation across many independent expert networks and uses a lightweight gating mechanism to activate only a subset of them for each input. The result is a model with hundreds of billions, or even trillions, of total parameters, but only a fraction of those parameters participate in any single forward pass.

The Core Mechanics of MoE

A standard transformer block contains a self-attention layer followed by a feed-forward network (FFN). In an MoE transformer, the FFN layer is replaced by an MoE layer consisting of N expert networks, each typically a standard FFN, and one gating network. For a given input token representation x, the gating network computes a probability distribution over the N experts. The top-k experts are selected, their outputs are weighted by the gate scores, and the results are summed. All other experts remain inactive for that token.

This sparse activation is what decouples total model capacity from per-token compute. A 671B parameter MoE model might only activate 37B parameters per token, delivering the representational benefits of a massive model at the throughput cost of a much smaller one. Oxlo.ai hosts several such models, including DeepSeek R1 671B MoE and GLM 5 744B MoE, which leverage this architecture for deep reasoning and long-horizon agentic tasks.

Routing and Load Balancing

The gating network is usually a learned linear projection followed by a softmax. To avoid collapse, where the gate always selects the same one or two experts, training incorporates an auxiliary load-balancing loss. This loss penalizes uneven routing distributions across experts within a batch. Modern implementations also apply noise to the gate logits before selection, which aids exploration early in training.

In production, routing is not just a training concern. Inference engines must batch tokens destined for the same expert to maximize GPU utilization. If one expert receives too many tokens, the excess may be dropped or spilled to a secondary queue, which introduces latency. Efficient MoE serving requires expert parallelism, where different experts reside on different GPUs, and all-to-all communication patterns that can become bottlenecked by interconnect bandwidth.

import torch

import torch.nn as nn

import torch.nn.functional as F

num_experts = 8

top_k = 2

hidden_dim = 4096


  
  
  Gating network: learns to route tokens to experts


gate = nn.Linear(hidden_dim, num_experts)


  
  
  Independent expert FFNs


experts = nn.ModuleList([

    nn.Linear(hidden_dim, hidden_dim) for _ in range(num_experts)

])

def moe_layer(x):

    # x shape: [batch, seq, hidden_dim]

    logits = gate(x)                      # [batch, seq, num_experts]

    weights, indices = torch.topk(

        F.softmax(logits, dim=-1), top_k, dim=-1

    )

    # Normalize top-k weights so they sum to 1

    weights = weights / weights.sum(dim=-1, keepdim=True)

output = torch.zeros_like(x)
for i in range(top_k):
    expert_idx = indices[..., i]
    expert_weight = weights[..., i : i + 1]
    # Dispatch to selected experts (simplified; real stacks use fused kernels)
    for e in range(num_exper