Mixture of Experts: How Sparse Routing Scales Transformers
You have probably seen a claim shaped like this on a model card: "70B total parameters, but only 13B active per token." That is not marketing rounding — it describes a real architecture change inside the transformer. Instead of one dense feed-forward block that every token pushes through in full, the model holds many smaller feed-forward networks called experts, and a small router decides, token by token, which handful of them actually run. Most of the model's weights sit idle for any single token; they only matter in aggregate, across the whole stream of tokens the model ever processes.
This guide assumes you already know how attention and dense feed-forward layers work — see Transformer Architecture and Self-Attention Explained if you need that first. Here we go one layer deeper: how a Mixture of Experts (MoE) layer replaces a single dense feed-forward block, how the router picks which experts run, why that trades total parameter count for compute cost instead of just cutting parameters the way quantization or distillation do, and the failure modes — expert imbalance, routing instability, and the communication cost of spreading experts across GPUs — that make MoE genuinely harder to operate than it looks on a slide.
Dense Layers vs Sparse Expert Layers
In a standard transformer block, the feed-forward sub-layer is dense: one shared network takes every token's hidden vector and pushes it through the same weights, every time. If that block has 200M parameters, every token's forward pass touches all 200M of them. Add capacity by making the feed-forward network wider, and you also make every token's pass more expensive — capacity and compute are locked together.
A Mixture of Experts (MoE) layer breaks that link. It replaces the single dense feed-forward network with several smaller networks called experts — say, 8 of them, each roughly the size of one dense feed-forward block — plus a small router (also called a gating network) that looks at each token's hidden vector and decides which experts should process it. With top-2 routing, only 2 of the 8 experts run for any given token; the other 6 sit unused for that token. Run the numbers: total parameters grow to roughly 8× one expert plus the router, but the compute for any single token stays close to 2× one expert — not 8×. For the rest of this guide, 8 experts, top-2 routing is the running example.
Quick reference
- Dense feed-forward layer: total parameters = active parameters, for every token, every time.
- MoE layer: total parameters = router + (number of experts × expert size); active parameters per token = router + (k × expert size).
- Attention layers and the router itself usually stay dense — MoE typically replaces only the feed-forward sub-layer, not the whole block.
- A model isn't required to make every layer sparse; many architectures interleave dense and MoE feed-forward layers.
k(experts per token) is a fixed architectural choice, usually 1 or 2 — raising it narrows the gap between active and total compute.- More experts at a fixed
kgrows model capacity (and memory) without growing the compute per token — that decoupling is the entire point of MoE.
Remember this
A dense layer spends every parameter on every token; an MoE layer spends the router's weights on every token but only k experts' weights — so total capacity scales with expert count while active compute scales with k, and those two numbers can now move independently.
Inside the Router: Scoring, Top-k, and Weighted Combine
The router is a small linear layer: it takes a token's hidden vector and produces one score per expert, then turns those scores into probabilities with a softmax. If there are 8 experts, the router for our example token — call it the vector for bank in "the central bank raised rates" — produces 8 numbers. Only the top-k highest-scoring experts are kept; the rest are set to zero. With top-2 routing, if Expert 3 scores 0.55 and Expert 7 scores 0.31, those are the only two that run. Each selected expert processes the full token vector independently and produces its own output; the layer's final output is the weighted sum of those two outputs, weighted by the router's renormalized probabilities.
It is tempting to picture Expert 3 as "the finance expert" and Expert 7 as "the syntax expert," as if routing sorted tokens by topic. Analyses of trained MoE routers usually find something messier: specialization tends to track shallow, token-level or positional patterns more than clean subject-matter boundaries, especially in early layers. Treat expert identity as a learned, opaque partition of the input space — useful for load and capacity math, not as a human-readable taxonomy.
Top-k selection is a hard, discrete operation, which is awkward for gradient descent — the network gets no gradient signal for why an unselected expert should have been picked instead. In practice the softmax probabilities double as the combine weights, so gradients keep flowing to whichever experts were chosen, and the router is trained end-to-end with the rest of the model. Some approaches add small noise to the router's logits before the top-k step (noisy top-k gating), purely to encourage exploration early in training, before the router has learned useful distinctions. The snippets below are illustrative pseudocode — swap in your framework's softmax/top_k helpers and this runs directly against real tensors.
Quick reference
- Router compute is tiny — one linear layer over (hidden size × number of experts) — compared to running even one expert's full feed-forward network.
- A token's routing decision is local to one MoE layer; the same token can route to different experts at layer 3 and layer 20.
- Batching means each expert sees a dynamically sized subset of the batch's tokens every step, not a fixed assignment.
- Router logits are usually kept at higher numeric precision than the rest of the model, even under quantization — small routing errors compound into large output differences.
- Renormalizing the top-k probabilities (dividing by their sum) keeps the combine weights on the same scale regardless of how confident the router was.
Remember this
The router — not the experts — is the mechanism that creates sparsity: it converts a per-token decision (which k experts run) into a smaller compute bill, while gradients still reach every expert often enough to keep training them.
Total Parameters vs Active Compute: Why the Split Exists
Two different numbers now describe the model: total parameters (every expert, everywhere, whether or not a given token uses it) and active parameters (router + attention + the k experts a token actually visits). Memory and storage scale with the total — you need enough combined GPU memory to hold every expert, because you don't know in advance which tokens will need which expert. Compute (FLOPs) per token scales with the active count. That split is exactly what a claim like "70B total parameters, 13B active per token" describes: a model with the storage footprint of a 70B dense model but closer to the inference compute of a 13B dense model, for each token.
That gap is not free. Serving an MoE model means every expert has to live somewhere in GPU memory — often sharded across multiple devices, a technique called expert parallelism — even though most experts are idle for most tokens. When a token's chosen expert lives on a different device than the token itself, the framework has to move that token's vector across the network (an all-to-all exchange) and bring the result back. At small batch sizes, or when experts are unevenly loaded, that communication and the underused GPUs eat into the compute savings the sparsity was supposed to buy. MoE lowers FLOPs per token; it does not lower the hardware footprint or the systems complexity of serving the model.
Quick reference
- Memory footprint tracks total parameters, not active ones — an MoE model with more total capacity needs more combined GPU memory even if inference is cheaper per token.
- Expert parallelism shards experts across devices; each MoE layer needs an all-to-all step to send tokens to the GPU holding their chosen expert and bring outputs back.
- Sparsity pays off best at larger batch sizes, where every expert reliably receives enough tokens to run efficiently; small or bursty batches under-utilize it.
- A per-expert capacity limit (max tokens per batch) is common — tokens that overflow a full expert get dropped or rerouted rather than stalling the batch.
- MoE and quantization or distillation solve different problems: MoE changes which weights run per token; quantization and distillation shrink or replace the weights themselves. They compose.
- If your workload doesn't need the broad capacity a large MoE model provides, a smaller dense model may beat it on cost and latency — see Foundation Models vs Small Language Models for that decision.
Remember this
MoE trades total parameter count (memory, storage) for active compute (FLOPs) per token — the saving is real, but it only shows up in practice when batching and expert parallelism keep every expert busy; otherwise communication and idle memory eat the budget.
Failure Story: Expert Collapse During Fine-Tuning
Trigger. A team fine-tunes an 8-expert, top-2 MoE checkpoint on a narrow domain — support transcripts, say — and drops the auxiliary load-balancing loss the base model was pretrained with, because the fine-tuning script only copied the main task loss. Symptom. Validation loss looks fine, even good, but inference throughput regresses compared to the base checkpoint, and when someone finally logs routing statistics, two of the eight experts are absorbing well over half the tokens while several others receive almost none.
Root mechanism. The router and experts train jointly, on the same gradients. If one expert gets a slightly larger share of tokens early — pure chance, or a head start from pretraining — it receives more gradient updates, gets marginally better at whatever tokens it sees, and the router's softmax nudges toward preferring it a little more next step. That is a rich-get-richer feedback loop: without a countervailing pressure, the router's easiest path is to collapse toward a handful of favored experts, which quietly turns a sparse model back into something close to a smaller dense one — at the storage cost of the original large one.
Diagnostic. Log a per-step or per-epoch histogram of tokens routed to each expert; a healthy router keeps that distribution roughly even given the workload, while a collapsed router shows a sharp skew toward one or two experts. Recovery. Reintroduce, or raise the coefficient on, the auxiliary load-balancing loss used in pretraining, and consider lowering the learning rate on the router relative to the experts so it stops compounding the imbalance while the balancing term corrects it. Prevention. Treat the load-balancing loss as part of the model's contract, not an optional extra — carry its coefficient into every fine-tuning run, and put expert-utilization histograms on the same dashboard as loss curves.
Quick reference
- The auxiliary load-balancing loss penalizes the product of (fraction of tokens sent to an expert) and (average router probability for that expert), pushing both toward uniform.
- Collapse is a training-dynamics problem, not a data problem — feeding a collapsed router more balanced data rarely undoes an existing feedback loop on its own.
- Routing instability can also appear as oscillation — tokens flipping between experts run to run — rather than outright collapse; a lower router learning rate helps both shapes.
- Per-expert capacity limits are a hard backstop: when a popular expert's token budget for a batch is full, overflow tokens are dropped or sent to a fallback rather than crashing the step.
- Idle experts are not free during collapse — expert parallelism still reserves their GPU memory and compute slots, so imbalance wastes real hardware, not just a metric on a chart.
- Watch this metric especially after checkpoint surgery — merging, pruning, or continuing pretraining on new data can all reset the balance the original training run had reached.
Remember this
Router and expert weights update from the same gradients, so a small early imbalance compounds into full collapse without an explicit load-balancing term — expert utilization belongs on the training dashboard next to loss, not as a debugging afterthought.
Key takeaway
Mixture of Experts does not make a transformer smarter by adding parameters; it changes which parameters a given token pays for. A router turns a per-token routing decision into the gap between total capacity and active compute — the mechanism behind every "70B total, 13B active" claim you'll see on a model card. The real engineering cost moves from raw FLOPs to memory footprint, cross-GPU communication, and keeping the router's load balanced, which is exactly where MoE deployments tend to fail in practice.
Practice (25 min). Paste the moe_layer_with_balance sketch from earlier into a Python file. Replace experts with 8 tiny functions (lambda x, i=i: x * (i + 1) for i in range(8)) and w_gate with a random 4×8 matrix. Feed it 200 random 4-dimensional token vectors with k=2 and print the histogram of top_k_idx across all 200 — expected result: roughly even counts, no expert far above the ~50-token average. Intentional break: multiply column 0 of w_gate by 5 before running, so expert 0 starts every route with an unfair head start, and skip computing balance_loss entirely. Recovery: compute balance_loss as shown and use it (scaled by a coefficient) to penalize the router weights that favor expert 0, then re-run. Pass criterion: after recovery, no single expert should receive more than roughly 2× the average token count across the 200 tokens — if one expert still dominates, the balancing coefficient is too small. For where MoE fits against the rest of the model landscape, see Layers of AI.
Related Articles
Explore this topic