31 Mixture-of-Experts
Context
Quality grows with the parameter count, but the compute per example grows along with it — expensive. Shazeer et al. break that link through conditional computation.
The idea and the mechanism
The layer holds many (thousands of) "experts" — small subnetworks. A learned gating picks only a few (top-k) of them for each token and mixes their outputs; the rest are never computed (sparse activation). So the TOTAL parameter count is enormous, while the compute per token is that of a small network.
optimization Sparse gating and load balancing
For a token x the gating gives a distribution over E experts; you take the top-k of it (say k=2), renormalize, and mix the outputs of the chosen ones only:
The collapse problem. Gating tends to send every token to a handful of "favourite" experts — the rest never learn, and the scale goes to waste. The cure is an auxiliary load-balancing loss that penalizes imbalance:
where fi is the fraction of tokens routed to expert i, and Pi is the mean gating probability for it. This product is at its minimum under uniform usage — so the router does not collapse onto a couple of experts.
PyTorch A top-k MoE layer (sketch)
import torch
def moe(x, gate, experts, k=2):
w, idx = gate(x).softmax(-1).topk(k, -1) # top-k experts + weights
w = w / w.sum(-1, keepdim=True) # renormalize
out = torch.zeros_like(x)
for j in range(k):
e = idx[:, j]
out += w[:, j:j+1] * run_expert(experts, e, x) # only the chosen ones
return outWhy it matters
The direct ancestor of the MoE frontier — Mixtral (#52), DeepSeek (#53) and, by rumour, GPT-4. The architecture already reached 137 billion parameters back in 2017. "Activate only the part of the network you need" is the key to cheap scaling: many parameters in total, few of them active.
Connections
Both are "many specialists instead of one". But a forest averages over all the trees, whereas MoE uses gating to activate only a few experts per input. Ensembling (everything at once) against conditional computation (only what is needed) — two different answers to "how do you combine many models".
Mixtral takes sparse MoE straight into the transformer: the FFN of every layer is replaced by 8 experts, and the router takes 2. A 2017 idea, matured into an open LLM with the quality of a large model at the price of a mid-sized one.
DeepSeek-V3 is a 671B-parameter MoE with ~37B active per token, plus engineering improvements to the routing (balancing without an auxiliary loss). A straight line from "Outrageously Large" in 2017 to the frontier models of 2025.
Questions worth asking
If few parameters are active, why is MoE called a "large" model — isn't that cheating?
Depends what you count. By capacity (the total parameters holding knowledge) the model really is enormous; by compute per token it is small. The trade: you save FLOPs but pay in memory (all the experts have to sit in VRAM) and in the complexity of routing and communication. Not cheating — just a different point on the quality / compute / memory curve.
Do the experts really specialize by meaning (syntax, topics)?
Less than one would like. Analyzes show the specialization is often not "human-readable" (not "the medicine expert") but rather along surface features — punctuation, particular tokens, position. Routing learns whatever lowers the loss, not an interpretable division of labour. The appealing "narrow specialists" intuition is blurry in practice.
Why is top-k > 1 needed — why not one expert per token?
At k=1 (as in Switch Transformer) the routing is non-differentiable with respect to the choice and training is less stable; k=2 gives a gradient through the mixing of two experts and trains more smoothly, at the cost of doubling the active compute. It is a balance between training stability and inference cost, and different models choose differently (Switch takes 1, Mixtral 2).
What to read in the original
Read the key parts — the gating mechanism and load balancing; this is the direct ancestor of MoE in every modern frontier model.