Mixture-of-experts (MoE) replaces the dense feed-forward network with many smaller 'expert' FFNs and a router that selects a few per token. Same total parameters, fraction of the per-token compute. Every frontier model since 2024 uses MoE.
- 01State the MoE architecture in one sentence.
- 02Distinguish total parameters from active parameters.
- 03Name three MoE models.
The architecture
For each token, a router network selects k experts out of N (typically k=2, N=8-256). Only the selected experts compute. Total parameters (all experts combined) can be enormous; active parameters (what runs per token) can be small.
The tradeoff
MoE trains harder — routing instability, expert collapse, load-balancing losses are all engineering challenges. But at inference, the compute-per-token is much lower than a dense model of equivalent capability. Mixtral 8x7B, DeepSeek-V3 (671B total, 37B active), and Qwen MoE all demonstrate the recipe.
- MoE decouples parameters from compute.
- Active parameters, not total, drive inference cost.