Every transformer layer contains a feed-forward network (FFN) that processes each position independently after attention. It is roughly two-thirds of the parameters and much of the compute. The FFN is where much of the model's stored knowledge lives.
- 01State the FFN's role and shape.
- 02Explain why FFN width is usually 4x the model dimension.
- 03Describe the mixture-of-experts variant.
The layer
Two linear layers with a nonlinearity between: FFN(x) = W2 · σ(W1 · x + b1) + b2. σ is typically GeLU, SwiGLU, or ReLU. Hidden dimension is usually 4x the residual stream dimension.
Knowledge storage
Mechanistic-interpretability work (e.g., transformer-circuits.pub) locates much of what the model 'knows' — facts, associations, template completions — in FFN weights. Attention moves information; FFNs store it.
MoE variant
Mixture-of-experts replaces a dense FFN with many smaller FFNs and a router that selects a few per token. Same total parameters, fraction of the compute. Mixtral, DeepSeek-V3, and modern frontier models use MoE FFNs.
- FFN = position-wise MLP; where facts are stored.
- MoE keeps parameters, reduces per-token compute.