Layer normalization stabilizes activations at each layer; residual connections let gradients flow through deep networks without vanishing. Neither is glamorous. Both are why training a 100-layer transformer is possible at all.
- 01State what layer normalization normalizes.
- 02State the residual-connection update rule.
- 03Explain pre-norm vs. post-norm and which won.
LayerNorm
Normalize each activation to zero mean and unit variance across the feature dimension, then apply learned scale and shift. This keeps activations well-conditioned regardless of layer depth.
Residuals
Each sublayer's output is x + Sublayer(x), not just Sublayer(x). The residual creates a direct gradient path from output back to input, sidestepping vanishing gradients in deep stacks. Every transformer block has two — one around attention, one around the FFN.
Pre-norm won
Original transformer used post-norm (norm after residual). Pre-norm (norm inside the residual) trains more stably at scale and is now standard. GPT-2 onwards is pre-norm.
- LayerNorm stabilizes; residuals preserve gradient flow.
- Pre-norm is the modern default.