Multi-head attention runs several attention operations in parallel, each with its own learned Q, K, V projections. The heads attend to different patterns; the outputs are concatenated and projected back. It is one of the transformer's most consequential design choices.
- 01State what changes between single-head and multi-head attention.
- 02Explain why d_model is typically divided across heads.
- 03Describe what different heads have been observed to learn.
The mechanism
Instead of one attention over d_model dimensions, run h attentions each over d_model/h dimensions. Concatenate the h outputs and apply a final linear projection. Total compute is unchanged; expressive capacity increases.
What heads learn
Interpretability research has found heads that track subject-verb agreement, heads that track coreference, heads that copy adjacent tokens, and many that resist clean interpretation. Circuit-level analysis (Anthropic's interpretability work) has traced specific behaviors to specific heads.
- Multi-head splits d_model across parallel attentions.
- Heads specialize in different linguistic patterns.