RoPE (Rotary Positional Embedding, Su et al. 2021) and ALiBi (Attention with Linear Biases, Press et al. 2022) are the two positional schemes behind nearly every modern LLM. RoPE rotates query and key vectors by position; ALiBi adds a distance-based bias to attention scores. Both handle long contexts better than the original sinusoidal approach.
- 01State how RoPE encodes position via rotation.
- 02State how ALiBi encodes position via bias.
- 03Explain why each generalizes to longer contexts.
RoPE
Rotate the query and key vectors by an angle proportional to their position before computing attention. The dot product then depends on relative position. LLaMA, Qwen, DeepSeek, and Mistral all use RoPE.
ALiBi
Subtract a linear penalty from attention scores proportional to distance. No embeddings needed. BLOOM and MPT use ALiBi. Simpler than RoPE; extrapolates cleanly to unseen lengths.
Context extension
RoPE supports position interpolation and NTK-aware scaling, which extend trained context lengths (2K → 32K → 128K) with modest quality loss. Every long-context model since 2023 relies on some variant.
- RoPE = rotation of Q and K; ALiBi = distance bias on scores.
- Both are the reason 128K+ context windows are practical.
- Su et al., 'RoFormer: Enhanced Transformer with Rotary Position Embedding' (2021)
- Press et al., 'Train Short, Test Long: Attention with Linear Biases' (2022)