Attention is order-invariant by construction — 'dog bites man' and 'man bites dog' would produce the same output. Positional encoding injects order back into the representation. It is a small trick with large consequences, and the specific scheme (sinusoidal, learned, RoPE, ALiBi) determines context-length behavior.
- 01State why attention needs positional information.
- 02Describe the original sinusoidal encoding.
- 03Name one limitation of learned absolute encodings.
The need
Self-attention computes weighted sums over all positions. Without positional signals, the model cannot distinguish word order. The 2017 transformer added a fixed sinusoidal encoding to each token's embedding to break the symmetry.
Absolute vs. relative
Absolute encodings tag each position with a signature. Relative encodings (T5-style, RoPE, ALiBi) encode the distance between positions. Relative schemes generalize better to sequence lengths not seen in training.
- Without positional encoding, attention is a bag of tokens.
- Relative encodings generalize to longer sequences than they were trained on.