Summary
Top-k sampling keeps only the k highest-probability tokens and renormalizes. The oldest constrained sampler, largely superseded by top-p, but still occasionally useful when the vocabulary distribution shape is known.
Objectives
- 01Define top-k sampling.
- 02State the k values typically used in production.
- 03Explain top-k's failure mode versus top-p.
The Lesson
The sampler
Rank all tokens by probability, keep the top k, set the rest to zero, renormalize, sample. k=40 is the historical default from early GPT-2 sampling. Modern systems typically prefer top-p.
The failure mode
Fixed k is wrong when the distribution is peaked (should have used smaller k) or flat (should have used larger k). Top-p adapts to distribution shape automatically.
Key Ideas
- Top-k = keep k highest-probability tokens.
- Top-p usually beats top-k because it adapts to distribution shape.