Summary
Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds p, then renormalizes and samples. It adapts to distribution shape and is the current default across production systems.
Objectives
- 01Define nucleus sampling.
- 02State typical p values.
- 03Explain why nucleus outperforms fixed top-k.
The Lesson
The sampler
Sort tokens by probability. Include tokens in order until cumulative probability reaches p (e.g., 0.9). Renormalize and sample from that set. When the distribution is peaked, few tokens qualify; when flat, many do.
The result
Nucleus sampling produces the most coherent stochastic outputs of the common samplers. p=0.9 is the standard chat default; p=0.95 is common for creative work; p=1.0 disables nucleus filtering.
Key Ideas
- Nucleus keeps enough tokens to hit cumulative probability p.
- Adaptive by construction; the modern default.
References
- Holtzman et al., 'The Curious Case of Neural Text Degeneration' (2020)