Summary
Sampling is how the model chooses the next token from a distribution over the vocabulary. Greedy sampling always picks the highest-probability token; stochastic sampling injects controlled randomness. The choice determines whether outputs are deterministic, diverse, or creative.
Objectives
- 01Distinguish greedy from stochastic sampling.
- 02Explain why greedy sampling can loop.
- 03State the default sampling approach used in production.
The Lesson
The choice
After the model outputs a distribution over ~100K possible next tokens, a sampling strategy picks one. Greedy = argmax. Random = draw from the distribution. Constrained samplers (top-k, top-p, temperature-scaled) sit between the two.
Why not always greedy
Greedy sampling is deterministic but tends to loop and produce dull, repetitive text. Some randomness is usually necessary for quality — the challenge is controlling how much.
Key Ideas
- Sampling turns a distribution into a token.
- Determinism (greedy) trades diversity for repeatability.