Speculative decoding uses a small 'draft' model to predict several tokens ahead, then verifies them with one forward pass of the big model. Verified tokens are accepted; the first mismatch is corrected. Net: 2-4x speedup with identical output distribution.
- 01Explain the draft-and-verify pattern.
- 02State the correctness guarantee.
- 03Name one production system that uses it.
The pattern
1. Draft model generates k candidate next tokens autoregressively (fast). 2. Big model does one forward pass, computing what each of those k+1 positions should be. 3. Accept tokens that match; take the big model's token at the first mismatch. Net: multiple tokens per big-model pass.
The guarantee
The output distribution is provably identical to sampling from the big model alone. Not an approximation — a speedup with no quality loss. Used in production by Groq, vLLM, TGI, and every major inference provider.
- Small model drafts, big model verifies.
- Same output distribution, 2-4x faster.
- Leviathan et al., 'Fast Inference from Transformers via Speculative Decoding' (2023)