Summary
Inference is autoregressive: the model predicts one token, appends it to the input, predicts the next, and repeats until an end-of-sequence token or a length limit. Every LLM output — every ChatGPT response, every code completion — is generated one token at a time in this loop.
Objectives
- 01State the autoregressive generation loop in three steps.
- 02Explain why generation is inherently sequential.
- 03Distinguish prefill from decode.
The Lesson
The loop
1. Compute logits over the vocabulary from the current input. 2. Sample the next token from the (possibly modified) distribution. 3. Append the token to the input and repeat until stop.
Prefill vs. decode
Prefill processes the input prompt in parallel — fast, throughput-bound. Decode generates output tokens one at a time — slow, latency-bound. Cost structures and latency SLOs differ radically between the two phases.
Key Ideas
- Generation is one token per forward pass.
- Prefill is parallel and cheap; decode is sequential and expensive.