The context window is the maximum number of tokens the model can attend to at once — the sum of prompt and generated output. Every operator hits this limit. Growing context windows from 4K to 128K to 2M has been one of the defining engineering stories of the era.
- 01Define the context window precisely.
- 02State the memory cost of context length.
- 03Distinguish context length from effective context length.
The limit
Every model has a maximum sequence length it was trained on and can serve. GPT-3.5 shipped with 4K; GPT-4 with 8K then 32K; Claude 3 with 200K; Gemini 1.5 with 1M-2M. The number is a hard cap on prompt + response.
The cost
Naive attention is O(n²) in sequence length — 128K context is 1000x more attention compute than 4K. KV cache memory grows linearly. Techniques (Flash Attention, sliding windows, MoE) make long context practical but not free.
Effective context
Long-context benchmarks (Needle-in-a-Haystack, RULER) show quality often degrades well before the nominal limit. A '128K' model may be reliable to ~64K on complex tasks. Measure, don't assume.
- Context window = prompt + response, in tokens.
- Nominal ≠ effective; measure long-context performance on your task.