KV caching stores the key and value tensors for every previously-processed token so they need not be recomputed at each decode step. Without KV caching, generating a 1000-token response would require re-attending to the full prompt 1000 times. With it, cost is linear in output length.
- 01Explain what KV caching stores.
- 02Compute the memory footprint of a KV cache for a given model.
- 03State how KV caching enables affordable serving.
The optimization
During decode, attention needs the Q for the new token and the K,V for all previous tokens. Since K and V for previous tokens don't change, cache them. Each new token requires only its own K,V computation and one attention pass over the cache.
The footprint
KV cache size = 2 × layers × heads × head_dim × seq_len × bytes. For Llama-3-70B (80 layers, 64 heads, 128 dim) at 8K context in fp16: ~2.6 GB per request. Multi-tenant serving requires paged attention (vLLM) or similar to manage KV cache memory efficiently.
Why 128K contexts strain hardware
For Llama-3-70B, KV cache at 128K context is roughly 40 GB — larger than the model weights themselves in some quantizations. Long-context serving is a memory problem before it is a compute problem.
- KV caching turns quadratic-in-length decode into linear.
- Cache memory often exceeds weight memory at long context.