Hoffmann et al. (Chinchilla, 2022) showed that most large language models were undertrained: for a given compute budget, more data with fewer parameters would produce a stronger model. The result reset the field's scaling strategy and set the recipe for LLaMA and every subsequent open-weights release.
- 01State the Chinchilla scaling ratio.
- 02Explain what it means to be 'compute-optimal.'
- 03Describe how the field responded.
The finding
For compute-optimal training, model size and training tokens should scale roughly equally: doubling parameters requires doubling tokens. Chinchilla (70B params, 1.4T tokens) beat Gopher (280B, 300B tokens) at less than half the compute.
The response
LLaMA 1 (7B/13B/33B/65B trained on 1-1.4T tokens) applied Chinchilla directly. LLaMA 3 pushed further — 8B/70B trained on 15T tokens, well past 'compute-optimal' because inference cost dominates at deployment time.
- Compute-optimal ≈ equal scaling of parameters and tokens.
- Inference cost has since pushed real-world training past Chinchilla ratios.
- Hoffmann et al., 'Training Compute-Optimal Large Language Models' (2022)