Summary
Kaplan et al. (2020) showed that language-model loss decreases smoothly and predictably with compute, data, and parameters — a power law over more than five orders of magnitude. Scaling laws let labs plan multi-year compute budgets against expected capability gains.
Objectives
- 01State the Kaplan scaling relationship qualitatively.
- 02Explain how scaling laws changed research strategy.
- 03Distinguish the Kaplan and Chinchilla findings.
The Lesson
The empirical finding
Loss = A / N^α + B / D^β + C, roughly, where N is parameters and D is data. The exponents are small and the law holds across orders of magnitude.
What it enabled
Labs could commit to $100M training runs with quantified expectations. It also drove a decade-defining strategy: scale first, invent later.
Key Ideas
- Loss falls as a power law in compute, data, and parameters.
- Scaling laws made 'just scale it' a defensible strategy.
References
- Kaplan et al., 'Scaling Laws for Neural Language Models' (2020)