Summary
The empirical picture in 2026: pretraining loss still scales predictably, but frontier gains increasingly come from post-training (RLHF, DPO, reasoning RL), tool use, and inference-time compute. Pure pretraining scale has diminishing returns; the frontier is now multi-axis.
Objectives
- 01State how the scaling picture has changed since 2020.
- 02Name the three current scaling axes.
- 03Explain why 'just scale it' is no longer sufficient strategy.
The Lesson
What still holds
Loss vs. compute is still a power law over many orders of magnitude. Kaplan's core result was correct.
What changed
Loss is not capability. On downstream tasks, post-training and inference-time compute produce gains that pretraining scale alone would not. o1/o3, Claude with extended thinking, and DeepSeek R1 all demonstrate this.
Key Ideas
- Pretraining scale is necessary but no longer sufficient.
- The frontier is now pretraining × post-training × inference-time compute.