Summary
The term 'foundation model' (Bommasani et al., Stanford CRFM, 2021) captures the shift: models trained once on broad data, adapted to many downstream tasks. GPT-3, CLIP, and later Llama, Claude, and Gemini are foundation models. The term is contested; the shift is not.
Objectives
- 01Define foundation model precisely.
- 02Explain what changed about the software supply chain.
- 03Name three criticisms of the term.
The Lesson
The shift
Previously: train a model per task, per dataset. Now: train one model on trillions of tokens, adapt via prompting or fine-tuning for thousands of tasks. The economics inverted — pretraining became the capital cost; adaptation became cheap.
Criticisms
The term obscures how much task-specific data still matters, concentrates power in labs with pretraining budgets, and imports biases from web-scale data into every downstream use.
Key Ideas
- One pretraining, many adaptations — the shift is economic as much as technical.
- Foundation models concentrate capability and risk in a handful of labs.