Pretraining is the compute-dominant phase of LLM development. A large corpus of text is shown to a randomly-initialized model, which is trained to predict the next token. Frontier pretraining runs cost tens of millions to hundreds of millions of dollars and produce the base capabilities that all subsequent fine-tuning refines.
- 01State the pretraining objective in one sentence.
- 02Explain why next-token prediction produces general capabilities.
- 03State approximate compute costs for frontier pretraining.
The objective
Given a sequence of tokens, predict the next token. The loss is cross-entropy between the predicted distribution and the actual next token. That single objective, applied to trillions of tokens, produces every downstream capability we observe.
Why it generalizes
To predict the next token in arbitrary text, the model must implicitly learn grammar, facts, reasoning patterns, code semantics, and translation. The pretraining corpus contains examples of all of these; a good enough model absorbs all of them from the same objective.
- Next-token prediction is the entire pretraining objective.
- Generality emerges from the corpus, not from special-purpose loss functions.