Summary
The transition from research artifact to product. Raw pretrained models predict likely continuations; they do not follow instructions. Instruction tuning (SFT on instruction-following data) and RLHF (learning from human preferences) turn a language model into something useful. ChatGPT is the paradigmatic result.
Objectives
- 01Distinguish pretraining, SFT, and RLHF.
- 02Explain why RLHF is separate from SFT.
- 03State the InstructGPT result.
The Lesson
The pipeline
Pretrain on trillions of tokens (base model). SFT on curated instruction-following examples (instruct model). Train a reward model on human comparisons of responses. Optimize the model against the reward with PPO or DPO. Ship.
Why RLHF matters
SFT alone cannot easily encode 'do not be harmful' or 'follow the spirit of the instruction.' A learned reward model can, because humans compare outputs on precisely those dimensions.
Key Ideas
- Base models are predictors; product models are aligned predictors.
- RLHF is the stage where product judgment enters the weights.
References
- Ouyang et al., 'Training Language Models to Follow Instructions with Human Feedback,' NeurIPS (2022)