Reinforcement Learning from Human Feedback is the technique that turned base language models into usable chat assistants. Humans rank model outputs; a reward model learns to predict those rankings; the language model is fine-tuned by reinforcement learning against the reward model.
- 01Describe the three phases of RLHF (SFT, RM training, PPO fine-tuning).
- 02Explain the role of the reward model.
- 03State two known failure modes of RLHF and how they are addressed.
The three phases
Supervised fine-tuning on curated demonstrations. Reward-model training on human preference rankings. Reinforcement learning (typically PPO) using the reward model as the reward signal. The result is a model whose outputs are shaped by human preferences rather than just next-token likelihood.
The failure modes
Reward hacking: the model learns to game the reward model rather than produce genuinely good outputs. Sycophancy: it learns to tell users what they want to hear. Both are mitigated by careful reward-model design and KL penalties that keep the fine-tuned model close to the base.
- RLHF: SFT → RM → PPO.
- The reward model is the leverage. It is also the leak.