DPO and its cousins

The efficiency improvements that displaced RLHF for many teams.

Summary

Direct Preference Optimization (Rafailov et al., 2023) is a simpler alternative to RLHF that skips the explicit reward model and RL step. Given preference pairs, DPO directly optimizes the policy to prefer the winner. It has largely displaced RLHF in the open-model ecosystem.

Objectives

01State how DPO differs from RLHF in workflow.
02Explain why DPO is easier to run.
03Name the situations where explicit RL still wins.

The Lesson

The insight

The RL step in RLHF can be rewritten as a supervised loss on preference pairs, using the base model as an implicit reward. This eliminates the reward model, the KL penalty tuning, and the RL infrastructure. Same data, simpler pipeline.

Key Ideas

DPO: preference pairs, no explicit reward model, no RL.
Simpler pipeline, comparable results in most settings.

← RLHF, in detail The Chinchilla result →