DPO, IPO, KTO, ORPO, GRPO — what each fixes, when each wins, and a decision tree for choosing the right one for your use case.
PPO RLHF requires three training stages: SFT, reward model training, and RL fine-tuning. Each stage introduces a new failure mode. The reward model is the most fragile piece: it must generalise well beyond its training distribution, and even small calibration errors are amplified by the RL optimisation loop.
Rafailov et al. (2023) — Direct Preference Optimisation: Your Language Model Is Secretly a Reward Model — showed you can mathematically re-derive the optimal RLHF policy without ever training a separate reward model. The key insight is that under the Bradley-Terry preference model, the optimal policy implicitly encodes the reward function — and you can optimise directly for the optimal policy using a classification-like loss on the preference data.
1. Train SFT model πSFT
2. Collect preference pairs (yw, yl)
3. Train reward model rφ
4. PPO: optimise πθ with rφ + KL(πθ || πSFT)
Four model copies in VRAM simultaneously
1. Train SFT model πSFT
2. Collect preference pairs (yw, yl)
3. DPO: optimise πθ directly from pairs
Two model copies: πθ (training) + πref (frozen)
On standard benchmarks (MT-Bench, AlpacaEval 2.0) DPO matches or slightly trails well-tuned PPO on single-turn instruction following. PPO retains an advantage on tasks requiring online exploration or non-differentiable rewards (code execution, math). For the vast majority of chat-style alignment work, DPO is the simpler and cheaper choice.
Starting from the RLHF objective — maximise expected reward minus KL divergence from the reference policy:
The normalisation constant Z(x) = ∑y πref(y|x) exp(r(x,y)/β) is intractable to compute — but when you subtract the implicit reward of the winner from the implicit reward of the loser (as the Bradley-Terry model does), Z(x) cancels exactly. This is why DPO works without ever computing or sampling from the partition function.
The DPO training objective is a binary cross-entropy over the log-ratio of policy probabilities relative to the reference, applied to pairwise preference data:
from trl import DPOTrainer, DPOConfig
training_args = DPOConfig(
beta=0.1,
loss_type="sigmoid", # "sigmoid" (DPO), "ipo", "kto_pair", "robust"
label_smoothing=0.0, # >0 reduces overconfidence on easy pairs
max_length=2048,
max_prompt_length=512,
per_device_train_batch_size=4,
gradient_accumulation_steps=8,
learning_rate=5e-7, # DPO needs lower LR than SFT; 1e-6–5e-7
)
# dataset must have columns: "prompt", "chosen", "rejected"
trainer = DPOTrainer(
model=model,
ref_model=ref_model, # or None to use the model's own frozen copy
args=training_args,
train_dataset=dataset,
)Set label_smoothing=0.1 to reduce overconfidence on clear-cut pairs. Enable length_normalization=True in DPOConfig (≥0.12) to normalise log-probabilities by sequence length — this prevents the loss from being dominated by long completions and is particularly important when your chosen/rejected pairs have very different lengths.
Azar et al. (2023) — A General Theoretical Paradigm to Understand Learning from Human Feedback — identified a subtle failure mode in DPO: the sigmoid in the DPO loss saturates when the margin between chosen and rejected log-probabilities becomes very large. Once the model is very confident in a preference direction, the gradient vanishes and learning stalls.
IPO (Identity Preference Optimisation) replaces the sigmoid with an identity link (squared loss), preventing saturation:
| Property | DPO | IPO |
|---|---|---|
| Loss shape | Logistic (sigmoid) | Squared (MSE-like) |
| Gradient at high margin | Vanishes (saturation) | Never saturates |
| Hyperparameter sensitivity | Moderate | More sensitive to β |
| Use when | Standard preference pairs, moderate-difficulty cases | Long training, hard/easy pair imbalance, Pareto-diverse preference data |
| TRL loss_type | "sigmoid" | "ipo" |
Ethayarajh et al. (2023) — KTO: Model Alignment as Prospect-Theoretic Optimisation — observed that pairwise preference data is expensive to collect. In practice, annotation platforms produce far more desirability labels ("this response is good/bad") than pairwise comparisons. KTO exploits this.
KTO models human utility using Kahneman-Tversky Prospect Theory: humans are loss-averse — they feel the pain of a bad output more than the pleasure of an equivalently good one. The loss function encodes this asymmetry:
# For a desirable response y_w (label = 1):
# Maximise: v(r(x, y_w) - z_ref) where v is the utility of gains
# For an undesirable response y_l (label = 0):
# Minimise: -v(z_ref - r(x, y_l)) where v is the utility of losses
# v is the value function (concave gains, convex losses):
# v(x) = log(1 + exp(x)) for gains
# v(x) = -lambda * log(1 + exp(-x)) for losses (lambda > 1 encodes loss aversion)
# z_ref = KL(pi_theta || pi_ref) acts as the reference baselineHong et al. (2024) — ORPO: Monolithic Preference Optimisation without Reference Model — eliminates the reference model entirely by incorporating the preference objective directly into the SFT loss. Instead of training SFT first then running DPO, ORPO does both in a single pass.
| Property | DPO | ORPO |
|---|---|---|
| Stages | 2 (SFT then DPO) | 1 (combined) |
| Reference model | Required (πSFT) | Not needed |
| VRAM | 2 model copies | 1 model copy |
| Data format | Pairs (chosen, rejected) | Pairs (chosen, rejected) |
| Training from scratch? | Requires SFT first | Can train from base model directly |
| Quality vs DPO | Slight edge with pre-trained SFT | Competitive; best when compute budget for 2-stage is unavailable |
ORPO is particularly attractive when you're starting from a base model (no instruction-tuned checkpoint available), when your GPU budget can't afford two 7B model copies, or when you want to compress the entire alignment pipeline into a single training run. The λ hyperparameter trades off SFT quality vs preference learning; start at 0.1 and tune.
DeepSeek-AI (2025) — DeepSeek-R1: Incentivising Reasoning Capability in LLMs via Reinforcement Learning — introduced GRPO (Group Relative Policy Optimisation) as a simplified PPO variant that eliminates the separate value network (critic), replacing it with a group-level baseline.
For each prompt x, generate G completions {y1, …, yG} using the current policy. Score all G with a reward function. The advantage of each completion is:
DeepSeek-R1 used two reward signals: accuracy reward (0 or 1 based on whether the final answer matched ground truth for math/code), and format reward (positive for correct use of <think>...</think> chain-of-thought markers, negative for empty or malformed thinking sections). No neural reward model was used — demonstrating that with a verifiable task, GRPO + rule-based rewards can achieve reasoning capability surpassing earlier RLHF-trained models.
The algorithm landscape can be confusing. This decision tree covers the majority of real-world use cases.
For most teams with a standard instruction-following alignment task: start with DPO using TRL's DPOTrainer, β=0.1, 1–3 epochs, learning rate 5e-7. If you observe the model ignoring the preference signal (log-ratio near zero throughout training), switch to IPO. If you're compute-constrained or fine-tuning from a base model, try ORPO. Save GRPO for tasks where you have verifiable ground truth.
Deck 05 covers Constitutional AI and RLAIF — Anthropic's approach to replacing expensive human preference labels with an AI model following a written constitution of principles. This is the technique behind Claude's training and the Harmless/Helpful/Honest balance.