Reward modelling from pairwise preferences, the PPO actor–critic loop, KL penalty, failure modes, and the compute cost of running three models simultaneously.
SFT teaches the model to imitate its training data. If the training data contains helpful responses, the model learns to produce helpful-looking responses. But imitation has a structural ceiling: you can only be as good as your data labels, and human labellers are expensive, inconsistent, and unable to evaluate subtle quality differences between nearly-equal responses.
RLHF was described by OpenAI in the InstructGPT paper (2022), then refined in their early work on summarisation from human feedback (Stiennon et al., 2020). Anthropic extended it with Constitutional AI (2022). The core PPO loop has been largely stable since then — later work (DPO, GRPO) mostly replaces or simplifies it.
A reward model (RM) is a language model with its final next-token-prediction head replaced by a single scalar output layer. It takes a (prompt, completion) pair and outputs a scalar score indicating quality. Training data is pairwise comparisons: for each prompt, human raters see two completions and pick the preferred one.
Given a chosen completion yw ("winner") and a rejected completion yl ("loser") for prompt x, the Bradley-Terry model defines the probability that yw is preferred:
| Design choice | Options | Trade-offs |
|---|---|---|
| RM base model size | Same as policy or 2–3× smaller | Larger RM better calibrates fine-grained quality; smaller RM trains faster, fits on fewer GPUs |
| Preference collection | Human raters, LLM-as-judge (RLAIF) | Human: higher quality, expensive; LLM: cheap, scalable, but inherits model biases |
| Ties / equal pairs | Discard or model as 0.5 probability | Discarding is simpler; modelling ties reduces dataset waste by ~15 % |
| Multi-attribute RM | Single scalar vs separate helpfulness/safety heads | Separate heads allow Pareto-optimal trade-offs; single scalar is simpler to train and use |
Proximal Policy Optimisation (Schulman et al., 2017) is the RL algorithm at the heart of most RLHF implementations. In the LLM context it operates on entire completion sequences as single actions.
The total reward is rtotal = rRM(x,y) − β · KL(πθ || πref). Without the KL penalty the actor quickly discovers degenerate completions that fool the reward model — this is reward hacking. The reference policy πref is the SFT model, frozen throughout PPO. A typical starting value for β is 0.02–0.1; higher values keep the policy closer to SFT at the cost of reward-signal responsiveness.
PPO has more hyperparameters than SFT, and they interact. Here are the ones that matter most and sensible starting ranges for LLM RLHF.
| Hyperparameter | Symbol | Typical range | Effect of raising |
|---|---|---|---|
| KL penalty coefficient | β | 0.02–0.2 | More conservative; policy stays closer to SFT; slower reward improvement |
| PPO clip ratio | ε | 0.1–0.2 | Allows larger policy steps; can destabilise training if >0.3 |
| GAE lambda | λ | 0.9–0.99 | Higher: lower variance, higher bias in advantage estimates; 0.95 is a good default |
| Rollout batch size | — | 64–512 | Larger = more stable gradient estimates; must fit in VRAM during rollout phase |
| PPO epochs per rollout | K | 1–4 | More reuse of rollout data; risk of off-policy distribution shift beyond K=4 |
| Value loss coefficient | cv | 0.1–1.0 | Weights critic update relative to actor; 0.1 is typical for LLMs (critic is secondary) |
| Entropy bonus | ce | 0.0–0.01 | Encourages exploration; prevents premature mode collapse on the output distribution |
Rather than fixing β, set a target KL divergence (e.g. 0.05 nats/token) and adjust β up if KL exceeds target, down if KL is too low. This is implemented in TRL's PPOTrainer via target_kl and adaptive_kl_ctrl=True. The adaptive controller prevents both runaway divergence (high KL, reward hacking) and stagnation (near-zero KL, no learning).
PPO for LLMs has a richer failure-mode taxonomy than standard RL because the action space (natural language) is so vast and the reward signal (a neural network) is so gameable.
The policy finds completions that score highly on the reward model but are clearly bad to humans. Classic examples: repetitive praise (the RM was trained on helpful-sounding text → model learns to produce effusive compliments), length bias (if RM conflates length with quality, model outputs extremely long responses), format exploitation (bullet points score higher → model bullet-points everything).
Mitigations: KL penalty, stronger RM, diverse preference data, reward normalisation.
The policy collapses to a narrow set of safe, generic completions. The model learns "I can't help with that" or "That's a great question!" are safe high-reward responses and over-produces them.
Mitigations: entropy bonus in PPO objective, diversity reward, broad prompt distribution in rollouts, curriculum (easier prompts first).
A sudden large divergence from the reference policy, often after a few thousand steps. The policy jumps to a high-reward region but the reward model has poor coverage there, causing instability. Frequently correlated with learning rate spikes or gradient explosions.
Mitigations: gradient clipping (1.0), warm restart of β, smaller learning rate (≤1e-6 for 7B), monitor token-level KL per step.
"When a measure becomes a target, it ceases to be a good measure." The reward model is a proxy for human preferences; optimising it directly will eventually over-fit to the proxy's imperfections. The Overoptimisation paper (Gao et al., 2023) showed that beyond ~5–10 × initial reward gain, gold-label human quality typically starts declining even as RM score keeps rising.
Full RLHF with PPO requires four model copies loaded simultaneously: actor (training), critic (training), reference policy (frozen SFT), and reward model (frozen). In practice the actor and critic often share the backbone with separate output heads, reducing this to three distinct weight sets.
| Component | Precision | Training? | VRAM (7B base) |
|---|---|---|---|
| Actor (πθ) | bf16 + Adam fp32 | Yes | ~98 GB (full); ~10 GB (QLoRA) |
| Critic (Vφ) | bf16 + Adam fp32 | Yes (value head only) | ~56 GB base + 4 B value head (full); ~6 GB (QLoRA) |
| Reference policy (πref) | bf16 inference only | No | ~14 GB |
| Reward model | bf16 inference only | No | ~14 GB |
| Total (full FT, 7B) | ≈ 182 GB+ — requires 4× A100 80 GB | ||
| Total (QLoRA, 7B) | ≈ 44 GB — fits 2× A100 40 GB |
For 70B+ models, DeepSpeed ZeRO-3 is standard: shard actor and critic states across all GPUs; keep ref policy and RM on separate GPU sets or offload to CPU RAM during the rollout phase. The rollout phase (inference) and the update phase (training) have different memory profiles — some pipelines alternate device allocation between phases.
TRL's PPOTrainer accepts a deepspeed config dict. Set stage: 3 for actor/critic sharding. Ref policy and RM can be loaded in stage: 0 (replicated) if they fit, or stage: 3 if not. Set offload_optimizer_device: cpu to move Adam state to RAM, freeing ~48 GB VRAM for a 7B actor at the cost of ~1.5× slower optimiser steps.
TRL (Transformer Reinforcement Learning, HuggingFace) provides PPOTrainer that handles rollout collection, reward scoring, and PPO updates. The minimal working loop is straightforward:
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from transformers import AutoTokenizer
config = PPOConfig(
model_name="meta-llama/Meta-Llama-3-8B-Instruct",
learning_rate=1e-6,
batch_size=64,
mini_batch_size=16,
gradient_accumulation_steps=4,
ppo_epochs=4,
kl_penalty="kl", # "kl", "abs", "mse", "full"
init_kl_coef=0.05,
target_kl=0.05, # adaptive KL target
adap_kl_ctrl=True,
cliprange=0.2,
vf_coef=0.1,
)
model = AutoModelForCausalLMWithValueHead.from_pretrained(config.model_name)
ref_model = AutoModelForCausalLMWithValueHead.from_pretrained(config.model_name)
tokenizer = AutoTokenizer.from_pretrained(config.model_name)
trainer = PPOTrainer(config, model, ref_model, tokenizer)
for batch in dataloader:
queries = batch["input_ids"]
responses = trainer.generate(queries, max_new_tokens=256)
rewards = [reward_model(q, r) for q, r in zip(queries, responses)]
stats = trainer.step(queries, responses, rewards)
trainer.log_stats(stats, batch, rewards)This TRL class wraps a standard causal LM and adds a linear value head on top of the last hidden state. The value head outputs V(x) — the critic's estimate of expected future reward from prompt x. During PPO the actor loss and critic loss are computed jointly; the original LM backbone weights are shared between both objectives.
DPO (Deck 04) and its cousins have dramatically simplified preference-based training for many tasks. But PPO retains real advantages in specific settings.
DeepSeek-R1 (2025) used a variant of online RL (GRPO — covered in Deck 04) against a verifiable reward (math/code correctness) without a learned reward model at all. This is PPO's best use case: when you have a ground-truth reward signal, the reward model is the weakest link in the chain and replacing it with a verifier unlocks dramatic gains.
Deck 04 presents DPO and its cousins (IPO, KTO, ORPO, GRPO) — methods that achieve alignment quality comparable to PPO while eliminating the reward model, the rollout loop, or both. Deck 05 covers Constitutional AI and RLAIF, where the human labellers are replaced by a model following a written constitution.