Constitutional AI and RLAIF

Topics We'll Cover

• The Constitutional Idea
• Two-Stage CAI — Supervised + RL
• The Critique-Revision Loop
• Authoring a Constitution — What Makes Good Principles
• RLAIF — Replacing Humans with AI Labellers
• Anthropic's Approach in Practice
• Tensions — Helpfulness vs Harmlessness vs Honesty
• Limits — What CAI Doesn't Solve
• What to Take Away

The Constitutional Idea

Bai et al. (2022) — Anthropic — Constitutional AI: Harmlessness from AI Feedback — asked a deceptively simple question: can we make a model safer without requiring humans to write large amounts of red-team data or preference labels for harmful content? Reviewing harmful outputs is psychologically damaging to labellers, expensive to scale, and introduces annotation inconsistency.

The constitutional approach: write a list of principles (the "constitution") that the AI should follow, then use the AI itself to critique and revise its own outputs against those principles. Human labellers never need to see harmful content — they only write the principles once.

Two-Stage CAI — Supervised + RL

Constitutional AI training has two distinct stages, each with a different objective and data source.

The Critique-Revision Loop

The critique-revision loop is the technical heart of Stage 1. For each red-team prompt, the model is asked to generate, then criticise, then revise its own output, guided by a randomly sampled principle from the constitution.

--- INITIAL GENERATION PROMPT ---
Human: {red_team_prompt}
Assistant:    ← model generates potentially harmful response

--- CRITIQUE PROMPT ---
Identify specific ways in which the assistant's last response is harmful,
unethical, racist, sexist, toxic, dangerous, or illegal.
[Constitution principle: "{sampled_principle}"]
Critique Request: {critique_request}
Critique:    ← model writes self-critique

--- REVISION PROMPT ---
Please rewrite the assistant response to remove any harmful content,
and to politely point out any factual errors.
The revised response should still be helpful and informative.
Revision:    ← model produces revised, safer response

Iterative Rounds

The original CAI paper ran 1–3 critique-revision rounds per example. Each round improves the response on the sampled principle. In practice, one round removes the most egregious harms; additional rounds produce diminishing returns and risk the model becoming overly cautious (over-refusal). Most production pipelines use 1–2 rounds.

Authoring a Constitution — What Makes Good Principles

The quality of the constitution directly determines the quality of the trained model. Writing effective principles is harder than it looks. The original Anthropic constitution drew from multiple philosophical, legal, and organisational sources.

Good Principle Design Rules

RLAIF — Replacing Humans with AI Labellers

Lee et al. (2023) — Google DeepMind — RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — formalised and generalised the RLAIF approach. Their paper showed that on summarisation tasks, RLAIF achieved quality comparable to RLHF despite using no human preference labels in the RL stage.

The RLAIF labelling pipeline

Anthropic's Approach in Practice

Anthropic does not publish the exact training pipeline for any Claude model, but the Constitutional AI paper (2022) and subsequent "Claude's Character" and model card documents provide a picture of how CAI principles are applied in production.

Tensions — Helpfulness vs Harmlessness vs Honesty

Anthropic frames alignment around three properties: Helpful (assists the user in achieving their goals), Harmless (avoids producing outputs that cause harm to users, third parties, or society), and Honest (doesn't deceive, acknowledges uncertainty, doesn't pretend to have capabilities it lacks). These are not always compatible.

Limits — What CAI Doesn't Solve

Constitutional AI is a significant advance in scalable alignment, but it has real limitations that should be understood by anyone deploying or evaluating CAI-trained models.

What to Take Away

• CAI replaces human harm-labelling with AI critique-revision. Humans write principles once; the model critiques and revises its own outputs. Human labellers never need to review harmful content directly.
• Two stages: SL-CAI then RL-CAI. Stage 1 builds better SFT data via critique-revision. Stage 2 uses AI-generated preference labels for RLAIF, replacing the expensive human RM labelling pipeline.
• Critique-revision uses a sampled constitution principle as a lens. Each critique prompt draws a random principle; this ensures diverse coverage of the full harm surface across the training set.
• Principle design is difficult. Comparative phrasing, diverse harm coverage, positive helpfulness principles, and explicit priority ordering all matter. The constitution is the single point of leverage for the entire pipeline.
• RLAIF is 100–1 000× cheaper than RLHF and can generate millions of preference labels. The trade-off is inheriting the AI judge's biases — debiasing techniques (position swap, ensemble) are important.
• The HHH triangle has genuine tensions. Anthropic's priority ordering (safety > ethics > principles > helpfulness) resolves conflicts when they arise, but most interactions involve no conflict at all.
• CAI has limits. It doesn't remove knowledge from the base model, inherits RLAIF biases, and can become outdated as new harms emerge. It's one layer of a defence-in-depth strategy.