Constitutional AI

Constitutional AI (CAI) is an Anthropic training method introduced in late 2022 that replaces most of the human labeling used in Reinforcement Learning from Human Feedback with model-generated critiques guided by an explicit written list of principles — a 'constitution'. It was originally developed to train the Claude family of assistants. CAI has two main phases. In the supervised phase, a helpful-only model generates responses, then critiques and revises its own outputs against constitutional principles such as 'do not produce harmful, unethical, racist, sexist, toxic, dangerous, or illegal content' and 'be helpful, honest, and harmless'. The revised responses are used as supervised fine-tuning data. In the reinforcement phase — called RLAIF, reinforcement learning from AI feedback — a model evaluates pairs of responses against the constitution, and those AI-generated preferences are used to train a reward model that drives the usual policy optimization step. The motivation was both practical and normative. Practically, soliciting harmlessness labels from human raters is slow, expensive, and exposes raters to disturbing content. Normatively, an explicit written constitution makes the intended values legible and auditable, in contrast to RLHF, where the values are implicit in aggregated rater preferences and prone to importing biases like a preference for agreeable answers, contributing to Sycophancy in LLM Responses. CAI does not fully eliminate sycophancy or other alignment failures — the constitution must still be authored and interpreted by a model whose judgments can be miscalibrated — but it provides a more controllable surface for adjusting behavior than raw preference data. Variants and extensions, including 'collective constitutional AI' that incorporates public input into the principles, have since been explored.