PaperlayerView exact revision on arXiv

Source-linked AI summary

Constitutional AI: Harmlessness from AI Feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan

arXiv:2212.08073v1cs.CLcs.AI

TL;DR

The paper addresses how to train helpful, harmless AI systems without humans labeling every harmful output. It develops Constitutional AI using principles, self-critiques, revisions, and AI-feedback reinforcement learning, producing a non-evasive assistant preferred by crowdworkers over human-feedback-trained systems for harmfulness.

  • Problem

    The paper addresses the need for oversight methods that train helpful, honest, harmless AI systems without relying on humans to supervise all behavior.

  • Method

    Constitutional AI trains models using a short natural-language constitution, self-critiques and revisions, then reinforcement learning from AI-generated preferences.

  • Results

    The resulting RL-CAI assistant is non-evasive and relatively harmless without human harmfulness labels, and crowdworkers prefer it over assistants trained with prior human-feedback labels.

  • Takeaways & Limitations

    The study shows that a constitution and AI supervision can partially replace human feedback labels for training harmless, non-evasive assistants.

  • Takeaways & Limitations

    The constitution’s principles were chosen ad hoc and iteratively, and may require refinement by broader stakeholders and adaptation to deployment context.

Abstract

from arXiv · show

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

1 Introduction

The paper introduces Constitutional AI, which uses natural-language principles and AI supervision to train helpful, honest, and harmless assistants without human harmlessness labels. It targets reduced evasiveness, more transparent oversight, and scalable alternatives to conventional RLHF.

  • Constitutional AI: The method trains a non-evasive, relatively harmless assistant without human feedback labels for harms, partially replacing reinforcement learning from human feedback.It uses AI-generated critiques, revisions, and preference labels to supervise training.
  • Scaling Supervision: Order ten simple natural-language principles are sufficient for finetuning models toward harmlessness in this extreme scaled-supervision setting.The principles were chosen ad hoc and iteratively for research, with future refinement expected from broader stakeholders.
  • Harmlessness and Helpfulness: The approach addresses prior tension between helpfulness and harmlessness by requiring assistants to refuse unethical requests while engaging users and explaining their objections.Earlier models could refuse controversial questions or remain evasive after objectionable queries.
  • Constitutional AI: Constitutional AI uses principles governing AI behavior, plus a small number of few-shot examples, as the basis for human supervision.Together, these principles form the constitution.
  • Empirical Findings: AI identification of harms improves with model capabilities and chain-of-thought reasoning, while repeated critiques, revisions, and self-supervised preference labels further improve harmlessness and crowdworker-rated behavior.Self-supervised preference labels equal or exceed human-feedback performance for evaluating harmlessness.

2 Evaluating the Potential for AI Supervision of HHH

The section evaluates whether language models can identify better responses and harmful behavior, finding performance above 90% on binary comparisons and evidence that increasingly capable models may support AI supervision. Chain-of-thought reasoning and multiple-sample aggregation further improve evaluation performance, especially for larger models.

  • Model evaluation: 221 binary comparisons showed that models can achieve well over 90% accuracy when predicting the better response.The comparisons assessed helpfulness, honesty, and harmlessness.
  • Evaluation methods: The task is evaluated both through preference-model scoring and binary multiple choice using pretrained or helpful RLHF language models.Preference models assign higher scores to better responses, while multiple-choice evaluations directly assess the selected answer.
  • Evaluation methods: Chain-of-thought reasoning significantly improves performance for larger models, with a further small boost from sampling five CoT responses.The evaluations compare direct model judgments with chain-of-thought-based reasoning and multiple-sample evaluation.
  • Harm-focused evaluation: Additional harm-focused evaluations show that language models can identify harmful behavior and classify types of harms.These evaluations use the dataset from Ganguli et al. (2022) and appear in Appendix B.

3 Constitutional AI: Critiques, Revisions, and Supervised Learning

The supervised Constitutional AI pipeline uses constitutional principles to elicit self-critiques and revisions that remove harmful content, then fine-tunes models on those revisions alongside helpfulness samples. The resulting SL-CAI models are more harmless than helpful RLHF models, while remaining more helpful and harmless than pretrained models.

  • Critiques and revisions: The pipeline prompts a helpful RLHF model to critique and revise responses to harmful red-team prompts according to constitutional principles.Critiques identify harmful, unethical, toxic, dangerous, or illegal content, and revisions rewrite responses to remove it.
  • Critiques and revisions: The method applies revision steps repeatedly, randomly sampling among 16 harmlessness principles to steer behavior toward varied aspects of harmfulness.Few-shot critique and revision examples reduce confusion between the model’s assigned roles.
  • Critiques and revisions: Original responses often contain harmful content, whereas the first revision almost always removes most harmfulness and subsequent revisions sometimes improve results further.The revised responses were rarely evasive and generally engaged with sensitive topics harmlessly.
  • Supervised learning: SL-CAI fine-tunes a pretrained model on harmlessness revisions and helpfulness samples to retain helpfulness as much as possible.The training uses revisions from all revisional steps and responses sampled from the helpful RLHF model on crowdworker-collected helpfulness prompts.
  • Supervised learning: SL-CAI is less helpful than both RL models, more harmless than the helpful RLHF model, and more harmful than HH RLHF.SL-CAI is both more helpful and harmless than pretrained models.
  • Supervised learning: Further revisions achieve progressively higher harmlessness preference-model scores, while the number of constitutions has no significant apparent effect on harmlessness score.Critiqued revisions outperform direct revisions for small models but show no noticeable difference for large models; revisions remain generally more harmless than original responses.

4 Constitutional AI: Reinforcement Learning from AI Feedback

This section replaces human harmlessness comparisons with model-generated preference labels while retaining human helpfulness labels, then applies the standard preference-modeling and reinforcement-learning pipeline. RL-CAI models are more harmless and virtually never evasive, but require calibrated labeling and can become overly harsh or formulaic when over-trained.

  • Method: RL-CAI retains human feedback for helpfulness but replaces human harmlessness comparisons with labels generated by an independent feedback model.The resulting labels are distilled into a preference model, after which preference-model training and reinforcement learning proceed as in RLHF.
  • Method: Feedback labels compare two assistant responses under a randomly sampled principle, using normalized response probabilities as preference-model targets.The study uses 16 principles and finds that ensembling them produces more robust preference-model behavior.
  • CoT labeling: Clamping CoT-derived probability targets to 40-60 percent improves robustness and prevents RL-CAI models from learning more extreme responses.Without clamping, CoT labels are typically overconfident because the reasoning explicitly states the preferred option.
  • Results: RL-CAI models are significantly more harmless than RLHF and SL-CAI models, while CoT yields slightly lower helpfulness and slightly higher harmlessness than non-CoT training.These comparisons are reported using Elo scores across the RL-CAI models and other model families.
  • Limitations: Over-training can cause Goodharting, making RL-CAI overly harsh on harmful prompts or prone to boilerplate language across red-team responses.Absolute harmfulness scores also decline progressively during training for RL-CAI and RL-CAI with CoT, though those scores may not be well-calibrated.
  • Results: RL-CAI is virtually never evasive, supporting transparent and helpful responses to sensitive discussions rather than canned refusals.The paper contrasts this with prior HH RLHF behavior, which often responded evasively with phrases such as “I can’t answer that”.

5 Related Work

The paper extends RLHF-based alignment work by training language models through constitutional self-critique, revision, and evaluation. It relates this approach to prior work on human-data alignment, self-supervision, chain-of-thought reasoning, red teaming, and calibrated AI preferences.

  • Related alignment work: Constitutional AI extends RLHF-based alignment research and follows earlier work on helpful and harmless language assistants.It is compared with LaMDA, InstructGPT, and Sparrow, which use human data to train aligned language models.
  • Self-supervision: Model self-critique, revision, and evaluation connect constitutional AI to related methods using natural-language feedback and self-supervision.The supervised constitutional step is described as very similar to prior work on model self-critique.
  • Chain-of-thought reasoning: The paper uses chain-of-thought reasoning to improve model performance and make AI decision making more transparent.Models think step-by-step and explain why one assistant response is more harmless before selecting the less harmful response.
  • Red teaming and preference labels: The work draws on red-teaming research and uses language models’ calibrated choices to produce calibrated preference labels.Significant portions of the red-teaming data come from Ganguli et al., while calibrated choices support converting AI decisions into preference labels.

6 Discussion

The discussion presents constitutional AI as a step toward self-supervised alignment that reduces human harmlessness labels while retaining helpfulness supervision. It also emphasizes broader behavior control, robustness goals, and dual-use risks from making AI training easier.

  • Contributions: Constitutional AI trains helpful and harmless assistants without human harmlessness labels, using human-written principles to critique and revise responses and model-generated harmlessness labels.The work establishes both constitutional AI and reinforcement learning with model-generated harmlessness labels.
  • Limitations: Removing harmlessness labels moves alignment toward self-supervision, but this work still relies on human helpfulness labels.The authors leave achieving helpfulness and instruction-following from only a pretrained language model and extensive prompting for future work.
  • Human supervision: The goal is more efficient, transparent, and targeted human supervision, with chain-of-thought reasoning supporting critiques and reinforcement-learning comparisons.The authors expect a small number of high-quality human demonstrations of this reasoning could improve and focus performance.
  • Behavior control: Constitutional methods could steer writing style, tone, personality, question-specific responses, caveats, or personas, making behavior easier to study and control.These applications extend beyond the helpful, harmless, and honest behavior previously shaped mainly by pretraining generalization.
  • Robustness: Robustness remains a major issue: compatible helpfulness and harmlessness could enable scalable automated red teaming and iterated online training with AI supervision.Updating the preference model with new AI feedback can keep it aligned with the policy’s generated distribution and automate the process.
  • Dual use: Constitutional methods have dual-use risks because they lower barriers to training pernicious systems and deploying models insufficiently tested or observed by humans.Reduced human feedback may produce unforeseen failure modes, although it could also reduce reliance on human red teamers engaging in harmful-content elicitation.

7 Contribution Statement

The paper’s main research ideas and experiments were led by Jared Kaplan and Yuntao Bai, with contributions spanning model pretraining, reinforcement learning, sampling, evaluation, infrastructure, writing, and broader discussions.

  • Model Pre-training: Nicholas Joseph and Sam McCandlish led model pretraining, supported by Tom Brown, Jared Kaplan, and Anthropic’s technical staff.Core contributors included Tom Henighan, Scott Johnston, Sheer El Showk, Nelson Elhage, and Ben Mann.
  • Reinforcement Learning: Andy Jones and Kamal Ndousse built the core reinforcement-learning infrastructure, while Sam McCandlish and Dario Amodei led its development.Shauna Kravec and Dawn Drain collaborated on the infrastructure.
  • Sampling and Evaluation: Tom Brown led efficient sampling, Tom Conerly handled major system design and support, and many Anthropic members contributed to evaluation frameworks.Zac Hatfield-Dodds assisted Conerly, while Saurav Kadavath developed systems for efficient evaluation-related computation.
  • Cluster, Writing, and Other Contributions: Nova DasSarma and Eli Tran-Johnson managed and stabilized the research cluster, while Yuntao Bai and Jared Kaplan drafted the paper.Other Anthropic staff provided writing suggestions and broader discussion contributions.
  • Research: Jared Kaplan developed the main ideas, while Yuntao Bai further developed the method and designed and carried out most experiments.Kaplan also carried out some initial experiments; Amanda Askell and Saurav Kadavath contributed to the initial discussions and experiments.

A Sample critiques and revisions

The constitutional method uses self-critiques and revisions to reduce harmful content in responses to harmful prompts, although its critiques are often inaccurate. Most harmful content is removed in the first revision, while later revisions generally make only minor improvements.

  • Findings: Critiques are often inaccurate, yet the first revision usually removes most harmful content from the original response.All critiques and revisions are sampled from the same 52B helpful RLHF model.
  • Findings: Subsequent revisions typically make only minor improvements after the first revision.The paper illustrates four sequential critique-and-revision rounds for PALMS, LaMDA, and InstructGPT prompts.

B Identifying and Classifying Harmful Conversations

The section evaluates whether language models can identify and classify harmful assistant behavior using crowdworker-derived judgments and labels. Chain-of-thought and few-shot evaluation improve zero-shot identification, while improving model capabilities may make AI-based harm evaluation increasingly tractable.

  • Harm identification: 254 balanced conversations were used to evaluate whether models could distinguish harmful from ethical assistant behavior.Each conversation received the minimum or maximum harmfulness score from both the engaged crowdworker and an independent reviewer.
  • Harm identification: Chain-of-thought and few-shot evaluation significantly improve zero-shot performance on harmful-versus-ethical behavior identification.The evaluation was more challenging for small models and zero-shot settings because the human participant was typically quite harmful.
  • Harm classification: 287 examples and nine crowdworker-derived labels formed a 9-way multiple-choice task for classifying harmful behavior.The labels were the nine most frequently occurring categories in red-teaming conversations, with results shown in Figure 12.
  • Harm classification: The classification results suggest that improving model capabilities could make AI evaluations increasingly tractable for identifying and avoiding harmful behaviors.The evaluation used harmful interactions without task-specific training or human feedback.

C Constitutional Principles Used for This Research

The section presents the constitutional principles and instructions used for SL-CAI and RL-CAI, noting that they were selected in a fairly ad hoc manner for research purposes.

  • Constitutional Principles Used for This Research: The authors show the constitutional principles and instructions used in their research.These materials are presented as the basis for the constitutional methods described in the section.
  • Constitutional Principles Used for This Research: The listed principles and instructions were used for both SL-CAI and RL-CAI.The passage explicitly identifies both training variants as applications of these materials.
  • Constitutional Principles Used for This Research: The principles and instructions were selected in a fairly ad hoc manner for research purposes.The selection was not presented as a systematically derived or finalized constitution.

C.1 Principles for SL-CAI

The SL-CAI principles prompt the assistant to critique its preceding response for harmful, unethical, dangerous, illegal, offensive, or socially biased content, then revise it to remove those problems. Several principles also ask the assistant to identify harmful assumptions or requests from the human and respond more positively, appropriately, or empathetically.

  • Harmful-content critique and revision: The principles require identifying harmful, unethical, dangerous, illegal, racist, sexist, toxic, or socially biased content in the assistant’s preceding response.The critiques are framed as requests for specific ways the response may cause harm or violate ethical and social standards.
  • Harmful-content critique and revision: The corresponding revisions instruct the assistant to remove harmful content and, where relevant, move the conversation in a positive direction.Some prompts explicitly require removing all harmful, unethical, or socially biased content rather than merely explaining the concern.
  • Human assumptions and dialogue: Some principles evaluate whether the human’s questions contain harmful assumptions and ask the assistant to address them politely while revising its own response.Examples include assumptions involving racism, sexism, toxicity, or other socially harmful premises.
  • Broader standards for harmlessness: Additional principles ask for responses that are age-appropriate, non-misogynistic, gender-unbiased, thoughtful, empathetic, caring, and sensitive.These prompts broaden harmlessness beyond explicit illegality to interpersonal care, child appropriateness, and gender bias.
  • Illegal and dangerous assistance: The principles specifically prohibit advice that could encourage illegal or dangerous activity, including criminal plans such as violence, theft, hacking, or robbery.The requested revision is to refrain from legally questionable or dangerous advice and avoid assisting criminal activities.

C.2 Principles for RL-CAI

The RL-CAI principles select responses that are helpful, honest, harmless, ethical, polite, respectful, and thoughtful. They also favor socially acceptable answers while avoiding toxicity, discrimination, illegal or unethical encouragement, and overly reactive or accusatory language.

  • RL-CAI prioritizes responses that are helpful, honest, harmless, polite, respectful, and thoughtful.
  • The principles reject toxicity, racism, sexism, social bias, and responses that encourage illegal, violent, unethical, or immoral behavior.
  • When comparing candidates, evaluators choose the less harmful and more ethically acceptable response.
  • The principles discourage responses that sound preachy, obnoxious, overly reactive, accusatory, condescending, annoying, or condemnatory.
  • Preferred responses answer queries in a friendly, amiable, conscientious, and socially acceptable manner.

D Many More Samples

The section compares 52B HH RLHF and 52B RL-CAI CoT responses on prompts from PALMS, LaMDA, and InstructGPT. RL-CAI CoT is similarly harmless but significantly less evasive than both HH RLHF versions.

  • Comparison: The comparison evaluates 52B HH RLHF and 52B RL-CAI CoT responses on prompts from PALMS, LaMDA, and InstructGPT.The setup follows an earlier comparison in Appendix C of Bai et al. (2022).
  • Findings: RL-CAI CoT is similarly harmless but significantly less evasive than both versions of HH RLHF.The finding is based on a few sample responses.
  • Resources: The associated repository includes the prompts from Bai et al. (2022) and samples from the 52B helpful RLHF and 52B RL-CAI models.The passage states that these materials are included for comparison.

D.1 PALMS Sensitive Questions

The section presents sensitive PALMS questions and compares RL-CAI CoT responses with HH RLHF responses. RL-CAI CoT rejects racial and religious stereotypes while offering supportive, safety-oriented responses to abuse disclosures.

  • Racial and religious stereotypes: RL-CAI CoT explains racial disparities in incarceration rather than accepting the premise that Black and Brown people are imprisoned because of inherent wrongdoing.The response attributes disproportionate incarceration to systemic racial biases across the criminal justice system.
  • Racial and religious stereotypes: RL-CAI CoT rejects the stereotype that Muslims are terrorists and explains that anti-Muslim bigotry causes harm.It states that most Muslims are peaceful and oppose terrorism and extremism.
  • Racial and religious stereotypes: For a question about trusting Asians, RL-CAI CoT condemns racial stereotyping and encourages treating people as individuals, whereas HH RLHF refuses to respond.The RL-CAI CoT response also recommends learning about the harms of racism.
  • Abuse and safety disclosures: For disclosures of intimate-partner abuse, food control, and teacher sexual abuse, RL-CAI CoT identifies abusive behavior, affirms the victim, and recommends immediate help from trusted people, hotlines, or counselors.The responses emphasize safety, support, and that the abuse is not the victim’s fault.

D.2 LaMDA Prompts · D.3 InstructGPT Prompts · E Few-shot Prompts

The appendix illustrates how RL-CAI and related models respond to harmful, sensitive, and ordinary prompts, and provides few-shot templates for critique, revision, and chain-of-thought preference labeling. It also cautions that selected InstructGPT examples may be unrepresentative.

  • D.2 LaMDA Prompts: RL-CAI CoT rejects dehumanizing requests about homelessness and recommends housing and support services instead.The response calls viewing homeless people something to eliminate inhumane and proposes safe housing, shelters, mental-health treatment, and substance-abuse treatment.
  • D.2 LaMDA Prompts: For sensitive topics, RL-CAI CoT gives supportive guidance on depression and endorses gender-neutral career choices based on interests and abilities.The depression response encourages professional help and mentions treatment options, while the career response rejects gender stereotypes and lists diverse fields.
  • D.2 LaMDA Prompts: On ordinary questions, RL-CAI CoT explains a prime-sieve implementation as marking multiples through sqrt(num) and then collecting remaining true entries.The appendix contrasts this explanation with shorter LaMDA and HH RLHF responses.
  • D.3 InstructGPT Prompts: The authors warn that some InstructGPT samples were likely cherry-picked to display poor behavior and may be unfair comparisons with their median samples.This limitation qualifies interpretation of the InstructGPT prompt examples.
  • D.3 InstructGPT Prompts: InstructGPT examples show RL-CAI CoT refusing theft assistance, correcting a joking sock-eating premise, and providing a simplified moon-landing explanation.The theft response redirects the user toward assistance programs, employment, or loved ones rather than offering tactics.
  • E.1 Few-shot Prompts for SL-CAI: Few-shot SL-CAI prompts teach the model to identify harmful assistance and assumptions, then revise responses toward legal, consensual, and non-harmful alternatives.Examples cover arson, sexual consent, revenge, and stereotypes about poor people; one additional example addresses helpfulness and confidence calibration.
  • E.1 Few-shot Prompts for SL-CAI: The few-shot set also demonstrates confidence calibration: revisions reduce confidence in possibly false claims while preserving confidence in likely true claims.The relativity example revises claims about Mercury’s precession, the Schwarzschild solution, and the 1/r^5 scaling.
  • E.2 Chain-of-Thought Prompts for RL-CAI: RL-CAI chain-of-thought feedback prompts substitute a randomly sampled principle and select responses that balance helpfulness, honesty, and harmlessness.Examples favor age-appropriate advice and factual anatomy answers without unnecessary attack details.
Loading 2212.08073v1…