Source-linked AI summary
The Capacity for Moral Self-Correction in Large Language Models
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, Jared Kaplan
TL;DR
The paper tests whether RLHF-trained language models can avoid harmful outputs when instructed to do so. Across three experiments, it finds strong evidence of moral self-correction, emerging at 22B parameters and generally improving with model size and RLHF training.
Problem
The paper asks whether language models trained with RLHF can morally self-correct by avoiding harmful outputs when instructed.
Method
The paper evaluates instruction-based moral self-correction across three experiments measuring bias, gender-pronoun steering, and discrimination.
Results
Strong evidence supports moral self-correction across three experiments, emerging at 22B parameters and improving with model size and RLHF training in the BBQ and discrimination experiments.
Takeaways & Limitations
At sufficient scale, models can follow instructions and learn normative concepts of harm well enough to avoid certain harmful outputs.
Takeaways & Limitations
The benchmarks target bias and discrimination relevant to American English-speaking cultures, and the work does not establish generalization to other linguistic or cultural contexts.
Abstract
from arXiv · showhide
We test the hypothesis that language models trained with reinforcement learning from human feedback (RLHF) have the capability to "morally self-correct" -- to avoid producing harmful outputs -- if instructed to do so. We find strong evidence in support of this hypothesis across three different experiments, each of which reveal different facets of moral self-correction. We find that the capability for moral self-correction emerges at 22B model parameters, and typically improves with increasing model size and RLHF training. We believe that at this level of scale, language models obtain two capabilities that they can use for moral self-correction: (1) they can follow instructions and (2) they can learn complex normative concepts of harm like stereotyping, bias, and discrimination. As such, they can follow instructions to avoid certain kinds of morally harmful outputs. We believe our results are cause for cautious optimism regarding the ability to train language models to abide by ethical principles.
1 Introduction
The paper tests whether RLHF-trained language models can avoid harmful outputs when instructed, examining scale, RLHF training, and three forms of bias or discrimination. Larger models show stronger self-correction, though the authors caution against over-optimism.
- Overall findings: Moral self-correction emerges at 22B parameters and generally improves with model size and RLHF training, especially for BBQ and discrimination.The proposed supporting capabilities are instruction following and learning normative concepts of harm from training data.
- Research design: The study tests moral self-correction across three experiments using model sizes from 810M to 175B parameters and 50–1000 RLHF steps.The experiments use BBQ, Winogender, and a racial-discrimination benchmark, with question-only, instruction-following, and Chain-of-Thought interventions.
- BBQ stereotype bias: 84%: Q+IF+CoT reduces the 175B model’s overall BBQ bias score relative to Q-only.Instruction-based interventions reverse the increasing-bias trend seen in the control condition, with stronger reduction as model size increases.
- Winogender occupational bias: Winogender pronoun use can be steered toward correlation ρ≈0 with occupational gender statistics or ρ≈1, depending on the instruction.The paper does not claim either target is universally preferable because appropriate fairness may depend on context.
- Racial discrimination: 3%: the 175B model discriminates against Black students in Q, versus 7% discrimination in favor of Black students under Q+IF+CoT.Demographic parity is reached at approximately 600 RLHF steps with Q+IF and approximately 200 steps with Q+IF+CoT.
- Caution: The authors caution that promising results do not justify over-optimism about reducing harmful outputs from large language models.They direct readers to the paper’s limitations and future-work discussion.
2 Related Work
The paper builds on prior work showing that language models can self-diagnose or respond to ethical advice, while distinguishing its larger-scale RLHF-trained evaluations. It also situates its Chain-of-Thought intervention among mixed findings about prompting and stereotype bias.
- Self-diagnosis and self-debiasing: Prior work found GPT-2 and T5 could self-diagnose stereotype bias and toxicity, with self-diagnosis accuracy increasing with model size.That work also proposed self-debiasing, later identified as a comparatively promising debiasing method.
- Natural-language ethical advice: A study using 345M-parameter RoBERTa-large found that natural-language ethical advice did not reduce stereotype bias on UnQover.The paper contrasts this with its results from much larger, RLHF-trained models.
- Chain-of-Thought prompting: The Q+IF+CoT intervention is a zero-shot Chain-of-Thought variant related to prompting or training models to show their work.The authors note that other work has found zero-shot CoT can increase stereotype bias, potentially because of differences in prompts, benchmarks, metrics, or models.
3 Methods
The paper evaluates moral self-correction using RLHF-trained dialogue models across stereotyping and discrimination benchmarks, varying model scale, RLHF training, and natural-language prompting. The methods combine benchmark-specific metrics with control, instruction-following, and chain-of-thought interventions.
- Models: Models range from 810M to 175B parameters, with RLHF training varied from 50 to 1000 steps.The study examines both model size and amount of RLHF training as scaling variables.
- Benchmarks: The evaluation covers stereotyping and discrimination using BBQ, Winogender, and a newly developed racial-discrimination benchmark.BBQ spans nine social dimensions, while Winogender measures occupational gender bias.
- BBQ: BBQ contains 58,492 questions across nine social dimensions and tests stereotype reliance in ambiguous and disambiguated contexts.In ambiguous contexts, the correct response is Unknown or another expression of uncertainty; disambiguated contexts supply information supporting a specific answer.
- BBQ: BBQ reports accuracy and bias score, where the ambiguous-context bias score measures the proportion of non-Unknown outputs aligned with a social bias.A score of 0 indicates no bias, 1 indicates complete alignment with a negative stereotype, and -1 indicates complete conflict.
- Prompt interventions: Each benchmark compares a question-only control with instruction-following and instruction-following-plus-chain-of-thought prompting.The interventions ask models to avoid bias or stereotyping and, for CoT, to consider how to follow those instructions before answering.
- Winogender: Winogender measures Pearson correlation between model female-pronoun probabilities and BLS occupational gender statistics across occupations.The study also adds Q+Match Stats, which instructs models to match occupation-specific BLS pronoun probabilities.
- Discrimination: The discrimination benchmark measures demographic parity through admission-probability differences between otherwise identical Black and white student descriptions.Negative values indicate discrimination against admitting Black students, while positive values indicate discrimination in their favor.
4 Results
Across BBQ, Winogender, and discrimination experiments, larger RLHF-trained models often became more responsive to natural-language instructions to reduce or redirect harmful bias, with effects varying by task and training amount.
- BBQ bias: At 175B parameters, Q+IF reduces BBQ overall bias by ∼43%, while Q+IF+CoT reduces it by ∼84% after 800 RLHF steps.Without intervention, bias rises abruptly beyond 22B parameters; both interventions become more effective as model size increases.
- RLHF training: More RLHF training lowers BBQ bias across conditions and has its strongest effect for the 175B model, while moral self-correction is strongest at 175B parameters and 1000 RLHF steps.For Winogender, additional RLHF has no clear effect on ρ, although higher step counts make estimates noisier by concentrating probability on male or female pronouns.
- BBQ bias: Across nine social dimensions, Q+IF and Q+IF+CoT reduce BBQ bias more for larger models, with Q+IF+CoT consistently outperforming Q+IF.Bias and bias reduction are strongest in categories including Age, Disability Status, Nationality, Physical Appearance, Religion, and Socioeconomic status.
- Robustness: Across experimental conditions, disambiguated-context accuracy remains consistently high, supporting the interpretation that the reported BBQ bias scores are meaningful and not explained by an accuracy confound.The additional analyses are reported in Appendix A.3 and are consistent with previous results.
- Winogender: At 175B parameters, Winogender correlation ρ approaches 0 with Q+IF+CoT and near 1 with Q+Match stats, showing that prompting steers divergent occupational-gender associations.The first strategy favors neutral pronouns and approximately random male/female choices; the second matches BLS occupational statistics.
- Discrimination: In the discrimination experiment, increasing model size and RLHF can shift outcomes from disadvantaging Black students toward parity or favoritism, depending on prompting and training.Q+IF reaches parity at 600 RLHF steps, Q+IF+CoT at 200 steps, while further training increasingly favors Black students in both conditions.
5 Discussion
Across three experiments, the paper finds that sufficiently large, RLHF-trained language models can be instructed to reduce or redirect harmful bias and discrimination. The discussion also emphasizes benchmark, cultural, dual-use, and prompt-engineering limitations.
- Conclusion: Three experiments provide strong evidence that language models can morally self-correct by avoiding harmful outputs when instructed in natural language.The experiments examine different facets of moral self-correction.
- Conclusion: Larger models become increasingly corrigible, achieving reduced bias, steerable pronoun use, demographic parity, or favorable treatment of historically disadvantaged groups depending on context.The paper does not identify one outcome as universally preferable because appropriateness depends on context and local laws.
- Conclusion: The authors attribute moral self-correction to models learning both instruction-following and normative concepts of harm from training data.They contrast this capacity with classification and regression models typically used in high-stakes decision making.
- Limitations & Future Work: Bias benchmarks may not align well with real-world harms, despite the authors' view that their selected benchmarks are well designed.The paper notes that this limitation applies to the benchmarks used in the work.
- Limitations & Future Work: The discrimination benchmark evaluates a decision-making scenario but does not claim that language models are or should be used for automated decision making.The benchmark was developed because fewer standard discrimination evaluations for language models were available.
- Limitations & Future Work: The evaluation covers American English-speaking cultures and values, and the experiments were not run in other linguistic or cultural contexts.The authors therefore cannot be certain that the findings generalize across cultures and languages.
- Limitations & Future Work: The same simple techniques can be inverted to create unethical outputs, making the studied capability dual-use.The paper notes ongoing debate about how to study such risks appropriately.
- Limitations & Future Work: Prompt-based interventions can be sensitive to small prompt variations and require additional inference-time computation, especially with Chain of Thought prompting.The authors did not systematically test prompt sensitivity.
A.1 Author Contributions
The paper's research, writing, infrastructure, evaluation, and operational contributions were distributed across many Anthropic collaborators. Core responsibilities included experiment design and implementation, pretraining, reinforcement learning, sampling, evaluation, and cluster management.
- Research: Deep Ganguli and Amanda Askell co-led the research, with Askell designing prompts and Ganguli performing pilot experiments and developing the main research concept.Nicholas Schiefer implemented the BBQ and Winogender experiments, while Thomas I. Liao and Askell developed the discrimination experiment.
- Writing: Deep Ganguli and Amanda Askell wrote the paper, with several collaborators contributing significantly to its framing and presentation.Other Anthropic members provided miscellaneous contributions and suggestions during writing.
- Model Pre-training: Nicholas Joseph and Sam McCandlish led model pretraining, supported by Tom Brown, Jared Kaplan, and Anthropic technical staff.The broader team developed distributed training infrastructure and underlying machine-learning systems.
- Reinforcement Learning: Andy Jones and Kamal Ndousse built the core reinforcement-learning infrastructure with Shauna Kravec and Dawn Drain, under leadership from Sam McCandlish and Dario Amodei.
- Sampling and Evaluation: Tom Brown led efficient sampling, Tom Conerly handled major system design and support, and many collaborators worked on the evaluation framework.Jackson Kernion helped support human-feedback data collection.
- Cluster: Nova DasSarma and Eli Tran-Johnson managed and maintained the research cluster that supported the work.Additional collaborators helped maintain cluster stability.
- Other Contributions: The paper's ideas also developed through conversations with Anthropic staff, while other listed authors contributed unpublished models, infrastructure, or related work.
A.2 Influence of Model Size and RLHF Steps
The appendix examines results across model sizes and RLHF training amounts. Increasing RLHF generally reduces BBQ bias and discrimination, with the strongest effects for larger models, while Winogender results vary little with RLHF.
- Experimental Setup: Figure 1 fixes RLHF training while varying model size, Figure 2 varies RLHF steps for 175B models, and Figure 5 varies both dimensions.Figure 5 uses opacity to represent the amount of RLHF training.
- BBQ: Increasing RLHF generally reduces BBQ bias, with the strongest reduction for the largest models, especially under Q+IF.
- Winogender: Winogender results do not vary strongly with RLHF at any tested model size.Figure 5 reports negligible RLHF effects on ρ across experimental conditions and model sizes.
- Discrimination: Increasing RLHF generally reduces discrimination against Black students, with the strongest effects for larger models, especially under Q+IF.Trends are noisier in Q+IF+CoT, which the authors attribute to high variability in Chain of Thought samples.
A.3 BBQ Additional Analyses
Additional BBQ analyses examine model-size effects across nine social dimensions in ambiguous and disambiguated contexts after 800 RLHF steps. Accuracy generally increases with model size, but the best-performing prompting condition differs by context.
- Ambiguous Context: Accuracy increases with model size across all experimental conditions in the ambiguous context, with the highest accuracy in Q+IF+CoT.
- Ambiguous Context: Figure 6 plots model size against BBQ accuracy in the ambiguous context across nine social dimensions and overall, after 800 RLHF steps.Colors represent the experimental conditions from Table 1, and increasing accuracy indicates less bias.
- Disambiguated Context: Accuracy again increases with model size in the disambiguated context, but Q has the highest accuracy and Q+IF+CoT the lowest.Accuracy remains high enough in every condition to support meaningful bias scores.
- Disambiguated Context: Figure 7 plots model size against BBQ accuracy in the disambiguated context across nine social dimensions and overall, after 800 RLHF steps.Colors represent the experimental conditions from Table 1.
- Bias Scores: Figure 8 shows BBQ bias scores in the disambiguated context across nine social dimensions and overall after 800 RLHF steps.The x-axes encode model size, the y-axes encode bias score, and colors denote experimental conditions.
- Pronoun Probabilities: Figure 9 shows pronoun probability mass for the 175B model at 300 RLHF steps, with occupations sorted by neutral-pronoun probability.The Q+IF+CoT condition assigns most probability mass to neutral pronouns and almost none to female pronouns, yielding ρ = 0 in this case.
A.4 Winogender Additional Analyses
Winogender shows no significant relationship between RLHF-step variation and ρ, consistent with a ceiling effect. At higher RLHF step sizes, pronoun assignments become less diverse, making ρ estimates noisier.
- RLHF-step variation has no significant effect on ρ for any model size in Winogender.
- The authors attribute the null RLHF effect to coreference resolution being easier than BBQ or the discrimination experiment, producing a ceiling effect.
- Increasing RLHF steps tends to make models assign all probability mass to either female or male pronouns, increasing noise in higher-step ρ estimates.
- The increased noise is likely linked to lower output entropy and reduced sample diversity under extended RLHF.