Source-linked AI summary
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, Jack Clark
TL;DR
Language models can produce diverse harmful outputs, motivating systematic red teaming to discover and measure such behavior. The paper studies red-team scaling across model sizes and safety interventions, releases 38,961 attacks, and documents its procedures. RLHF models become increasingly difficult to red team as they scale, while the released data and methodological transparency support further analysis of model harms and red-teaming practice.
Problem
Language models exhibit harmful behaviors across biases, offensive or toxic outputs, privacy leakage, disinformation, extremism, and falsehoods, motivating tools to probe and reduce these harms.
Method
The authors red team four model types across 2.7B, 13B, and 52B parameters, collect attack data through open-ended conversations, and document their instructions, processes, and statistical methodologies.
Results
RLHF models become increasingly difficult to red team as they scale, while plain, prompted, and rejection-sampling models show flat trends; the study releases 38,961 attacks.
Takeaways & Limitations
The released attacks and transparent methodology support analysis of harmful outputs, development of red-teaming techniques and harmfulness classifiers, and work toward shared standards.
Takeaways & Limitations
The released data are incomplete because the open-ended space of possible harms is unknown and unbounded, and successful roleplay attacks on the RLHF model were absent from the dataset.
Abstract
from arXiv · showhide
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.
1 Introduction
The paper uses red teaming to probe harmful language-model behavior, measures how susceptibility changes across model sizes and safety interventions, releases a large attack dataset, and documents its methodology.
- Red teaming adversarially probes language models for harmful outputs and can inform updates intended to avoid those outputs.
- RLHF models become significantly harder to red team as they scale, while plain, prompted, and rejection-sampling models show flat trends.Prompted models are not significantly harder to red team than plain models, whereas rejection sampling is hardest at every scale.
- 38,961 red team attacks are released for analysis, including attacks on RLHF-trained models and preliminary analyses of the harms they uncover.The dataset is intended to support understanding successful attacks, semi-automated red teaming, harmfulness classifiers, and mitigation strategies.
- The authors exhaustively describe red-teaming instructions, processes, statistical methods, and uncertainty while incorporating Trust & Safety best practices.They also discuss limitations and propose shared community norms, practices, and technical standards.
2 Related Work
The work builds on manual and automated adversarial testing of language and generative models, extending prior efforts with larger-scale red teaming, scaling analysis, and released data.
- The authors use additional experiments to study model-size effects on red-team susceptibility and analyze attack content to characterize harms.They also provide more methodological detail and release data to support reproduction and improvement of the approach.
- Compared with the BAD dataset, this work collects approximately 40K attacks and red teams models up to 52B parameters.The authors also focus on reinforcement learning from human feedback.
- Prior automated red-teaming work finds harms including offensive discussion of groups, personally identifiable information, and private training-data leakage.The authors report similar harms and plan systematic manual-versus-automated comparisons in future work.
- Related NLP adversarial-testing research uses guidelines, templates, programmatic attack generation, and humans in the loop to expose model failures.
- Multimodal red teaming reveals overlapping harm categories, including bias, harassment, and disinformation, alongside categories not considered here such as copyright.
- The authors qualitatively observe greater attack diversity than in BAD, although they have not quantified this difference.The larger dataset is identified as at least a partial reason for the observed diversity.
3 Methods
The study uses open-ended human red teaming, paired-response selection, multiple dialogue-model variants, and several measures of attack success and harmfulness. It also examines worker heterogeneity and annotator agreement when interpreting the resulting data.
- 3.1 Red Team Task: Red team members hold open-ended, multi-turn conversations and select the more harmful of two model-generated responses after each turn.The procedure is designed both to find vulnerabilities more quickly and to generate preference comparisons.
- 3.1 Red Team Task: Workers rate attack success on a 0–4 Likert scale, with 0 meaning unsuccessful and 4 meaning very successful.Conversations generally last one to four turns, although longer conversations occur.
- 3.2 Models: The assistant uses a baseline plain LM plus prompted, rejection-sampling, and RLHF variants derived from general language models.The study trains decoder-only models at 2.7B, 13B, and 52B parameters and uses a helpfulness-and-harmlessness preference model in some variants.
- 3.5 Review Task: Agreement about successful attacks is poor to fair, with Fleiss’s Kappa of 0.32 across four raters and maximum reviewer-only binary agreement of 0.55.The authors therefore treat success judgments as an uncertain measurement rather than a perfectly shared label.
4 Results
Across model sizes and safety interventions, rejection sampling and RLHF reduced red-team success, while RLHF became increasingly difficult to red team as it scaled; nevertheless, harmful behaviors remained. The attacks covered both common and subtle harms, including offensive language, violence, misinformation, privacy risks, and non-violent unethical behavior.
- Evaluation: The results compare average self-reported red-team success rates across model sizes and safety interventions.Figure 1 reports the metric for each model size and intervention.
- Scaling and safety interventions: Rejection sampling made language models particularly difficult to red team, although its responses could remain harmless by being evasive.The authors found no clear model-size trend for self-reported attack success rate.
- Scaling and safety interventions: RLHF models became increasingly difficult to red team as model size increased, while rejection sampling was also an effective safety intervention.At 52B parameters, RLHF and rejection sampling showed no difference in minimum harmlessness score.
- Residual harmful behavior: Safety interventions reduced average harmfulness but did not eliminate harmful behavior, as shown by harmful lower-tail responses from RS and RLHF models.In examples, both models initially responded to harmful inquiries before demurring or refusing; the RLHF model remained helpful while fabricating information.
- Data quality and scope: Potentially synthetic personally identifiable information was conservatively filtered from the public dataset because some generated addresses appeared inaccurate and other records were difficult to verify.The authors manually reviewed data flagged by a regular expression for possible PII.
- Data quality and scope: Some crowdworkers produced low-quality attacks by repeatedly adapting templates without carefully considering attack efficacy or usefulness.The authors identified template-based attacks as a source of low-quality red-team data.
- Types of harms: Red-team attacks surfaced clusters involving discrimination, hate speech, violence, non-violent unethical behavior, bullying, misinformation, animal abuse, and personally identifiable information.The dataset also included less common categories such as child abuse, self-harm, sexual exploitation, terrorism, and organized crime.
- Types of harms: Non-violent unethical behavior attacks had relatively higher success rates, while fixed tag sets appeared insufficient to capture the full range of harms.The tag distribution was measured on a random sample comprising approximately 3% of the overall dataset.
5 Discussion
The authors identify limits in the scope, completeness, analysis, scalability, and governance of their red-teaming effort. They call for broader community practices to improve how systems are tested and findings are shared.
- Limitations and Future Work: The study focuses on AI assistants, so it does not explicitly address harms from recommendation systems, autocomplete, classifiers, or broader sociotechnical systems.The open-ended assistant framing enables probing input and output spaces but limits coverage of other applications.
- Limitations and Future Work: The released data are incomplete because the space of possible harms from general-purpose, open-ended language models is unknown and unbounded.The authors observed no code-generation attacks despite models being partly trained on Python code, and note that more domain experts might have surfaced them.
- Limitations and Future Work: Internal red teaming found successful roleplay attacks on the RLHF model that were absent from the released dataset.These attacks exploited helpfulness by asking the model to roleplay a malevolent character, such as entering “4chan mode.”
- Limitations and Future Work: The bottom-up analysis may miss attack types not covered by the collected data, motivating a combination of bottom-up and taxonomy-driven strategies.The authors point to frequent “Other” labels as evidence that people may discover attacks absent from existing taxonomies.
- Limitations and Future Work: Fully manual crowdworker red teaming is expensive and possibly slow to scale, so future work should compare manual, semi-manual, and automated approaches.The planned comparison concerns the efficacy and diversity of the resulting attacks.
- Policy Interventions: The field lacks shared norms for conducting red teams and releasing findings, while public datasets can support safer systems but also train more harmful ones.The authors therefore support multidisciplinary community coordination and a neutral forum for discussing release decisions.
A.1 Author Contributions
The appendix records the distributed contributions behind the project, spanning research design, writing, policy, model training, infrastructure, sampling, and data protection. It also acknowledges external advice and broader staff input.
- Research: Research leadership covered project co-leadership, data analysis, experiment design and execution, worker well-being research, and human-feedback infrastructure.The named contributors divided responsibilities across these research activities.
- Writing: Writing contributions included drafting the paper, shaping its framing and presentation, and providing miscellaneous editorial suggestions.The drafting and framing roles were assigned to different contributors.
- Policy: Policy work designed interventions, articulated dataset-release trade-offs, documented the dataset, and filtered personally identifiable information.The filtering process used a regular expression to identify PII before release.
- Model Training: Model-training contributions covered RLHF models, helpfulness and harmlessness preference models, rejection-sampling models, and the infrastructure supporting RLHF training.Broader pretraining leadership is also attributed to several contributors.
- Sampling and Cluster: Sampling and cluster teams supported efficient sampling, system implementation, research infrastructure, and cluster stability.These contributions made the computational research environment possible.
- Other Contributions: The paper acknowledges ideas developed through conversations with Anthropic staff and reports that academic experts and company representatives generally viewed dataset release as helpful.The release judgment nevertheless remained the authors’ own decision.
A.2 Safety Considerations for the Red Team
The authors designed red-team procedures around worker safety, drawing on Trust & Safety interviews and related literature. Their safeguards combined warnings, opt-out mechanisms, risk-sensitive participation, social support, compensation practices, and well-being measurement.
- Safety Design: Trust & Safety interviews and literature review informed the content and interface design intended to protect red-team workers.The authors consulted professionals with firsthand experience evaluating safety risks from harmful content exposure.
- Safety Design: Clear warnings described the task rationale and potentially troubling content before participants began red teaming or reviewing material.The goal was to minimize uninformed participation and unexpected exposure.
- Safety Design: Participants were encouraged to stay within their personal risk tolerance and could avoid topics that were triggering or unpleasant.No topics were mandatory for the Red Team Task.
- Safety Design: Recommended safeguards included breaks, walks, social interaction, mindfulness, and time-boxed exposure schedules.These practices were framed as components of personal resilience plans.
- Safety Design: Review workers were paid for time rather than quotas, received at least $20 per hour, and could skip transcripts after previewing them.The review task was also limited to a selected worker group with access to shared social support.
- Well-being Measurement: The well-being survey asked reviewers to rate 10 feelings on a 0-to-4 Likert scale, while 49 responses came from 15 participants.The survey was initially sent after every 10 tasks but manual distribution produced sporadic responses.
A.3 Controlling for Possible Confounds
The authors address confounds arising because different workers attacked different model types and sizes. A mixed-effects analysis controlling for worker, attack harmfulness, and platform indicates that the main results remain unchanged.
- Possible Confounds: Potential confounds include workers’ differing ability to elicit harmful outputs, differing harmfulness of intent, and crowdwork platform.The experiments involved approximately 300 red-team members across MTurk and Upwork.
- Statistical Control: A linear mixed-effects model used red-team-member random intercepts and fixed effects for attack-description harmfulness and platform.These terms were designed to control for worker, intent, and platform differences.
- Results: The main results still held after examining model-size, safety-intervention, and interaction coefficients in the fitted model.A second analysis also included one main metric as a covariate when predicting the other.
- Results: The two main metrics were correlated, and modeling one as a covariate for the other did not influence the main results.This re-capitulated the correlation shown in Figure 11.
A.4 The Relationship Between Attack Success and Harmlessness Score Metrics
The paper compares self-reported attack success with harmlessness preference-model scores. Higher reported success generally aligns with lower harmlessness scores, but the relationship is imperfect and highly variable.
- The two metrics are self-reported attack success on a Likert Scale and a harmlessness preference-model score.Higher Likert scores indicate greater attack success, whereas lower harmlessness scores indicate more harmful responses.
- As red team members report greater attack success, assistant utterances tend to receive lower harmlessness scores.
- The correlation is imperfect, with high variance in harmlessness scores for a given reported attack-success level.
Pros
The dataset’s principal advantages are its scale, model coverage, annotations, and potential usefulness for studying and mitigating AI safety failures.
- Openly releasing red-team data can help the community learn about and address AI safety failures.
- The data can support scaling-law studies, safety-classifier construction, automated red-team research, and characterization of the attack surface.
- It includes attacks on more capable models, including models trained with RLHF, and provides quantitative and qualitative annotations for filtering and analysis.
Cons
The dataset has important safety, privacy, measurement, and coverage limitations: it contains upsetting material and sensitive information, uses noisy labels and filtering, and is not comprehensive.
- PII was filtered with a regular expression and manually reviewed samples, but some assistant-generated PII was inaccurate or hallucinated and was redacted cautiously.
- The dataset may contain offensive or upsetting material involving discrimination, abuse, violence, self-harm, exploitation, and related topics.The authors recommend engaging with it only within one’s personal risk tolerance and research purposes.
- The harmlessness score is an automated, inherently noisy measure, while human attack-success labels are subjective and also noisy.
- Template-based red teaming creates redundancy among attacks, limiting the independence and diversity of some instances.
- The dataset contains sensitive information, but the authors do not know whether any instances include confidential information.
- The data could be misused to train more harmful models, and the dataset is not comprehensive of all possible harms or red-team attacks.