Source-linked AI summary
OpenAI's Approach to External Red Teaming for AI Models and Systems
Lama Ahmad, Sandhini Agarwal, Michael Lampe, Pamela Mishkin
TL;DR
Rapidly evolving AI systems pose risks that red teaming can help assess. This paper describes OpenAI’s external red teaming and reports how it surfaced issues and informed mitigations and evaluations.
Problem
Assessing AI model and system risks requires attention to evolving capabilities, user experiences, and real-world factors such as cultural context.
Source: The value of external red teaming · p. 2, §5 · p. 10Method
Results
External red teaming surfaced unauthorized voice-generation behavior and DALL-E 3 risk gaps, with findings informing mitigations and selected automated evaluations.
Source: Data synthesis and alignment to… · p. 8, Creating automated evaluations · p. 8Takeaways & Limitations
External red teaming is one input to AI safety practice alongside automated red teaming, benchmarks, third-party assessments, and research on AI impacts.
Source: §5 · p. 10Point-in-time red teaming findings may no longer reflect evolving production systems, so red teaming alone is not a panacea for risk assessment.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Red teaming has emerged as a critical practice in assessing the possible risks of AI models and systems. It aids in the discovery of novel risks, stress testing possible gaps in existing mitigations, enriching existing quantitative safety metrics, facilitating the creation of new safety measurements, and enhancing public trust and the legitimacy of AI risk assessments. This white paper describes OpenAI's work to date in external red teaming and draws some more general conclusions from this work. We describe the design considerations underpinning external red teaming, which include: selecting composition of red team, deciding on access levels, and providing guidance required to conduct red teaming. Additionally, we show outcomes red teaming can enable such as input into risk assessment and automated evaluations. We also describe the limitations of external red teaming, and how it can fit into a broader range of AI model and system evaluations. Through these contributions, we hope that AI developers and deployers, evaluation creators, and policymakers will be able to better design red teaming campaigns and get a deeper look into how external red teaming can fit into model deployment and evaluation processes. These methods are evolving and the value of different methods continues to shift as the ecosystem around red teaming matures and models themselves improve as tools for red teaming.
1 Introduction
This section situates OpenAI’s external red teaming within a varied practice, describes its value for risk assessment, and outlines approaches used across settings.
1 Introduction
- OpenAI’s external red teaming collaborates with domain experts to evaluate AI model and system capabilities and risks.The paper presents its processes as potentially informative for other deployers and stakeholders incorporating human red teaming into risk assessment.Source: §1 · p. 1
Red teaming as a safety practice: industry and policy
- Red teaming is part of industry and policy approaches to AI risk assessment, with government and company practices also adopting or publishing methods.OpenAI’s work includes efforts since DALL-E 2 and published System Cards for several model deployments.Source: Red teaming as a safety practice:… · p. 2
The value of external red teaming
- External red teaming discovers novel risks, stress-tests mitigations, brings domain expertise, and adds independent input that can strengthen trust.For rapidly evolving models, findings can identify risk areas where reliable benchmarks are unavailable and inform automated or human-AI evaluations [15].Source: The value of external red teaming · pp. 2–3
Emerging methods for red teaming
- Manual, automated, and mixed red teaming are complementary approaches, and internal teams can help shape campaigns or address practical constraints.The paper describes manual testing as human-crafted prompts, automated testing as generated inputs and sometimes classifier grading, and mixed methods as scaling manually seeded prompts.Source: Emerging methods for red teaming · pp. 3–4
2 Designing a red teaming campaign
A red teaming campaign is designed around its testing goals, with decisions about the cohort, model access, and support provided to testers.
2 Designing a red teaming campaign
- Choose the red teaming cohort based on the campaign’s goals and prioritized testing domains.The paper also identifies open questions and threat models as considerations for scoping the work.Source: §2 · p. 4
- Determine which model or system versions red teamers can access.Source: §2 · p. 4
- Create interfaces, instructions, and documentation guidance for red teamers.Source: §2 · p. 4
4. Synthesizing the data and creating evaluations
Campaign design aligns testing domains, access, and tester guidance with the model, system, and goals of the red teaming effort.
Areas of testing and composition of red team
- Prioritize testing areas through threat modeling, pair them with motivating questions, and compose cohorts to match the domains and model under test.The paper notes that cohort composition can vary by professional background, education, gender, age, geography, and languages; existing evaluation maturity also affects whether testing is manual or benchmark-based.Source: Areas of testing and composition of… · pp. 4–5, Figure 1 · p. 5
Tailoring model access to red teaming goals
- Choose model access to suit campaign goals: early access can reveal base-level risks, while post-mitigation access can test residual risks and mitigation robustness.Earlier snapshots can yield obsolete or unrepresentative findings, so the optimal access design depends on the model or system’s needs.Source: Tailoring model access to red teaming… · pp. 5–6, Figure 2 · p. 6
Instructions, interfaces, and documentation
- Instructions, interfaces, and documentation shape testing: guidance may be open-ended or structured, interfaces serve different purposes, and recorded findings can support evaluations.Documentation can capture prompts or conversations, finding categories, risk levels, and supporting context; it may need to evolve for multi-turn, multimodal, or tool-using interactions.Source: Instructions, interfaces, and… · pp. 6–7, Figure 1 · p. 7
3 Human red teaming to automated evaluations
External red-teaming findings can inform desired behaviors and broader risk assessment, while human-generated data can seed repeatable automated evaluations. The examples show how discoveries about model behavior become policy-aligned responses and targeted evaluations.
Data synthesis and alignment to desired behaviors
- After a campaign, teams must decide whether examples fall under existing policies, violate them, or warrant a new policy or desired model behavior.OpenAI’s policies are informed by resources including Usage Policies, the Moderation API, and the Model Spec.Source: Data synthesis and alignment to… · pp. 7–8
- Red-teaming data surfaced unauthorized voice generation and informed mitigations and evaluations that detect deviations from reference voices.The diverse voices and accents in the data also helped create evaluations across voices on capability benchmarks and refusal behavior.Source: Data synthesis and alignment to… · p. 8
- Red teaming informs risk assessment beyond policy violations by surfacing potential risks and desirable or undesirable model behaviors.Careful consideration of results matters because external teamers and deployers may differ in expectations of safe or acceptable behavior.Source: Data synthesis and alignment to… · p. 8
Creating automated evaluations
- Human red-team prompts and judgments can seed repeatable safety evaluations, including prompts paired with ideal behavior measured by rule-based classifiers.These evaluations can be deployed more quickly and inexpensively, and red-team data can help identify valuable evaluation targets and provide harder, unique prompts.Source: Creating automated evaluations · p. 8
- DALL-E 3 red teaming uncovered gaps involving misinformation-prone images, jailbreaks for sexually explicit content, and self-harm imagery, prompting selected-domain automated evaluations.Red-team prompts seeded the evaluations; GPT-4 generated synthetic data, and a GPT-4 classifier checked whether requests were refused or sufficiently rewritten before image generation.Source: Creating automated evaluations · p. 8
- GPT-4 red-team findings about Base64 text encryption and decryption prompted further evaluations of text encryption or compression followed by decryption.The evaluations studied one model encrypting or compressing text and another model decrypting it.Source: Creating automated evaluations · pp. 8–9
- Some policies are enforced through system-level mitigations, such as monitoring and account-level actions, rather than behavior observable in a single-turn interaction.A refusal in a single interaction therefore does not capture every policy enforcement mechanism described here.Source: Creating automated evaluations · p. 8
4 Limitations and risks of red teaming
External red teaming has limitations as a standalone risk-assessment practice: findings can age as systems evolve, and the work can impose substantial resource, participant-safety, and information risks. The paper also identifies fairness concerns and rising demands on human expertise, while pointing to automation as a partial aid.
Limitations and risks of red teaming
- Red-teaming findings are point-in-time: production models and systems evolve, so surfaced risks may be under-assessed or no longer reflect an updated system.The paper therefore says red teaming should not be treated as a panacea for AI-system risk assessment.Source: §4 · p. 9
- The described form of red teaming requires substantial operational time and financial resources, limiting some less-resourced organizations’ ability to use it effectively at scale.Developing evaluations where possible is presented as a way to maximize the impact of external red-teaming investments.Source: §4 · p. 9
- Thinking like adversaries and interacting with harmful content may reduce productivity or cause psychological harm, especially for participants from marginalized groups.The paper identifies mental-health resources, fair compensation, and informed consent as risk-mitigation steps.Source: §4 · p. 9
- Red teaming, particularly with frontier AI systems, can create information hazards when newly exposed jailbreaks or harmful-content techniques may enable misuse.The paper calls for information control, stringent access protocols, and responsible disclosure practices.Source: §4 · p. 9
- Early access may give red teamers with outside research or business interests an unfair advantage.The paper says this issue needs addressing to maintain fairness and integrity.Source: §4 · p. 9
- More capable models raise the expertise threshold for humans to judge output risks, while more robust models may take greater effort to jailbreak.Source: §4 · p. 9
- Improving automated red teaming can help mitigate some limitations and focus human effort on higher-value areas while reducing time spent reviewing undesirable content.Source: §4 · p. 9
5 Conclusion
External red teaming can support proactive risk assessment and more current safety evaluations, but it is one input among broader practices and does not capture every consideration. The paper also calls for public perspectives and externally specified accountability thresholds.
5 Conclusion
- External experts can help discover and test risks, informing benchmarks and safety evaluations that can be reused and updated over time.No singular process captures all relevant considerations.Source: §5 · p. 10
- Red teaming should be paired with public input on model behavior and policies, plus externally specified thresholds and accountability practices for discovered risks.Source: §5 · p. 10
- External red teaming is one tool alongside automated red teaming, safety benchmarks, third-party assessments, and research on AI impacts; its methods are expected to evolve.The paper presents its methodologies and tradeoffs as potentially useful for designing more clearly defined, actionable campaigns.Source: §5 · p. 10
Appendix A
Appendix A offers illustrative questions for examining model capabilities, limitations, and possible risks across application domains and types of behavior.
Code / Writing Software and Systems Architecture
- Questions probe whether tool use, software synthesis, or cybersecurity capabilities augment models in ways that affect risks, including transactions, dangerous planning, vulnerabilities, phishing, and private information.Source: Appendix A · p. 14
Politics and Election Considerations
- The illustrative questions ask how models might support political strategy and whether bias could affect sensitive topics or decisions such as hiring, education, and credit.Source: Appendix A · pp. 14–15
Natural Sciences
- Questions cover natural-science and legal capabilities and limitations, including how they may affect risks in high-stakes contexts.The appendix also asks how capabilities in natural sciences may change speed, accuracy, efficiency, cost, or required expertise, and raises CBRN threat creation.Source: Appendix A · pp. 13–15
Situational Awareness / Autonomous Replication
- Behavioral questions include situational awareness, autonomous replication, refusal of harmful requests, inclusive responses, persuasive arguments, and sensitive emotional or personal advice.Source: Appendix A · p. 15