PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-Red: Automated Red Teaming via Self-Play at Scale

Eric Wallace, Christopher A. Choquette-Choo, Nikhil Kandpal, Sam Toyer, Dylan Hunn, Stephanie Lin, Yuxin Wen, Xiangyu Qi, Christopher Wolff, Zizhao Wang, Milad Nasr, Sicheng Zhu, Chuan Guo, Juan Felipe Cerón Uribe, Kaiwen Wang, Aiden Low, Kai Xiao, Kai Chen

arXiv:2607.26115v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Existing adversarial-training datasets are limited and can leave LLMs vulnerable to adaptive prompt-injection attacks. GPT-Red uses scalable attacker–defender self-play to discover novel attacks, transfers across settings, and helps make GPT-5.6 highly robust to prompt injections and other attacks.

  • Problem

    Limited adversarial-training datasets can cause models to overfit known attack patterns while remaining vulnerable to adaptive adversaries.

  • Method

    GPT-Red trains an attacker and defender through scalable self-play, rewarding valid failures and resistance across diverse red-teaming environments.

  • Results

    GPT-Red finds working attacks against prior GPT-5 models and transfers broadly across unseen adversarial goals, harnesses, and target models.

  • Takeaways & Limitations

    GPT-Red can be used to train and evaluate production models, with GPT-5.6 achieving high robustness across broad robustness evaluations.

  • Takeaways & Limitations

    GPT-Red has seen less training on multimodal environments, multi-turn attack scenarios, and content-policy jailbreaks.

Abstract

from arXiv · show

We introduce \textbf{GPT-Red}, an automated red-teaming agent that is trained to discover novel prompt injection attacks against frontier LLMs. The goal of this model is to evaluate and improve the robustness of our production systems. To this end, we use it to adversarially train GPT-5.6, our most robust model to prompt injections to date. To create GPT-Red, we design a scalable self-play algorithm where the model is tasked with attacking a diverse population of simultaneously-trained defender agents. We train the model on realistic red-teaming environments using compute on the same scale as some of our largest RL post-training runs, making it the single-largest LLM safety training run ever documented. GPT-Red excels at red-teaming: it reliably breaks our past models up to GPT-5.5, it finds more successful attacks than human red-teamers, and it generalizes to held-out environments, defender models, and harnesses. In the future, we expect that as we improve the robustness of each new GPT model, it will in turn will provide better learning signal for \textit{even stronger} red-teamer agents, thus unlocking a self-improvement flywheel.

1 Introduction

GPT-Red addresses the limited size and diversity of conventional adversarial-training data by using scalable self-play to train an adaptive red-teaming agent. Scaling inference-time compute, environment diversity, and RL training compute produces an agent that transfers broadly and supports progressively stronger model robustness.

  • Motivation: Conventional RL-based adversarial-training datasets are limited in size, causing models to quickly overfit to specific attack patterns.The datasets may come from human red-teamers, prompted LLMs, or real-world production attacks.
  • Approach: GPT-Red trains an attacker and defender through self-play, rewarding valid failures for the attacker and resistance while completing the original task for the defender.The approach is designed for repeated interaction, adaptation to target-model behavior, and varied adversarial goals, environments, and attack surfaces.
  • Scaling strategy: The algorithm scales attacker training through inference-time compute, environment diversity, and large-scale RL training compute.An agentic harness enables interactive attack refinement, realistic environments expose editable tool outputs or local file regions, and diverse defenders reduce overfitting to one model.
  • Results: GPT-Red shows strong test-time scaling, flexible tool use, and broad transfer across unseen adversarial goals, red-teaming harnesses, and target models.It finds working attacks against all prior GPT-5-series models and successfully prompt injected a real-world AI-powered vending machine.
  • Robustness improvement: GPT-Red is used during RL training to generate adversarial prompts and during evaluation, with progressively stronger attackers supporting successive production models since GPT-5.2.The stated goal is to keep pace with expanding model capabilities, more complex scenarios, and more capable adversaries.

2 Background and Related Work

This section frames adversarial robustness around prompt injections and content-policy jailbreaks, and positions the work as a scalable, LLM-based alternative to prior red-teaming and adversarial-training methods.

  • Threat Models: The paper studies automated red-teaming against prompt injections and content-policy jailbreaks to improve model robustness under malicious inputs.Prompt injections conflict with developer or user intent, while jailbreaks attempt to bypass safety boundaries around harmful requests.
  • Threat Models: Instruction hierarchy defenses assign privilege levels to inputs and require models to prioritize higher-privilege instructions when conflicts arise.The hierarchy can include system, developer, user, and tool-response instructions.
  • Related Work: Prior red-teaming datasets construct realistic human- or model-generated scenarios, whereas this work targets substantially larger and more diverse data generation.The passage contrasts existing datasets with the paper’s proposed methods for generating larger and more diverse amounts of red-teaming data.
  • Related Work: Earlier automated red-teaming used discrete optimization, but these methods often require white-box access, many defender queries, and struggle to transfer across models or settings.The paper treats those optimization-based methods as distinct and complementary to its automated LLM-based approach.
  • Related Work: Prior LLM-based approaches used prompted attackers, narrow training goals, few environments, or test-time compute independent of model training.The paper identifies model-training-based scaling as its core focus and distinguishes it from test-time compute scaling.
  • Adversarial Training: Modern reasoning LLMs are safety-trained with SFT and RL, and this work uses an automated red-teamer within that adversarial-training framework.SFT relies on fully labeled demonstrations, whereas RL uses malicious prompts to reinforce safe or proper behavior.

3 Vision for Auto Red-Teaming

The vision is an automated red-teaming agent that discovers attacks against frontier models to evaluate and improve their robustness. It should generalize across targets and scenarios, generate diverse valid attacks, and operate under a realistic threat model with constrained access and objectives.

  • Core capabilities: The agent should discover direct and indirect prompt injections and content-policy violations against defender models, including attacks that pursue goals such as exfiltrating private data.It should iteratively craft attacks that achieve the specified adversarial goal when encountered by the defender.
  • Generalization: The red-teamer should generalize as a drop-in attacker across unseen models, agentic workflows, production environments, and attacker scenarios.Examples include attacking next-generation GPT models and writing adversarial Slack messages without task-specific training.
  • Diversity: It should produce diverse attack styles for the same goal or environment, enabling adversarial training without many specialized attacker models.This supports using substantial compute to optimize against one attacker population.
  • Validity and threat model: The agent should follow realistic constraints rather than control privileged rollout components or exploit access unavailable to a real adversary.GPT-Red may modify one tool response or a non-privileged set of files, while privileged files and tool responses remain protected.
  • Threat model and objectives: GPT-Red receives limited rollout information and black-box defender queries, while proposed attacks are placed at specified locations and automatically checked against format, length, and other constraints.Training objectives include exfiltrating simulated sensitive data, running untrusted code, and destructive code actions.

4 Inference Scaling of Attackers

GPT-Red scales attacker inference by querying a defender-model tool that supports sequential, structured attack refinement. This agentic harness improves attack success rate over independent best-of-N sampling at equal query budgets, and GPT-Red outperforms GPT-5.5 in the same setting.

  • Inference scaling: The defender_model tool lets GPT-Red test prospective attacks against the defender while reasoning over attack strategies and malicious prompts.The tool provides a mechanism for scaling attacker inference compute.
  • Inference scaling: Within a single rollout, attackers can branch over message variants, edit attacks token-efficiently, and generate attacks programmatically through terminal code.The terminal supports file creation and execution for templating, repetition, and composing attack strategies.
  • Test-time scaling: Scaling attacker compute improves ASR for both best-of-N and agentic attackers, while defender-model queries achieve higher ASR at equal query budgets, especially with more calls.The agentic harness queries the defender sequentially in context, enabling attack refinement rather than independent trials.
  • Test-time scaling: GPT-Red achieves higher ASR than GPT-5.5 with the same number of defender_model tool calls and tends to use the tool more.This comparison uses the same setting as Figure 1, with attackers defining attacks in the Dziemian et al. environment.

5 Self-Play Training

GPT-Red is trained through self-play reinforcement learning against a diverse, simultaneously trained population of defenders, with attackers rewarded for eliciting failures and defenders for resisting them. This setup yields stable co-adaptation, improves held-out defender robustness and attacker capability, and avoids narrow attack-strategy collapse through multi-defender training.

  • Training objective: Self-play jointly trains GPT-Red and diverse defender LLMs, rewarding valid failures for the attacker and successful resistance for defenders.As defenders become more robust, GPT-Red must discover stronger attacks.
  • Training dynamics: Attackers adapt as defenders improve, with patched strategies producing attack-success-rate dips before attackers move to new approaches.These dynamics provide a useful curriculum for both models.
  • Defender robustness: Self-play-trained attackers produce the fastest-improving and best-final defender robustness on held-out evaluations, outperforming frozen attackers with or without defender query access.Held-out mean direct prompt injection robustness rises roughly monotonically over self-play.
  • Attacker capability: Later attacker checkpoints are nearly monotonically better at breaking earlier and later defender checkpoints, whereas attacker-only training is narrower and worse on held-out defenders.This indicates that self-play expands attacker capability rather than causing local overfitting.
  • Multi-defender diversity: Training against many diverse defenders prevents attackers from mode-collapsing onto a narrow attack class and encourages probing each defender’s unique weaknesses.The effect is strongest when defenders are highly robust to different attack types.

6 Red Teaming RL Environments

The section develops diverse safety RL environments that convert realistic capability tasks into constrained adversarial settings for training red-teamers and robust defenders. It covers direct, indirect, multimodal, and refusal-jailbreak attacks with verifiable or rubric-based rewards.

  • Environment design: The environments convert existing capability tasks into adversarial training settings, including realistic defender tasks and constrained attacker-controlled edits.Attackers may edit designated tool responses or local file regions while defenders perform tasks drawn from distributions such as GPT-5.6 post-training data.
  • Direct prompt injections: Direct prompt-injection environments restrict attackers to editing one lower-priority message and evaluate behaviors such as secrecy, regex replacement, and structured parsing.Rewards are computed programmatically based on whether the defender or attacker succeeds.
  • Indirect prompt injections: Indirect prompt-injection environments let attackers modify untrusted website results, connector outputs, function-call responses, or selected local files in browsing and coding tasks.Insertion points include email bodies, calendar descriptions, web-page regions, function outputs, and files.
  • Multi-modal prompt injections: Multimodal environments require attackers to submit rendered images that modify constrained visual content while preserving task-relevant facts, labels, values, and source cues.Supported settings include computer-use screenshots and browsing or document-retrieval results, with attacks designed to appear natural and convincing.
  • Jailbreaking safety refusals: Jailbreak environments train attackers to steer benign conversations toward disallowed assistance, using single- and multi-turn tasks scored by harmfulness rubrics.Defenders minimize the rubric score by safely completing or refusing, while filters reject attacks containing disallowed content or superficial transformations.

7 GPT-Red

GPT-Red is trained at unprecedented safety-RL scale against past defender models and discovers diverse, controllable prompt-injection attacks. It achieves non-trivial success across training settings, generalizes to held-out GPT versions and tasks, outperforms existing red-teaming approaches in held-out evaluations, and succeeds against a real-world vending-machine agent.

  • Training: GPT-Red is initialized from GPT-5.5 and trained across all Section 6 environments plus standard RL post-training environments, using compute roughly comparable to recent GPT-5 production post-training runs.This makes GPT-Red the largest published safety RL run to date.
  • Attack capabilities: GPT-Red discovers novel, complex, diverse, and steerable attacks, including fake Chain-of-Thought attacks and realistic multimodal prompt injections generated with its image tool.It can generate artifacts such as a realistic receipt containing a seamlessly rendered fake exfiltration URL, and its behavior is controllable with few-shot examples or high-level strategy guidance.
  • Generalization: GPT-Red achieves non-trivial attack success rates across every trained-against model and red-teaming setting, with attacks succeeding up to nearly twice as often on earlier GPT-5 models through GPT-5.2.The attacks were replayed against held-out GPT versions, including earlier models through GPT-5.2.
  • Benchmark evaluation: GPT-Red outperforms prompted frontier frozen LLMs and human red-teamers on held-out indirect prompt injection tasks, although this does not establish universal superiority over humans.Humans may discover novel scenarios or attack classes not considered by GPT-Red.
  • Real-world evaluation: GPT-Red succeeds at all three objectives against a production AI-powered vending machine: repricing an expensive item, offering a new $100+ item for $0.50, and canceling another customer’s order.Its production-system attack used fabricated metadata claiming the message came from a trusted administrator.

8 Robustness of GPT-5.6

GPT-Red attacks are incorporated into GPT-5.6’s RL training and evaluation, producing strong robustness to held-out and human prompt-injection attacks while preserving broad capability evaluation.

  • Training and evaluation: GPT-Red generates adversarial prompts for GPT-5.6’s RL training and evaluates progress on in- and out-of-distribution prompt-injection metrics.The attacks are included during capability and safety training, causing GPT-5.6 to hillclimb on overall robustness.
  • Held-out robustness: GPT-5.6’s robustness on strong held-out attacks exceeds 50% and reaches 89% across held-out attacks, datasets, and domains.The held-out evaluations include separate task distributions and an entirely held-out agentic-coding domain.
  • Held-out robustness: ASR is below 4% for GPT-5.6 on a held-out automated red-teaming exercise targeting indirect prompt injections, with robustness improving monotonically over time.The exercise evaluates indirect prompt-injection robustness on held-out red-teaming attacks.
  • Human attacks: GPT-5.6 achieves 100% robustness on the IPI 2025 Challenge human-attack dataset, apart from one misgraded success validated by human inspection.This result concerns the rerun of the human attacks on GPT-5.6.
  • Direct prompt injections: GPT-5.6 is evaluated against direct prompt-injection attacks using IID generalization and human red-teaming benchmarks, with comparisons to GPT-5.1.GPT-5.1 received substantial robustness training but did not include attacks from strong automated red-teamers such as GPT-Red.

9 Conclusion and Future Work

GPT-Red demonstrates a scalable self-play approach for training autonomous red-teamers and improving frontier-model robustness. Future work will broaden scenario coverage and scale training to expose more failures and support a safety flywheel.

  • Conclusion: GPT-Red uses scalable self-play to train strong autonomous red-teamers and improve GPT-5.6’s robustness to prompt injection.The training run was conducted at unprecedented scale for a single safety-focused run.
  • Conclusion: Robustness improves when attacker inference-time search, self-play training compute, and safety training environments are scaled together.These are the three ingredients identified as driving improved robustness.
  • Future Work: GPT-Red currently has less training on multimodal environments, multi-turn attacks, and content-policy jailbreaks.Planned work will broaden coverage while continuing to scale compute, data, and self-play training.
  • Future Work: Future advances are expected to produce more capable red-teamers that expose a wider range of failures in future frontier models.The paper frames this progress as a safety flywheel in which GPT-Red improves the robustness, alignment, and trustworthiness of later models.

A Additional Figures

Figure 14 shows that attackers improve their test-time compute usage through training: adding the defender_model tool at test time significantly improves performance, while training with it yields further attack-success gains.

  • Additional Figures: Adding the defender_model tool at test time significantly improves attackers’ performance, and training with the tool produces further improvements in attack success.This training setup is used in GPT-Red.

B Additional Attack Examples

Figures 15–22 present eight random GPT-Red attack examples, comparing GPT-5.1–GPT-5.5 with GPT-5.6 across goals including data exfiltration, theft, security compromise, execution, and package installation. Figures 23 and 25–30 add further attacks in direct prompt-injection scenarios.

  • Additional direct-injection examples: Figures 23 and 25–30 show additional attacks found by the attacker LLMs in instruction-hierarchy direct prompt-injection scenarios.These figures extend the examples beyond Figures 15–22.

ANSWER

The answer section illustrates GPT-Red attacks that extract secrets, override instructions, exploit formatting and counting weaknesses, and target safety refusals or rigid classifiers. One example uses n-ary search to identify the secret PIN “3207.”

  • Instruction hierarchy: The attacker finds system-override attacks that attempt to supersede higher-priority instructions.One example includes conflicting high-priority instructions, while another directly attempts to override previous system messages.
  • Constraint evasion: Other attacks probe constraints by constructing prohibited content indirectly or encoding banned words before the defender checks for conflicts.Examples include using a regex that would contain a concealed PIN and assembling a prohibited word from character codes.
  • Model weakness exploits: GPT-Red exploits LLM weaknesses by requesting deeply nested JSON, relying on limited character counting to induce mismatched brackets and invalid output.The example requests at least 50 nested levels and an end marker at the deepest object.
  • Safety and classifier jailbreaks: The safety environments test whether attackers can elicit disallowed content or break exact-output classifiers under multi-turn, lower-priority user messages.Defender outputs are scored against harmful-answer rubrics or strict JSON classification rules.

C Additional Evaluation Details

The evaluation mixes internal and external instruction-hierarchy-violation benchmarks, including LLM-generated attacks aligned with the training distribution and attacks from human red teaming. It covers specialized attack types, multi-defender settings, assorted self-play sources, and the TensorTrust robustness dataset.

  • Instruction hierarchy violations: The benchmark suite combines internal and external evaluations, including LLM-generated attacks and attacks produced through human red teaming.LLM-generated benchmarks more closely match the distribution induced by training.
  • Instruction hierarchy violations: Specialized benchmark categories cover fake chain-of-thought attacks, system-instruction overrides, and short edge-case prompts exploiting formatting, multilingual text, special characters, or ambiguity.System-override attacks use fake conversations, with verifiable code as the reward.
  • Instruction hierarchy violations: Multi-Defender Attacks are collected from a trained attacker run against ten different strong defenders.This category evaluates attacks across multiple defender models.
  • Instruction hierarchy violations: Assorted Attack Categories draw from past self-play runs, frozen-attacker runs, and best-of-n attackers.This aggregates attacks from several prior attacker-generation procedures.
  • Instruction hierarchy violations: The TensorTrust Robustness dataset places password-protection instructions in a Developer message and attacks in a User message, rewarding verifiable “Access Granted” outputs.The benchmark tests whether the model outputs the required access-granted response.
Loading 2607.26115v1…