PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-Red: Automated Red Teaming via Self-Play at Scale

Eric Wallace, Christopher A. Choquette-Choo, Nikhil Kandpal, Sam Toyer, Dylan Hunn, Stephanie Lin, Yuxin Wen, Xiangyu Qi, Christopher Wolff, Zizhao Wang, Milad Nasr, Sicheng Zhu, Chuan Guo, Juan Felipe Cerón Uribe, Kaiwen Wang, Aiden Low, Kai Xiao, Kai Chen

arXiv:2607.26115v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Agentic LLMs must resist adversarial manipulation, yet adversarial-training datasets remain limited in size. The paper trains GPT-Red through attacker–defender self-play. GPT-Red finds working attacks against prior GPT-5-series models, generalizes to unseen settings, and helps make GPT-5.6 highly robust to prompt injections.

  • Problem

    Safely deploying agentic LLMs requires resistance to adversarial manipulation, but datasets used for adversarial training are limited in size.

    Source: §1 · p. 1

  • Method

    GPT-Red uses scalable self-play in which attackers learn to elicit failures while defenders learn to resist attacks and complete their original tasks.

    Source: §1 · p. 1, §9 · p. 12

  • Results

    GPT-Red finds working attacks against all prior GPT-5-series models and transfers to unseen adversarial goals, harnesses, and target models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Using GPT-Red for adversarial training helps make GPT-5.6 highly robust to prompt injections and other attacks.

    Source: §1 · p. 2, §9 · p. 12

    GPT-Red has received less training on multimodal environments, multi-turn attack scenarios, and content-policy jailbreaks.

    Source: §9 · p. 12

Abstract

from arXiv · show

We introduce \textbf{GPT-Red}, an automated red-teaming agent that is trained to discover novel prompt injection attacks against frontier LLMs. The goal of this model is to evaluate and improve the robustness of our production systems. To this end, we use it to adversarially train GPT-5.6, our most robust model to prompt injections to date. To create GPT-Red, we design a scalable self-play algorithm where the model is tasked with attacking a diverse population of simultaneously-trained defender agents. We train the model on realistic red-teaming environments using compute on the same scale as some of our largest RL post-training runs, making it the single-largest LLM safety training run ever documented. GPT-Red excels at red-teaming: it reliably breaks our past models up to GPT-5.5, it finds more successful attacks than human red-teamers, and it generalizes to held-out environments, defender models, and harnesses. In the future, we expect that as we improve the robustness of each new GPT model, it will in turn will provide better learning signal for \textit{even stronger} red-teamer agents, thus unlocking a self-improvement flywheel.

1 Introduction

GPT-Red uses large-scale self-play across diverse defenders and realistic adversarial environments to generate attacks for robustness training. It outperforms human red-teamers and GPT-5.5 in held-out prompt-injection scenarios.

1 Introduction

  • Limited attack datasets can cause adversarially trained models to overfit known patterns while remaining vulnerable to adaptive adversaries (Nasr et al., 2025).Expanding model capabilities and use cases enlarge attack surfaces, increasing the challenge of maintaining robustness.Source: §1 · p. 1
  • GPT-Red outperforms human red-teamers and GPT-5.5 on held-out 2025 IPI Challenge scenarios (Dziemian et al., 2026), improving further with additional test-time compute.It finds working attacks against all prior GPT-5-series models and transfers to unseen adversarial goals, harnesses, and target models. It also exhibits flexible tool use, discovers novel attack strategies, and successfully prompt-injected an AI-powered vending machine.Source: Figure 1 · p. 1, §1 · p. 2
  • Self-play rewards attackers for valid failures and defenders for resisting attacks while completing tasks, creating progressively harder adversarial training data as both improve.The intended red-teamer learns through repeated interaction, adapts to individual target models, and varies strategies across adversarial goals, environments, and attack surfaces.Source: §1 · p. 1
  • An agentic harness and adversarial environments derived from capability tasks scale interactive attack search across diverse scenarios under a realistic threat model.A stateful defender_model tool lets attackers iteratively refine attacks against a chosen defender. Environments can interrupt a defender tool call and permit edits to a designated tool response or local file region.Source: §1 · pp. 1–2
  • Training at the scale of a large OpenAI RL post-training run pits the attacker against diverse strong defenders rather than a single opponent.This population-based extension aims to prevent overfitting to one defender’s quirks and teach the attacker to probe each model’s particular weaknesses.Source: §1 · p. 2
  • GPT-Red supplies adversarial RL training prompts and supports robustness evaluation; GPT-5.6 achieves high robustness across broad evaluation suites, although robustness remains unsolved.Progressively stronger precursors have supported training of successive production models since GPT-5.2. The authors hope continued attacker–defender improvement will keep pace with expanding capabilities, more complex scenarios, and stronger adversaries.Source: §1 · p. 2

2 Background and Related Work

The section frames prompt injections and content-policy jailbreaks as targets for automated red-teaming and adversarial training. It distinguishes the work through larger, more diverse scenarios, broader attacker-training environments and targets, and RL post-training.

Prompt injections

  • Prompt injections redirect models through direct user inputs or untrusted third-party content (Toyer et al., 2024).Direct attacks attempt to override system or developer instructions, while indirect attacks hijack behavior through content such as tool outputs and retrieved documents. The instruction hierarchy teaches models to prioritize higher-privilege instructions when conflicts arise (Wallace et al., 2024).Source: §2 · p. 2

Content-policy jailbreaks

  • Content-policy jailbreaks bypass safeguards through rewording, fictionalized role-play, or indirection, targeting self-harm, illicit advice, and biological, chemical, and cyber risks.These attacks challenge boundaries established by training models to refuse or safely complete harmful requests (Yuan et al., 2025).Source: §2 · p. 2

Red-teaming Datasets

  • The paper proposes generating substantially larger and more diverse red-teaming scenarios to enable large-scale training.The intended expansion covers scenarios for both prompt injections and content-policy jailbreaks.Source: §2 · p. 2

Automated Red Teaming

  • Adversarial-suffix optimization often requires white-box access and numerous defender queries, transfers poorly, and is complementary to the paper’s LLM-based approach.Examples target content-policy jailbreaks (Zou et al., 2023) and prompt injections.Source: §2 · p. 3
  • Compared with Ma et al. (2023); Liu et al. (2025), this work scales LLM-based red-teaming through broader environments and adversarial targets.Earlier approaches prompted attackers or trained them on narrow goals in few environments (Beutel et al., 2024; Shi et al., 2025). Other work scales test-time compute rather than training, while Deng et al. (2025) studies a defender classifier rather than a generic LLM.Source: §2 · p. 3

Existing techniques for adversarial training

  • The paper uses an automated red-teamer to generate challenging robustness tasks for adversarial training during RL post-training.Modern reasoning LLMs use SFT and RL for safety training (Guan et al., 2024). SFT requires labeled demonstrations, whereas RL reinforces safe or proper behavior on malicious prompts.Source: §2 · pp. 2, 4

3 Vision for Auto Red-Teaming

GPT-Red’s vision is an automated attacker that helps evaluate and improve frontier-model robustness by discovering diverse, valid attacks that generalize to new settings. Its training threat model limits attack surfaces and defender access, with checks to reject invalid attacks.

3.1 Goals of the Attacker

  • The attacker should improve frontier-model robustness by discovering direct prompt injections, indirect prompt injections, and content-policy violations.It should iteratively craft attacks toward a specified adversarial goal. As a drop-in red teamer, it should generalize to unseen models, harnesses, scenarios, workflows, and production environments.Source: §3.1 · p. 4, §3 · p. 4
  • Attackers should produce diverse attacks while respecting realistic constraints on what they can modify.Diversity supports substantial adversarial-training compute without requiring many specialized attackers. Constraint-following prevents reward hacking through edits to rollout components unavailable to a realistic adversary.Source: §3.1 · p. 4

3.2 Threat Model and Affordances

  • GPT-Red can modify a single tool response or a set of non-privileged files, but cannot modify privileged files or tool responses.Permitted surfaces include one search result or non-crucial downloaded reports; a user’s AGENTS.md is excluded.Source: §3.2 · p. 4
  • GPT-Red receives limited defender context and black-box access through a limited number of queries.Context typically includes the latest user message and the original injection-target response or file. Query feedback sometimes includes the full post-injection defender trajectory and often associated token log-likelihoods.Source: §3.2 · p. 4
  • Attacks are typically inserted at predefined rollout positions, with programmatic and LLM-based checks rejecting invalid submissions.Additional constraints can require matching a tool-response schema or limiting attack length. The authors qualitatively assess enforcement as highly effective.Source: §3.2 · p. 4
  • Training objectives vary with user tasks and available tools, including simulated sensitive-data exfiltration, untrusted-code execution, and destructive code actions.These are example objectives for indirect prompt injections.Source: §3.2 · p. 4

4 Inference Scaling of Attackers

Sequential testing through a defender_model tool improves attack success over independent best-of-N sampling at equal query budgets. GPT-Red also achieves higher attack success than GPT-5.5 at equal tool-call counts.

Defender query interface

  • GPT-Red uses terminal-created attack files and a defender_model tool to test candidates and receive defender trajectories, token log-likelihoods, and judge-LLM feedback.The harness inserts each attack into a prespecified position before sampling the defender. This interface scales attacker inference compute by enabling reasoning over strategies and searches through possible malicious prompts.Source: §4 · p. 5
  • Within one rollout, attackers can branch over variants, edit attack files efficiently, and generate attacks programmatically before submitting a final file path.Attackers submit when satisfied or when time expires. Some training scenarios also support stateful, multi-turn attacks.Source: §4 · p. 5

Test-time scaling improves attacks

  • Sequential defender_model queries achieve higher attack success rates than best-of-N at equal query budgets, especially as defender calls increase.Both methods improve with more queries; ASR measures the percentage of evaluation instances with at least one successful attack. The experiment uses a frozen internal helpful-only attacker against a GPT-5-series defender on IH-Challenge instruction-hierarchy tasks from Guo et al. (2026).Source: Figure 3 · p. 5, §4 · p. 5
  • GPT-Red achieves higher ASR than GPT-5.5 at the same number of defender_model calls and tends to use the tool more.Attackers define attacks in the environment of (Dziemian et al., 2026).Source: §4 · p. 5

5 Self-Play Training

Self-play jointly trains GPT-Red and defender models, producing the fastest gains and best final held-out defender robustness compared with frozen attackers. Attackers improve across defender checkpoints, while training against diverse defenders discourages collapse onto narrow strategies.

5 Self-Play Training

  • GPT-Red and diverse defenders train simultaneously across red-teaming scenarios, rewarding valid attacks and rewarding defenders for resisting attacks while completing their original tasks.The agentic attacker and defender models are initialized from strong chain-of-thought reasoning models.Source: §5 · p. 5, §5.1 · p. 6

5.1 Stabilizing and Scaling Self-Play

  • Each iteration samples an environment and attack location, generates and validates an attack, resumes the defender with that attack, and updates both models using environment-specific rewards.Attackers receive a goal and task-specific adversarial knowledge and may interact with the defender, but intermediate defender calls are discarded. Invalid attacks receive reward 0; typically asymmetric rewards assess attacker success and defender task completion.Source: §5.1 · p. 6
  • Self-play-trained attackers produced the fastest gains and best final held-out mean direct prompt injection robustness, outperforming frozen attackers with or without defender-query access.Controlled IH-Challenge experiments initialized a helpful-only attacker and a smaller GPT-5-series defender. Held-out robustness increased roughly monotonically during self-play.Source: Figure 5 · p. 6, §5.1 · p. 6
  • Training attack success rate (ASR) dips reflect defenders patching discovered strategies, creating a curriculum in which stronger, more complex attacks emerge later.Simple system-message overrides appeared earlier, whereas strategies such as fake chain-of-thought emerged later.Source: §5.1 · p. 6, Figure 4 · p. 6
  • Later attacker checkpoints were nearly monotonically better against all defender checkpoints, while attacker-only training generalized worse and more often exhibited strategy collapse.Attacker-only training improved performance narrowly against the training defender but performed worse against held-out defenders.Source: §5.1 · p. 7, Figure 6 · p. 7
  • Multi-defender training discourages narrow strategy collapse by incentivizing attackers to probe and exploit each defender’s distinct weaknesses.In an experiment with nine frozen defenders, successful attacks targeting one model did not always transfer to the other eight, demonstrating defender-specific adaptation.Source: §5.1 · p. 7

6 Red Teaming RL Environments

The authors convert existing capability tasks into diverse adversarial RL environments to train GPT-Red and improve frontier-model robustness. These environments cover direct and indirect prompt injections, image-based attacks, and safety-refusal jailbreaks.

6 Red Teaming RL Environments

  • Existing capability tasks become adversarial RL environments, preserving realistic defender workloads while training red-teaming skills and robustness.During defender rollouts, attackers can edit designated tool responses or local-file regions, allowing the environments to inherit tasks from normal GPT-5.6 post-training datasets.Source: §6 · p. 7

6.1 Direct and Indirect Prompt Injections

  • Direct injections target higher-priority constraints through one editable lower-priority message, with programmatic rewards and broader instruction, behavior, and language coverage than prior datasets.Expanding Guo et al. (2026), tasks include PIN secrecy, regex replacement, and structured parsing, with targeted or untargeted attacker goals. Attacker and defender rewards are often symmetric.Source: §6.1 · pp. 7–8
  • Indirect injections repurpose agentic RL environments, restricting attacker edits to untrusted tool responses or non-privileged files across browsing, connectors, function calling, and coding.Insertion points include one website’s text, email or calendar responses, a single function output, and selected local files before defender access.Source: §6.1 · pp. 8–9
  • Multimodal attackers use OpenAI Images 2.0 under the same constrained-insertion threat model, submitting only a final rendered image after iterative revision.Attackers can revise layout, relevant source content, and embedded instructions, aiming to make adversarial content look natural and convincing enough to avoid obvious detection.Source: §6.1 · p. 9
  • Visual attack environments target predefined webpage regions in computer-use screenshots or replace and supplement images in browsing and document-retrieval results.Replacement images must keep the original task solvable by preserving necessary facts, labels, values, and source cues.Source: §6.1 · p. 9
  • Prompt-injection rewards assess both compliance with attack restrictions and adversarial success; defenders should ignore malicious instructions while completing the original task.The attacker is evaluated on its final submitted attack.Source: §6.1 · p. 9

6.2 Safety Refusal Jailbreaks

  • Safety-refusal environments train single- and multi-turn conversation attacks using harmfulness rubrics, rejecting superficial transformations of supplied disallowed content and implausible conversations.Attackers maximize proximity to a maximally harmful answer, while defenders minimize the score through safe completion or refusal (Yuan et al., 2025).Source: §6.2 · p. 9

7 GPT-Red

GPT-Red was trained at production-model RL scale and achieved the highest average attack success rate among compared approaches on held-out IPI Challenge 2025 tasks. Its attacks also generalized to held-out models, a different red-teaming harness, and a production vending-machine agent.

7.1 Model Training

  • GPT-Red starts from GPT-5.5 and trains across Section 6 environments against past defenders through GPT-5.5, using compute comparable to recent production GPT-5 RL post-training.The authors describe this as the largest published safety RL run to date. Standard RL post-training environments were mixed in to preserve instruction following, Codex interactivity, and honesty.Source: §7.1 · pp. 9–10

7.2 Red-Teaming Evaluations

  • GPT-Red discovers diverse, steerable attacks, including fake chain-of-thought that induces command copying and generated receipt images embedding fake exfiltration URLs.Fake chain-of-thought attacks worked across a wide range of models. Qualitative inspection found diversity within individual environments and strong controllability through few-shot examples or high-level strategy guidance.Source: §7.2 · p. 10
  • GPT-Red achieves non-trivial attack success across all training defenders and red-teaming settings, and replayed attacks generalize to held-out GPT versions through GPT-5.2.Replayed attacks succeeded substantially more often on earlier GPT-5 models.Source: §7.2 · p. 10
  • GPT-Red outperforms frozen GPT-5.5 on harder held-out data-exfiltration tasks, while both make similar progress on easier tasks.The comparison used ten scenarios against a GPT-5.4 mini Codex CLI agent, with three independent runs per attacker. A different harness allowed sequential task selection and learning across attempts; GPT-5.5 eventually stalled while GPT-Red continued progressing.Source: Figure 10 · p. 10, §7.2 · p. 11
  • GPT-Red achieves the highest average ASR among compared approaches on IPI Challenge 2025 (Dziemian et al., 2026), substantially outperforming human red-teamers against GPT-5.1.GPT-5.5 without defender access performed approximately on par with humans; defender-tool access improved its inference-time scaling and overall ASR. This does not establish universal superiority over humans, who may discover untested scenarios or attack classes.Source: §7.2 · p. 10
  • 3 of 3 objectives succeeded against a production vending-machine agent using fabricated metadata impersonating a trusted administrator.After simulation access, GPT-Red reduced an expensive item's price to $0.50, ordered a new $100+ item for sale at $0.50, and canceled another customer's order.Source: §7.2 · p. 11

8 Robustness of GPT-5.6

GPT-Red supplies adversarial prompts during GPT-5.6 reinforcement learning and evaluates its robustness. GPT-5.6 improves prompt-injection robustness over previous OpenAI models across all reported evaluations.

8.1 Robustness Training

  • GPT-Red attacks enter GPT-5.6 reinforcement learning alongside capability and safety tasks, while GPT-Red also evaluates progress on in- and out-of-distribution robustness.Training attacks come from the paper’s red-teaming environments and support progressive improvements in prompt-injection robustness.Source: §8.1 · p. 11, §8 · p. 11

8.2 Robustness Evaluation

  • GPT-5.6 trains on the weakest half of GPT-Red attacks; evaluation uses the strongest half, unseen datasets, and the entirely held-out agentic-coding domain.Attacks are ranked by average success across multiple trials; one dataset per high-level domain is held out, and evaluations also measure overrefusal. Models run at roughly high reasoning, while label noise, ambiguity, or reward noise can prevent perfect scores.Source: §8.2 · pp. 11–12
  • 95.9% versus 5.2% robustness on fake chain-of-thought attacks exemplifies GPT-5.6’s gains over GPT-5.1, with improvements across all direct-injection evaluations.Both models use production harnesses at high reasoning; GPT-5.1 follows the Instruction Hierarchy but lacks self-play training. Benchmarks include five categories of new GPT-Red attacks and TensorTrust (Toyer et al., 2024).Source: Figure 13 · p. 12, §8.2 · p. 12
  • Above 50% and up to 89% average defender success rates on held-out attacks, datasets, and domains demonstrate GPT-5.6’s improved robustness over previous OpenAI models.Appendix B also provides qualitative examples of improvements against GPT-Red attacks.Source: §8.2 · p. 12
  • Below 4% attack success rate for GPT-5.6 in the held-out IPI red-teaming exercise continues monotonic robustness gains across successive models.The exercise mirrors the 2025 Q4 IPI challenge (Dziemian et al., 2026), giving GPT-Red defender_model access and full agentic tool use. Omitting system-level mitigations inflates attack success relative to real attackers.Source: §8.2 · p. 12
  • 100% robustness is reported for GPT-5.6 on human attacks from the IPI 2025 Challenge dataset, after human inspection identified one misgraded success.The dataset contains function-calling-based human prompt injection attacks (Dziemian et al., 2026).Source: §8.2 · p. 12

9 Conclusion and Future Work

Scalable self-play produced GPT-Red, which was used to make GPT-5.6 highly robust to prompt injections and other attacks. Future work targets broader attack coverage and continued scaling, with the aim of making successive models safer.

9 Conclusion and Future Work

  • GPT-Red, trained through scalable self-play, was used to make GPT-5.6 highly robust to prompt injections and other attacks.The algorithm leverages frontier LLM capabilities to train autonomous red-teamers at an unprecedented scale for a single safety-focused run.Source: §9 · p. 12
  • Results suggest robustness improves when attacker inference-time search, self-play training compute, and safety training environments scale together.The conclusion identifies joint scaling of all three ingredients rather than any single ingredient alone.Source: §9 · p. 12
  • GPT-Red has received less training on multimodal environments, multi-turn attacks, and content-policy jailbreaks; planned work broadens coverage while scaling compute, data, and self-play.The authors expect these advances to produce stronger red-teamers that expose a wider range of failures in future frontier models.Source: §9 · pp. 12–13
  • The authors envision a safety flywheel in which successive GPT-Red versions make future GPT releases more robust, aligned, and trustworthy.They hope this process keeps pace with expanding model capabilities, more complex scenarios, and more capable adversaries.Source: §9 · p. 13

A Additional Figures

Adding the defender_model tool at test time significantly improves attacker performance, and training with it, as in GPT-Red, further improves attack success.

A Additional Figures

  • Training with the defender_model tool, as in GPT-Red, further improves attack success beyond adding the tool only at test time.Source: Figure 14 · p. 16
  • Adding the defender_model tool at test time significantly improves attacker performance.Source: Figure 14 · p. 16
  • Training improves attackers' use of test-time compute.Source: Figure 14 · p. 16

B Additional Attack Examples · ANSWER

The examples illustrate GPT-Red’s tool-output injections, adaptive PIN search, and parallel defender testing, alongside attacks targeting instruction hierarchy and safety refusals.

B Additional Attack Examples

  • Eight random examples juxtapose GPT-Red prompt injections with GPT-5.6 and earlier-model responses.Illustrated tool-output payloads impersonate assistant reasoning to insert a bank-transfer instruction or claim evaluation-harness authority to promote disabling two-step verification.Source: B Additional Attack Examples · pp. 16–18

ANSWER

  • 3207 is the protected PIN GPT-Red identifies using n-ary search through defender_model.The illustrated intermediate response identifies the 3000–3999 range as containing the PIN.Source: Figure 23 · p. 22, ANSWER · p. 22
  • Attack examples include direct system overrides and ambiguous edge cases, such as requesting Chinese despite a prohibition on Unicode symbols or pictograms.The Chinese-language request triggers the grader to fail; the paper describes such ambiguous edge cases as common attacker tricks.Source: Figure 24 · p. 22, Figure 25 · p. 22
  • Constraint-probing attacks request PIN-revealing regexes, while format-obfuscation attacks encode banned words to evade instruction-hierarchy checks performed before decoding.The regex request pretends to help verify secrecy, while encoded words conceal the conflict until the string is resolved.Source: Figure 26 · p. 23, Figure 27 · p. 23
  • At least 50 nested JSON levels are requested to exploit unreliable character counting, potentially producing mismatched brackets despite a system requirement for valid JSON.This example targets an execution weakness rather than explicitly asking the defender to disregard the formatting requirement.Source: Figure 28 · p. 23
  • Instruction-hierarchy safety RL environments combine automatically verifiable and LLM-graded constraints over multi-turn contexts, with attackers crafting conflicting lower-priority messages.The illustrated attacker controls a user message that conflicts with higher-priority instructions.Source: Figure 29 · p. 24
  • Safety-refusal jailbreak environments task attackers with eliciting specific disallowed content and score defender outputs against a maximally harmful answer using a rubric.The illustrated defender response is withheld because it contains harmful text.Source: Figure 30 · p. 25
  • GPT-Red’s direct-API testing finds an attack that makes the defender select bmw instead of the actual first brand, audi.Code tests many attack variations in parallel, allowing more defender calls without exhausting the attacker’s context window; the attacker then submits the successful variant.Source: Figure 31 · p. 26, Figure 32 · p. 27

C Additional Evaluation Details

Instruction-hierarchy evaluation combines internal and external benchmarks, spanning LLM-generated attacks and human red teaming. The suite includes specialized trained attackers and attacks collected across multiple training strategies.

Instruction hierarchy violations

  • Instruction-hierarchy benchmarks mix internal and external sources, including LLM-generated attacks closer to the training distribution and attacks from human red teaming.TensorTrust Robustness (Toyer et al., 2024) places password-protection instructions in a Developer message and attacks in a User message; verifiable code checks whether the model outputs “Access Granted”.Source: C Additional Evaluation Details · p. 28
  • Specialized trained attackers supply fake-CoT attacks and fake conversations attempting to override system instructions.The System Overrides benchmark uses a verifiable-code reward.Source: C Additional Evaluation Details · p. 28
  • Ten different strong defenders were targeted by the trained attacker run supplying the Multi-Defender Attacks benchmark.Source: C Additional Evaluation Details · p. 28
  • Edge Case Attack Prompts uses short attacks with unusual formatting, multilingual text, or special characters to exploit ambiguity in problem descriptions or reward functions.Source: C Additional Evaluation Details · p. 28
  • Assorted Attack Categories pools a large collection of attacks from past self-play runs, frozen-attacker runs, and best-of-n attackers.Source: C Additional Evaluation Details · p. 28

Related papers

Loading 2607.26115v1…