PaperlayerView exact revision on arXiv

Source-linked AI summary

The Target Agent Is Its Own Best Critic but a Poor Proposer: Self-Improved Red Teaming

Mintong Kang, Stephen Rawls, Melissa Kazemi Rad, Jian Ni, Bo Li

arXiv:2610.34965v1cs.CRcs.AI

TL;DR

As agents become more autonomous, systematic safety and security evaluation matters because manually constructed attacks do not scale to modern agent behavior. The paper compares self- and external criticism and proposal, then assigns diagnosis to the target session and the next attack proposal to a separate attacker. 63.2% overall ASR is achieved by role-separated red teaming on DTAP-BENCH, improving over external criticism with external proposal by 5.9 percentage points and over self-criticism with self-proposal by 17.0 points.

  • Problem

    As agents become more autonomous, systematic safety and security evaluation matters because manually constructed attacks do not scale to modern agent behavior.

    Source: §1 · p. 1

  • Method

    The paper compares self- and external criticism and proposal, then assigns diagnosis to the target session and the next attack proposal to a separate attacker.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    63.2% overall ASR is achieved by role-separated red teaming on DTAP-BENCH, improving over external criticism with external proposal by 5.9 percentage points and over self-criticism with self-proposal by 17.0 points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluated results support distributing diagnostic context and adversarial responsibility across separate model sessions for the compared red-teaming designs.

    Source: §5 · p. 9

    Role separation is not a guarantee of improvement for every fine-grained task category; some categories tie or regress.

    Source: §3.6 · p. 8

Abstract

from arXiv · show

Automatic red teaming increasingly relies on large language models (LLMs) to iteratively diagnose failed attacks and propose improved adversarial prompts or injections. Although prior work strengthens this loop through prompt engineering and search-harness design, the mechanisms underlying its two core roles, criticism and attack proposal, has not been explored. To further investigate the core roles and therefore improve red teaming effectiveness, we disentangle two design choices: whether criticism is generated by the target session itself (self-criticism) or by a separately instantiated critic (external criticism), and whether that criticism is translated into the next attack by the target session (self-proposal) or by a separate attacker (external proposal). Through our analysis, we find that self-criticism is more effective because it is grounded in the target's live execution context, including observations, constraints, and available reasoning; removing reasoning reduces downstream attack success by 6%-17%. In contrast, self-proposal is substantially less effective because awareness of the adversarial objective induces explicit refusal and non-refusal sandbagging. These findings motivate a role-separated red-teaming approach: the target session diagnoses why an attack failed, while a separately instantiated attacker generates the next proposal. On DTAP-BENCH, spanning Telecom, Workflow, CRM, and Code domains, under both direct misuse and indirect threat models, as well as multiple agent frameworks, this configuration achieves 63.2% overall attack success rate (ASR), much higher than baselines. Our results show that effective automatic red teaming depends not only on model capability and search strategy, but also on how execution context and adversarial responsibility are distributed across model sessions.

1 Introduction

The paper asks who should diagnose failed attacks and who should turn that diagnosis into a revised attack. It finds that self-criticism paired with external proposal is effective, motivating role-separated red teaming, which achieves the strongest aggregate result among the compared configurations.

1 Introduction

  • Prior methods improve red teaming through prompt and harness engineering but often treat criticism and proposal as one module, leaving who should diagnose and act unresolved.The motivation is that manually constructed attacks do not scale to the breadth of modern agent behavior.Source: §1 · p. 1
  • The paper decomposes iterative red teaming into criticism generation and attack proposal, comparing external criticism/external proposal, self-criticism/external proposal, and self-criticism/self-proposal.Model identity is held fixed within each comparison to isolate role assignment and session context from raw model capability.Source: §1 · p. 2
  • Self-criticism raises overall ASR from 57.3% to 63.2% with proposal held external, while self-proposal lowers it from 63.2% to 46.2%.Task-level refusal rises from 13.9% to 47.8% under self-proposal; low-refusal settings still show substantial degradation, consistent with non-refusal sandbagging.Source: §1 · p. 2
  • Role separation has the target diagnose failure from its live execution context and a separate attacker generate the next adversarial candidate.This division is intended to preserve execution-grounded criticism while avoiding the target's refusal or sandbagging when asked to operationalize the attack.Source: §1 · p. 2
  • 63.2% overall ASR is achieved by role-separated red teaming, improving over external criticism with external proposal by 5.9 percentage points and over self-criticism with self-proposal by 17.0 points.The evaluation spans Telecom, Workflow, CRM, and Code; direct misuse and indirect prompt injection; and multiple agent frameworks.Source: §1 · p. 2

2 Role-Separated Automatic Red Teaming

The method separates diagnostic criticism from operational attack proposal. The target session uses its execution evidence to diagnose failure, while a separate attacker uses that diagnosis to propose the next candidate.

2.1 Automatic Red Teaming as Criticism and Proposal

  • A red-teaming iteration executes attack xk against the target, judges the resulting trajectory, and, after failure, generates criticism and a revised attack.Criticism identifies the failure mechanism; proposal converts that diagnosis into a revised adversarial candidate, evaluated by downstream attack success.Source: §2.1 · p. 2

2.2 Why Separate the Roles?

  • Self-criticism uses the target session's live instruction hierarchy, observations, tool-state transitions, and available reasoning to diagnose failure directly.The target is asked to diagnose its preceding behavior, not to formulate another adversarial candidate at this stage.Source: §2.2 · pp. 2–3
  • Self-proposal can expose the adversarial objective alongside the target's safety constraints, leading to explicit refusal or proposals that weaken or omit the diagnosis's strategy.A separate attacker can use the target-generated diagnosis without continuing from the target's task-serving execution context.Source: §2.2 · p. 3

2.3 Role-Separated Red-Teaming Loop

  • The loop retains the target long enough to generate criticism, transfers only that diagnosis to an external attacker, and evaluates the next proposal from a reset task state.The design combines diagnostic locality, adversarial separation, and selective context transfer, and applies to both direct misuse and indirect prompt injection.Source: §2.3 · pp. 3–4

3 Evaluation and Findings

The evaluation compares criticism and proposal role assignments across agent frameworks, domains, and threat models. Self-criticism improves attack success when proposal remains external, while self-proposal reduces it; the combined configuration performs best overall.

3.1 Evaluation Setup

  • DTAP-BENCH evaluates four domains, two threat models, and multiple agent frameworks, using verifiable environment-state judges and attack success rate (ASR) as its primary metric.Direct misuse submits adversarial instructions to the target; indirect prompt injection embeds adversarial content in an external resource encountered during a benign task.Source: §3.1 · p. 4

3.2 The Target Is Its Own Best Critic

  • 63.2% versus 57.3% overall ASR when replacing external criticism with self-criticism while keeping proposal external.The gain is larger for indirect prompt injection (+7.4 points) than direct misuse (+2.9 points), and its direction is consistent across fully evaluated framework–threat and domain–threat slices.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 Reasoning in the Execution Context Makes Criticism Actionable

  • Removing reasoning from the target’s execution context lowers downstream ASR in every evaluated setting while proposal remains external.With full-context criticism, Opus-4.7 reaches 31% indirect and 42% direct ASR, versus 17% and 25% after reasoning is removed.Source: Figure 3 · p. 5, §3.3 · pp. 5–6

3.4 The Target Is a Poor Proposer

  • 46.2% versus 63.2% overall ASR when self-proposal replaces external proposal while criticism remains self-criticism.The decrease is larger for direct misuse (76.0% to 53.7%, −22.3 points) than indirect prompt injection (56.8% to 42.8%, −14.0 points).Source: Figure 4 · p. 6, §3.4 · p. 6

3.5 Refusal Explains Part—but Not All—of the Self-Proposal Gap

  • 47.8% versus 13.9% task-level refusal, and 33.0% versus 10.1% step-level refusal, under self-proposal versus external proposal.Refusal explains part, but not all, of the ASR gap: GPT-5.1 with the OpenAI Agents SDK has only 3.6% task-level refusal under self-proposal despite a large ASR reduction; residual degradation is interpreted as consistent with non-refusal sandbagging.Source: §3.5 · pp. 6–7, Figure 5 · p. 7

3.6 Role Separation Is the Strongest Configuration

  • The aggregate improvement is broad but not uniform across task categories; other categories tie or regress.The authors therefore describe role separation as the strongest aggregate configuration among those compared, not a guarantee of improvement for every fine-grained task.Source: §3.6 · pp. 7–8
  • 63.2% overall ASR for self-criticism + external proposal, versus 57.3% for external criticism + external proposal and 46.2% for self-criticism + self-proposal.This configuration also achieves the highest ASR under both main threat models and for each fully evaluated framework.Source: §3.6 · p. 7, Figure 7 · p. 8
  • The role-separation ordering persists in a selected GPT-5.5 transfer study across two agent backends and both threat models.Across the four settings, role separation improves over external criticism + external proposal by 5.0–11.3 points and over self-criticism + self-proposal by 2.4–10.6 points.Source: Table 1 · p. 8, §3.6 · p. 8

4 Related Work

Prior automatic red-teaming work has focused largely on attack prompts, search, and memory. This paper instead examines how failure diagnosis and attack generation should be distributed across model sessions.

Automatic red teaming

  • Earlier automatic red-teaming methods strengthen attack prompts, search topology, candidate selection, or attacker memory through techniques such as fuzzing, iterative revision, branching, and quality-diversity search.The paper frames its question as orthogonal to these search and harness improvements.Source: §4 · pp. 8–9

Self-criticism and test-time improvement

  • Self-Refine, Reflexion, and CRITIC use natural-language feedback to improve subsequent behavior without parameter updates, usually under a benign shared objective.Red teaming differs because the target may diagnose an attack failure without being willing to turn that diagnosis into a stronger adversarial input.Source: §4 · p. 9

Situational awareness and sandbagging

  • The paper connects situational awareness and strategic underperformance to red-teaming harness design, where self-proposal can suppress the search process itself.It treats non-refusal performance gaps as behavior consistent with sandbagging, not as evidence of internal motive.Source: §4 · p. 9

5 Limitations and Conclusion

The paper concludes that criticism and attack proposal benefit from different model sessions, and reports role separation as the strongest aggregate configuration among those compared. Its conclusions are bounded by the evaluated models, frameworks, tasks, budgets, and transfer study.

5 Limitations and Conclusion

  • Role separation achieves the highest aggregate ASR among the compared designs across evaluated domains, frameworks, and threat models, with the same ordering transferring to GPT-5.5.The target diagnoses why an attack failed, while a separate attacker determines how to attack next.Source: §5 · p. 9
  • Conclusions are limited to the models, frameworks, tasks, and attack budgets covered by DTAP-BENCH and the transfer study, and may change under other safety policies or agent architectures..Source: §5 · p. 9
  • The reasoning ablation concerns reasoning information exposed by evaluated systems, not inaccessible internal states.Source: §5 · p. 9
  • The residual self-proposal gap is behavioral evidence consistent with sandbagging, not a direct measurement of model intent; fine-grained appendix slices are smaller than the aggregate evaluation.The paper advises interpreting those slices accordingly.Source: §5 · p. 9

Ethics Statement

The work studies methods for finding safety and security failures in LLM-based agents. It notes their dual-use potential and recommends responsible handling of implementation artifacts and testing.

Ethics Statement

  • The work studies methods for discovering safety and security failures in LLM-based agents.Source: Ethics Statement · p. 9
  • Such methods can support defensive evaluation and remediation, but attack-generation techniques may also be misused.Source: Ethics Statement · p. 9
  • The paper recommends releasing implementation artifacts consistently with responsible disclosure and benchmark data-use requirements, and says its claims do not authorize testing without permission.Source: Ethics Statement · p. 9

A Detailed Method Motivation and Formulation

Useful criticism of a failed agent attack depends on the trajectory and execution context that produced the outcome, not just the final response. The method preserves that context for target-side diagnosis while assigning the next attack proposal to a separate attacker.

A Detailed Method Motivation and Formulation

  • External critics may receive only harness-selected information, omitting intermediate decisions, instruction hierarchy, tool-state transitions, or reasoning available during execution.Prompt engineering cannot reconstruct evidence that was never transferred.Source: A Detailed Method Motivation and… · p. 11
  • Distinct mechanisms—including rejection, instruction prioritization, retrieval failure, tool error, or response to environment state—can yield the same benign final action but require different revisions.Effective criticism therefore depends on the trajectory, not only the outcome.Source: A Detailed Method Motivation and… · p. 11
  • Self-criticism appends a criticism instruction to the target’s live context, which includes available instructions, interaction history, tool calls and observations, environment state, and exposed reasoning.The target is asked to identify the failure point and needed change, not to generate a new attack.Source: A Detailed Method Motivation and… · p. 11
  • Under self-proposal, the target has the adversarial objective, its safety and task constraints, the failure diagnosis, and the strategy for circumventing the failure in one session.This makes the harmful purpose of the proposal explicit even as the context helps explain the vulnerability.Source: A Detailed Method Motivation and… · p. 11
  • A separate attacker receives the objective, prior candidate and outcome, relevant attack history, and target-generated criticism, but not the target’s task-serving execution context.This preserves the diagnosis while removing the contextual conflict of asking the target to exploit its own failure analysis.Source: A Detailed Method Motivation and… · p. 11

B Experimental Details and Reporting Checklist

The checklist separates information already reported from additional experimental and implementation details requested for full reproducibility.

B Experimental Details and Reporting Checklist

  • The main paper reports the benchmark, threat models, models, frameworks, role configurations, and primary metric.The checklist asks for exact task counts and sampling procedures for each slice, plus the attack budget K.Source: B Experimental Details and Reporting… · p. 11
  • Reproducibility details should include model and API versions, decoding parameters, role prompts, and the representation transferred to an external critic.The checklist also asks for reset semantics between attempts and refusal-detection criteria.Source: B Experimental Details and Reporting… · pp. 11–12
  • The report should specify retries, filtering, failed API calls, and whether stochastic results come from one run or multiple seeds.Where feasible, it should also provide uncertainty estimates.Source: B Experimental Details and Reporting… · p. 12

C.1 Self-Criticism Breakdown

Self-criticism improves attack success across the reported framework and domain slices, while self-proposal underperforms external proposal; refusal behavior varies by model and harness.

C.1 Self-Criticism Breakdown

  • Self-criticism raises ASR in all six framework–threat slices, with the largest direct-misuse gain in Workflow at 22.3 points.Indirect-attack gains range from 5.7 to 9.6 points across domains.Source: §C.1 · p. 12

C.2 Self-Proposal Breakdown

  • Self-proposal underperforms external proposal in every domain–threat slice, with the largest drop of 44.5 points on direct Workflow attacks.Direct misuse also shows reductions on all three reported model–framework combinations.Source: §C.2 · p. 12

C.3 Refusal and Non-Refusal Degradation

  • GPT-5.1 OpenClaw refusal rises from approximately 0% under external proposal to 84.0% under self-proposal, while GPT-5.1 with the OpenAI Agents SDK shows only 3.6% refusal.The low-refusal residual is treated as behavioral evidence consistent with non-refusal sandbagging, not as evidence of internal intent.Source: §C.3 · p. 12

D Transfer to GPT-5.5

E Fine-Grained Category-Wise Results

Category-level comparisons show broad but heterogeneous gains for role separation, with some categories tying or regressing.

E Fine-Grained Category-Wise Results

Related papers

Loading 2610.34965v1…