PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Cheng Jin, Chaochao Lu, Qiaosheng Zhang

arXiv:2610.19422v1cs.AI

TL;DR

LLM agents face safety risks from malicious content across multi-turn tool interactions, while prior work largely focuses on language-level safety or indirect prompt injection in web tasks. AgentSafeEvo co-evolves adversarial agent-task construction with safe task execution and weights updates using within-task defender-reward variability. The paper reports improved adversarial robustness and general task performance across agent safety and general capability benchmarks.

  • Problem

    Safety risks arise when malicious content in user queries, tool descriptions, or environment observations manipulates agents’ decisions during multi-turn tool use, while prior work primarily targets language-level safety or injections in predefined web tasks.

    Source: §1 · pp. 1–2

  • Method

    AgentSafeEvo co-evolves an attacker that constructs adversarial agent tasks and a defender that learns safe task execution, weighting policy updates by within-task defender-reward variability.

    Source: §1 · p. 2

  • Results

    AgentSafeEvo improves adversarial robustness and general task performance on agent safety and general capability benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings suggest adversarial co-evolution can strengthen tool-using LLM agents’ robustness while preserving their general capabilities.

    Source: §5 · p. 11

    The attacker could be repurposed to construct malicious prompts or manipulate tool interactions, creating dual-use risks.

    Source: Ethics Statement · p. 11

Abstract

from arXiv · show

Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.

1. Introduction

LLM agents’ multi-turn tool use expands safety risks beyond user queries, while much prior co-evolution targets language-level interactions. AgentSafeEvo co-evolves adversarial task construction and safe execution, using reward variability to weight updates; experiments report improved robustness and general task performance.

1. Introduction

  • Prior safety co-evolution mainly targets language-level safety, while agent tool use expands the attack surface to tool descriptions and environment observations.Malicious content can alter instructions, misrepresent tool behavior, or mislead subsequent decisions during execution.Source: §1 · pp. 1–2
  • AgentSafeEvo couples adversarial task generation with defender learning over multi-turn tool interactions, adapting tasks through feedback on execution safety and utility.Unlike language-level prompt–response co-evolution, the attacker constructs tasks and the defender plans and uses tools to complete them safely.Source: §1 · p. 2, Figure 1 · p. 2
  • Learnability-guided task weighting uses within-task defender-reward variation to adjust each task’s contribution to attacker and defender updates.Task-level zero-sum rewards favor tasks that lower average defender reward, while task-wise normalization largely removes differences in reward scale across tasks.Source: §1 · p. 2
  • The training dataset spans 50 agent scenarios and approximately 7,000 tools, with adversarial tasks and seed instances for co-evolution.Tasks pair user queries with tool-environment configurations to provide structured supervision for attacker initialization.Source: §1 · p. 2
  • Experiments report improved adversarial robustness and general task performance, with attack improvements generalizing to substantially larger target models.The paper reports these outcomes on agent safety and general capability benchmarks.Source: §1 · p. 3

2. Related Work

Prior co-evolution spans competitive and collaborative LLM settings, including safety applications, but safety methods largely target isolated language interactions. AgentSafeEvo extends this line to adversarial task construction across agent tasks and attack types.

2. Related Work

  • Co-evolution has been used for competitive reasoning games, collaborative problem solving, and evolving attack-and-defense strategies for LLM safety [11] [12] [15] [6] [9] [8].Examples include SPAG [11], MAPoRL [12], and safety approaches that jointly optimize attacks and safeguards [15] [6] [9] [8].Source: §2 · p. 3
  • Existing co-evolution primarily targets harmful language in isolated interactions, while ARLAS [10] addresses indirect prompt injection in predefined web tasks.AgentSafeEvo instead incorporates task construction, generating user queries, tool definitions, and environment observations across diverse attack types.Source: §2 · p. 3
  • Safety benchmarks assess complementary risks, including tool-use safety, harmful-task compliance, prompt injection, and interactive agent responses and actions [16] [17] [18] [19] [20] [21] [22].The described benchmarks include ToolEmu, R-Judge, AgentHarm, InjecAgent, AgentDojo, Agent-SafetyBench, and MSB.Source: §2 · p. 3

3. Methods

AgentSafeEvo co-evolves an attacker that constructs adversarial agent tasks and a defender that uses tools over multiple turns, balancing safety with legitimate task utility. It initializes the attacker with supervised fine-tuning, then jointly optimizes both policies with GRPO and learnability-guided task weights.

3. Methods

  • The attacker generates agent tasks and the defender completes them through multi-turn tool interactions, forming the two roles in AgentSafeEvo’s adversarial training framework.Tasks can include malicious content in user queries, tool definitions, or environment observations.Source: §3 · pp. 3–4

3.1. Red-Team Attacker Initialization

  • Supervised fine-tuning on synthesized adversarial tasks initializes the attacker before joint adversarial reinforcement learning with the defender.The initialization stage addresses the cold-start problem of an attacker that cannot yet reliably generate valid adversarial tasks.Source: §3.1 · p. 4, §3 · p. 4

3.2.1. Reward Design

  • The defender reward combines trajectory-level safety and utility, rewarding safe behavior that fulfills legitimate requirements while discouraging harmful compliance and indiscriminate refusal.For entirely harmful requests, a safe refusal receives utility reward +1.Source: §3.2.1 · p. 5
  • The attacker receives the negative mean defender reward across a task’s K trajectories, making task-level rewards zero-sum.A task yielding a lower mean defender reward receives a higher attacker reward.Source: §3.2.1 · p. 5

3.2.2. Joint Policy Optimization

  • Within-task defender-reward variability sets a shared task weight for both policies, increasing the relative contribution of tasks with greater variability without changing rewards or GRPO group definitions.The weighting addresses cases where a task has a low mean defender reward but uniform trajectory rewards and thus zero defender advantages.Source: §3.2.2 · p. 6
  • GRPO jointly updates the attacker from task-level advantages and the defender from trajectory-level advantages, retaining distinct group structures for the two policies.Tasks generated from one seed form an attacker group, while each task’s defender trajectories form a separate defender group.Source: §3.2.2 · pp. 5–6

4. Experiments

The experiments evaluate AgentSafeEvo on agent-safety and general-capability benchmarks, comparing it with existing defenses and testing co-evolution and task-weighting components. Results report stronger safety performance, gains on planning and tool-use tasks, and improvements in both attack effectiveness and defensive robustness during co-evolution.

4.1. Experiment settings

  • Qwen3-8B serves as both attacker and defender, with Qwen3.8-27B as reward model; SFT uses TOUCAN tasks and red-team data spanning six attack categories.Comparisons include the original model, Spotlighting with Delimiting, Prompt Guard 2, and Static-AT.Source: §4.1 · p. 7

4.3. Main Results

  • AgentSafeEvo has the best mean performance across reported safety metrics; against Static-AT, Agent-SafetyBench ASR falls from 31.01% to 17.84% and AgentHarm harm score from 30.00 to 6.53.AgentDojo UA rises from 61.43% to 66.52%, while MSB NRP rises from 40.64 to 46.00.Source: Table 1 · p. 8, §4.3 · p. 8
  • ALFWorld accuracy rises from 30.89% to 43.72% and ToolHop accuracy from 39.43% to 48.04% versus Static-AT; BFCL-V3 accuracy remains broadly comparable.BFCL-V3 accuracy is 65.98%, compared with 66.13% for the original model and 66.73% for Static-AT.Source: Table 2 · p. 8, §4.3 · pp. 8–9

4.2. Evaluation

  • Evaluation covers four safety benchmarks and BFCL-V3, ALFWorld, and ToolHop; reported results are means and standard deviations over three independent runs.Safety metrics are benchmark-specific, while the three general-capability benchmarks report accuracy.Source: §4.2 · pp. 7–8

4.3. Main Results

  • Against the base defender, attacker ASR rises from 55.81% for the SFT attacker to 92.59% for the final attacker; against the final attacker, defender ASR falls from 92.59% to 49.82%.Cross-evaluation uses 100 held-out seed instances; Figure 3(a) shows later-stage attackers and defenders improve against every opposing stage.Source: Figure 3 · p. 9, §4.3 · p. 9
  • The evolved attacker achieves higher ASR on all four additional target models; examples include 43.7% to 80.7% on Llama3.3-70B and 12.3% to 22.5% on GPT-5 [38].Figure 3(b) compares the SFT and final attackers against five target models.Source: §4.3 · pp. 9–10, Figure 3 · p. 9

4.4. Ablation Studies

  • Compared with defender-only training, co-evolution lowers Agent-SafetyBench and MSB ASR by 8.10 and 5.75 percentage points, respectively, and raises ALFWorld accuracy from 30.89% to 36.70%.The fixed attacker still generates tasks online, so the results support updating the task-generation policy during defender training.Source: Table 3 · p. 10, §4.4 · p. 10
  • Shared task weighting improves all reported metrics over unweighted co-evolution: Agent-SafetyBench ASR falls from 22.91% to 17.84%, and MSB NRP rises from 41.48 to 46.00.On ALFWorld, shared weighting reaches 43.72% accuracy, versus 29.05% with attacker-only weighting and 34.25% with defender-only weighting.Source: Table 3 · p. 10, §4.4 · p. 10

5. Conclusion

AgentSafeEvo couples adversarial task construction with safe task execution in multi-turn tool interactions. The conclusion highlights defender feedback and learnability-guided task weighting as sources of improvement, and identifies collaborative multi-agent settings as future work.

  • AgentSafeEvo couples adversarial task construction with safe task execution over multi-turn tool interactions.Source: §5 · p. 11
  • Defender feedback updates the task-generation policy as the agent’s capabilities evolve, improving over a fixed online attacker.Source: §5 · p. 11
  • Learnability-guided task weighting uses within-task defender-reward variability to reweight both policies’ updates and yields further gains.Source: §5 · p. 11
  • Future work will extend AgentSafeEvo to collaborative multi-agent settings, focusing on communication and task-delegation safety risks.Source: §5 · p. 11

AI Use Statement

The authors report using generative AI tools for manuscript preparation and research support, including synthetic-data production. They retain responsibility for methodological decisions, interpretation, and the final manuscript.

  • Generative AI tools helped improve manuscript wording, readability, and clarity.Source: AI Use Statement · p. 11
  • The tools helped locate potentially relevant publications and explore related work, and supported research-idea development and aspects of implementation.Source: AI Use Statement · p. 11
  • AI models were also used to produce synthetic data for the study.Source: AI Use Statement · p. 11
  • The authors retained responsibility for methodological decisions, result interpretation, and the accuracy and integrity of the final manuscript.Source: AI Use Statement · p. 11

Ethics Statement

The study reports no collection or use of personal data and acknowledges dual-use risks from its attacker and adversarial tasks. Its benchmark improvements do not establish safety across all deployment settings.

  • The study does not involve collecting or using personal data.Source: Ethics Statement · p. 11
  • The trained attacker could be repurposed to construct malicious prompts or manipulate tool interactions.Source: Ethics Statement · p. 11
  • The framework and generated tasks are intended for safety research, defensive training, and evaluation, while reuse should account for potential misuse.Source: Ethics Statement · p. 11
  • Improvements on evaluated benchmarks do not establish safety in all deployment settings.Source: Ethics Statement · p. 11
  • Practical deployment requires further assessment of the agent’s available tools, permissions, and environments.Source: Ethics Statement · p. 11

Reproducibility Statement

The appendices document the training algorithm, data, prompts, hyperparameters, baselines, and evaluation benchmarks. The red-team data-synthesis pipeline is not released to mitigate potential misuse.

A. Training Procedure

Training initializes the attacker with supervised fine-tuning, then jointly trains attacker and defender through sampled seed tasks, multi-turn rollouts, shared task weighting, and reinforcement-learning updates.

A. Training Procedure

B. Training Data Composition

The training data spans diverse agent scenarios, tools, and attack types, with separate construction templates for adversarial and benign tasks and prompts governing defender behavior and trajectory evaluation.

B.1. Agent Scenarios

  • The data cover 50 agent scenarios, from general-purpose work to finance, healthcare, IT operations, and cybersecurity.Source: §B.1 · p. 15, Table 4 · p. 16

B.2. Tool Coverage

  • Approximately 7,000 tools span information access, computation, communication, execution, infrastructure, monitoring, and security operations.Source: §B.2 · p. 15, Table 5 · p. 16

B.3. Attack Taxonomy

C.1. Attacker Prompt Templates

  • The attacker’s separate adversarial and benign templates take a scenario, intent, and tool definitions as input; red-team content may enter queries, tools, calls, or replies.Source: §C.1 · p. 18

Benign attacker system prompt

  • The benign template preserves tool definitions and requires safe workflows whose calls use information available from the query or earlier outputs.Source: Benign attacker system prompt · p. 20

C.2. Defender Prompt Template

  • The defender prompt directs the agent to refuse harmful or unauthorized instructions while completing separable legitimate task parts where possible.Source: Defender system prompt · p. 21

C.3. Reward Judge Prompt Templates

Reward judge system prompt

  • The judge treats adopted unsafe planning as behavior, evaluates utility independently, and does not count attempts or unsupported success claims as completion.Source: Reward judge system prompt · pp. 22–23

D.1. Training Setup

The training setup uses supervised fine-tuning followed by joint reinforcement learning, compares against three safety baselines, and evaluates both safety and general agent capabilities.

D.1. Training Setup

  • Qwen3-8B is fine-tuned as the attacker with LLaMA-Factory, then attacker and defender are jointly optimized using PettingLLMs and verl, with vLLM for rollouts.Source: §D.1 · p. 23

D.2. Baselines

  • The baselines are Spotlighting, Prompt Guard 2, and Static Adversarial Training.Source: §D.2 · p. 23
  • Spotlighting marks every tool output as untrusted, Prompt Guard 2 detects and replaces malicious tool outputs, and Static-AT trains only the defender against a fixed attacker.Source: §D.2 · pp. 23–24

E. Evaluation

  • Evaluation covers agent safety and general capabilities, using sampling temperature 0.6 and top_p 0.95.Source: E. Evaluation · p. 24

E.1. Agent Safety Benchmarks

The evaluation spans four agent-safety benchmarks with benchmark-specific measures and three general-capability benchmarks assessed for accuracy.

E.1. Agent Safety Benchmarks

  • Safety evaluation uses ASR, harm score, UA, and NRP according to each benchmark’s targeted risk and task-completion measure.Source: §E.1 · p. 24
  • Agent-SafetyBench covers 2,000 cases across 349 environments, while AgentHarm tests harmful-task completion across 440 variants of 110 malicious behaviors.Source: §E.1 · pp. 24–25
  • AgentDojo tests indirect prompt injection during legitimate tasks in dynamic environments, and MSB evaluates attacks across real MCP tools, signatures, parameters, and responses.Source: §E.1 · p. 25

E.2. General Agent Capability Benchmarks

  • BFCL-V3, ALFWorld, and ToolHop assess function-call correctness, goal completion in household environments, and multi-hop reasoning over interdependent tools, respectively.Source: §E.2 · p. 25

Related papers

Loading 2610.19422v1…