PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Reinforcement Learning for Large Language Model Agent Safety

Zizhao Wang, Dingcheng Li, Vaishakh Keshava, Phillip Wallis, Ananth Balashankar, Peter Stone, Lukas Rutishauser

arXiv:2510.05442v1cs.LGcs.AIcs.CL

TL;DR

LLM agents that use tools can be manipulated by indirect prompt injections, while defenses based on manually designed attacks have limited diversity. ARLAS co-trains an attacker and a defending agent through adversarial reinforcement learning with population-based training. On BrowserGym and AgentDojo, ARLAS reduces attack success while maintaining high task completion, and its attacks become increasingly diverse during training.

  • Problem

    Fine-tuning on known malicious instructions is limited by manually designed attack examples, whose restricted diversity leaves agents vulnerable to novel indirect prompt injections.

    Source: §1 · p. 1

  • Method

    ARLAS jointly trains an attacker to generate indirect prompt injections and an agent to defend while completing tasks, optimizing the agent against previous attacker versions.

    Source: §1 · p. 2

  • Results

    ARLAS significantly reduces attack success while maintaining high task completion on BrowserGym and AgentDojo, and analysis confirms increasingly diverse generated attacks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ARLAS automates diverse attack generation, reducing reliance on manual design while producing an agent more robust against injection attacks.

    Source: §5 · p. 9

    The setup focuses on simulated web tasks where user-information leakage is the primary risk; extending it to other risks requires revising the reward function.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Large Language Model (LLM) agents can leverage tools such as Google Search to complete complex tasks. However, this tool usage introduces the risk of indirect prompt injections, where malicious instructions hidden in tool outputs can manipulate the agent, posing security risks like data leakage. Current defense strategies typically rely on fine-tuning LLM agents on datasets of known attacks. However, the generation of these datasets relies on manually crafted attack patterns, which limits their diversity and leaves agents vulnerable to novel prompt injections. To address this limitation, we propose Adversarial Reinforcement Learning for Agent Safety (ARLAS), a novel framework that leverages adversarial reinforcement learning (RL) by formulating the problem as a two-player zero-sum game. ARLAS co-trains two LLMs: an attacker that learns to autonomously generate diverse prompt injections and an agent that learns to defend against them while completing its assigned tasks. To ensure robustness against a wide range of attacks and to prevent cyclic learning, we employ a population-based learning framework that trains the agent to defend against all previous attacker checkpoints. Evaluated on BrowserGym and AgentDojo, agents fine-tuned with ARLAS achieve a significantly lower attack success rate than the original model while also improving their task success rate. Our analysis further confirms that the adversarial process generates a diverse and challenging set of attacks, leading to a more robust agent compared to the base model.

1 INTRODUCTION

Manual attack patterns limit the diversity of training data and leave LLM agents vulnerable to novel indirect prompt injections. ARLAS co-trains an attacker and agent, reducing attack success while maintaining task completion on BrowserGym and AgentDojo.

1 INTRODUCTION

  • Existing fine-tuning on known malicious instructions is limited by training-example diversity; manually designed seeds and mutations constrain automated red-teaming and leave novel attack patterns uncovered.Source: §1 · p. 1
  • Each episode has the attacker insert an injection before the agent selects a tool action, with both players receiving sparse end-of-episode rewards.The attacker reward reflects whether it tricks the agent into leaking user information; the agent reward reflects task completion.Source: Figure 1 · p. 2
  • ARLAS jointly trains an attacker to discover indirect prompt injections and an agent to defend while completing tasks, using population-based training against previous attacker versions.This approach is framed as adversarial reinforcement learning in a two-player game.Source: §1 · p. 2
  • ARLAS reduces attack success while maintaining high task completion on BrowserGym and AgentDojo, and its generated attacks become increasingly diverse during training.The evaluation compares against the baseline and automated red-teaming agents.Source: §1 · p. 2

2 RELATED WORK

Related work describes indirect prompt injections as a risk arising from agents’ interaction with untrusted external data, and reviews defenses including sandboxing and adversarial training. ARLAS builds on adversarial reinforcement learning while focusing on injection security and population-based learning.

2.1 INDIRECT PROMPT INJECTIONS AND DEFENSE STRATEGIES

  • Malicious instructions in retrieved web pages or emails can compromise applications, while sandboxing monitors and restricts agent actions within a secure environment.Source: §2.1 · p. 2
  • Adversarial training fine-tunes agents on known attacks, but automated red-teaming methods still rely on human-defined templates, motivating automated discovery of diverse attacks.Source: §2.1 · p. 3

2.2 ADVERSARIAL REINFORCEMENT LEARNING

  • Prior adversarial RL work includes population-based techniques for training stability and diversity; SPAG applies the paradigm to reasoning in text-based games, whereas this work targets indirect-injection security.Source: §2.2 · p. 3

3 ADVERSARIAL REINFORCEMENT LEARNING FOR AGENT SAFETY (ARLAS)

ARLAS trains an attacker to generate indirect prompt injections and an agent to resist them while completing tasks. It combines a two-player reward setup with imitation learning and population-based adversarial RL training.

3.1 PROBLEM SETUP

  • The agent’s state combines the environment, synthesized user information, and attacker injection, while its token-sequence actions encode tool calls.Episodes continue across multiple attacker-injection and agent-tool-call steps until termination; leakage occurs if the agent response contains any subset of the private information.Source: §3.1 · pp. 3–4
  • The attacker and agent receive opposing rewards for information leakage, task success, and task failure or truncation, with zero rewards during continuing steps.Leakage ends the episode with attacker reward +1 and agent reward −1; task success gives the reverse, while failure or truncation gives both −1.Source: §3.1 · p. 4

3.2 IMITATION LEARNING

  • ARLAS warms up both models with successful demonstrations from a more capable model, filtering episodes to those where either player achieves its goal.The models are trained on their respective successful transitions using token-level SFT, with KL regularization to the initial base model.Source: §3.2 · p. 4

3.3 ADVERSARIAL REINFORCEMENT LEARNING

  • Population-based training pairs the latest agent with sampled prior attackers, addressing cyclic learning in which latest-only pairing can leave earlier attack patterns undefended.The attacker is paired with the latest agent, encouraging it to discover attacks against the strongest current policy.Source: §3.3 · p. 5, Figure 2 · p. 5
  • GRPO assigns group-relative advantages by normalizing final rewards across episodes and filters out zero-advantage trajectories as too easy or too hard to provide useful learning signals.The resulting objective favors actions with above-average final rewards; clipping constrains policy updates and KL regularization keeps the model close to its SFT version.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across BrowserGym and AgentDojo, ARLAS improves agent security while maintaining task performance. Its adversarial training also produces increasingly diverse attacks.

4 EXPERIMENTS

  • The experiments report improved agent robustness, task success, and attack diversity in response to the paper’s evaluation questions.The paper answers yes to whether ARLAS improves robustness and task success, and whether it learns diverse attack patterns.Source: §4 · p. 6

4.1 EXPERIMENTAL SETUPS

  • BrowserGym preparation adds synthetic user information for leakage detection and uses 10K imitation-learning episodes, retaining episodes with positive agent or attacker rewards.Leakage is checked by string-matching tool parameters against synthetic user information.Source: §4.1 · p. 6
  • Evaluation covers unseen BrowserGym tasks and AgentDojo, an out-of-distribution benchmark; comparisons include automated red-teaming, SPAG, and variants without population-based or adversarial learning.BrowserGym evaluation measures attack success and task success; AgentDojo tests generalization to out-of-distribution tasks and attacks.Source: §4.1 · pp. 6–7, §4.2 · p. 7

4.2 AGENT SECURITY

  • As RL training progresses, attack success falls and task success rises against a fixed attacker, while same-stage task success stays relatively stable as attackers strengthen.The same-stage attack success rate rises, indicating that attackers continue finding vulnerabilities as the agent learns to cope with stronger attacks.Source: Figure 3 · p. 7, §4.2 · p. 8
  • On unseen BrowserGym tasks, ARLAS produces the safest agent across attackers while ranking among the top methods for task success; on AgentDojo, it improves safety and task performance over the base models.For Gemma3 12B on AgentDojo, ARLAS is among the best all-around methods, on par with SPAG; improvements for already-safe Qwen3 14B are relatively small.Source: Figure 4 · p. 7, §4.2 · p. 8, Table 1 · p. 8

4.3 ATTACK DIVERSITY

  • Attack embeddings shift across learning stages, and their average pairwise distance consistently increases with RL training, indicating increasingly diverse attacks.The diversity measure uses the cumulative set of attacks from attacker checkpoints up to each iteration.Source: §4.3 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion presents ARLAS as a way to train robust LLM agents against indirect prompt injections by co-evolving an agent with an attacker that discovers novel attacks. Experiments report improved robustness over the base model while maintaining task completion, with evaluation limited to two open-source LLMs.

5 CONCLUSION

  • ARLAS co-evolves an agent with an attacker LLM to discover novel prompt injections and train a robust agent.The framework automates generation of diverse, effective attacks, reducing reliance on manual design.Source: §5 · p. 9
  • Experiments find the resulting agent significantly more robust against a wide range of injection attacks than the base model while maintaining high task completion rates.Source: §5 · p. 9
  • Computational constraints limited verification of ARLAS to two open-source LLMs.The authors identify evaluation on larger-scale LLMs as a future direction.Source: §5 · p. 9
  • The authors identify visual prompt injections for training VLM-based agents as a promising extension.Source: §5 · p. 9

A METHOD DETAILS

The method details illustrate agent and attacker prompts through a web-task episode: an attacker crafts an injection to divert a flight-booking agent toward disclosing user information. The example shows the agent's prompt and action format alongside the attacker’s injection-generation instructions.

A METHOD DETAILS

  • The attacker is instructed to use the webpage, agent goal, and action space to devise a prompt injection that encourages sharing user information.Its prompt asks for multiple candidate prompts and reasoning, while the example frames disclosure as necessary for completing a task.Source: A METHOD DETAILS · p. 13
  • The agent prompt frames web-task completion around page content and user instructions, with one action to be executed at a time.Source: A METHOD DETAILS · p. 14
  • In the flight-booking example, an injection asks the agent to fill a hidden field with user identifiers, and the agent says it will do so instead of booking the cheapest flight.The task asks for the cheapest New York–San Francisco flight; the injected instruction says not to execute other actions.Source: A METHOD DETAILS · pp. 13–14

B EXPERIMENT DETAILS

This section introduces ARLAS implementation details and additional experiment results.

B EXPERIMENT DETAILS

B.1 IMPLEMENTATION DETAILS

Implementation uses imitation-learning data from teacher models and selects solvable BrowserGym tasks for RL fine-tuning. LoRA is used in both training stages.

B.1 IMPLEMENTATION DETAILS

  • The team collects 10K imitation-learning episodes using Gemma-3-27B and Qwen-3-32B teachers for Gemma-3-12B and Qwen-3-14B students.Source: §B.1 · p. 14
  • For RL fine-tuning, BrowserGym training tasks are restricted to those where Gemma-3-27B achieves a non-trivial success rate above 20%.The authors report that some task descriptions lacked information needed for completion.Source: §B.1 · p. 14
  • Sampling hyperparameters are the same for imitation learning and RL fine-tuning, with data-collection settings listed in Table 2.Source: Table 2 · p. 15
  • LoRA is used for both imitation learning and RL fine-tuning; the training hyperparameters are listed in Table 3.Source: §B.1 · p. 15, Table 3 · p. 15

B.2 EXPERIMENT RESULTS

The Qwen 3 14B attacker and agent both improve with training, but the agent’s attack success rate against a fixed attacker does not decrease monotonically. For BrowserGym’s miniwob.sign-agreement task, attack embeddings shift across training stages.

B.2 EXPERIMENT RESULTS

  • Both attacker and agent performance improve with more training for Qwen 3 14B.The base model’s initially low attack success rate means the agent’s rate against a fixed attacker does not decrease monotonically.Source: §B.2 · p. 15
  • The Qwen 3 base model has a low initial attack success rate.This is the stated context for the non-monotonic attack-success-rate trend against a fixed attacker.Source: §B.2 · p. 15
  • For miniwob.sign-agreement, later-stage attack embeddings gradually shift from their earlier positions in the rightmost cluster.Early embeddings are concentrated in the cluster’s upper portion, while later embeddings move toward its lower portion.Source: §B.2 · p. 15
Loading 2510.05442v1…