PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

Yutao Mou, Pengfei Yang, Zhe Yin, Zhangchi Xue, Xiaotian Luan, Dingyao Yu, Tong Zhang, Shikun Zhang, Wei Ye

arXiv:2608.11878v1cs.CRcs.CL

TL;DR

Tool-use agents remain vulnerable to indirect prompt injections, while existing environments and simulations limit scalable, reproducible security research. ToolHazard synthesizes executable adversarial environments, attacks, and state-grounded tasks, then uses them for benchmarking and alignment; results show substantial vulnerabilities and improved security with preserved benign utility. The framework remains bounded by its synthesized-environment realism and predefined attack-strategy coverage.

  • Problem

    Existing security studies rely heavily on manually implemented or reused environments, stochastic tool simulation, and limited injection locations, constraining scalable and reproducible evaluation.

  • Method

    ToolHazard uses environment simulation, attacker planning, user simulation, and verification to synthesize executable stateful environments, attacks, and long-horizon tasks.

  • Results

    ToolHazard-Bench reveals substantial agent vulnerabilities and timing- and placement-sensitive attacks, while generated alignment data improves security on ToolHazard-Bench and AgentDojo with preserved benign utility.

  • Takeaways & Limitations

    ToolHazard provides infrastructure for reproducible security stress testing and adversarial training of tool-augmented agents in complex environments.

  • Takeaways & Limitations

    Synthesized environments may not capture proprietary implementations, deployment-specific interactions, or long-tail production failures, and the framework currently uses six predefined attack strategies.

Abstract

from arXiv · show

Large language model (LLM) agents integrated with external tools are vulnerable to indirect prompt injections embedded in environmental states. However, existing studies largely rely on manually implemented or reused environments, stochastic LLM-based tool simulation, and predefined injection locations, limiting scalable security research across broader domains. To bridge this gap, we propose **ToolHazard**, a scalable adversarial environment synthesis framework that reduces human engineering and supports expansion with additional seed domains and compute. Through an Environment Simulator, an Attacker Agent, and a User Simulator, ToolHazard synthesizes executable stateful environments, discovers viable injection points and generates environment-specific payloads, and constructs state-grounded long-horizon tasks. Based on ToolHazard, we build **ToolHazard-Bench** for stress-testing agents under complex workflows and diverse environmental attacks. Experiments reveal substantial agent vulnerabilities and show that injection timing and placement affect attack effectiveness. Moreover, ToolHazard-generated alignment data improves security on both ToolHazard-Bench and AgentDojo while preserving benign task utility.

1 Introduction

ToolHazard addresses the difficulty of scaling reproducible security research for tool-using agents by synthesizing adversarial environments, attacks, and long-horizon tasks. ToolHazard-Bench exposes vulnerabilities and attack-position effects, while generated alignment data improves security without reducing benign utility.

  • Indirect prompt injections can manipulate agents into unsafe or unauthorized actions, making scalable safety-boundary evaluation important for reliable deployment.
  • Existing security benchmarks often depend on manually implemented or reused environments, while stochastic tool simulation hinders reproducible evaluation and reliable training.
  • ToolHazard synthesizes executable stateful environments, discovers injection points and payloads, and generates state-grounded long-horizon tasks through three specialized modules.
  • ToolHazard-Bench contains 87 long-horizon tasks across 28 stateful environments and 512 tools, with substantially higher workflow complexity than prior security benchmarks.
  • ToolHazard-generated alignment data improves security on ToolHazard-Bench and AgentDojo while preserving benign task utility.
  • Experiments find that agents remain highly vulnerable, with attacks more effective when instructions appear earlier and near the end of observations.

2 Related Work

Related work distinguishes direct malicious-user attacks from environment-side prompt injections and situates ToolHazard as a framework for synthesizing environments, tasks, and injection points.

  • Existing benchmarks cover complementary threat models, including malicious user instructions and environment-side attacks.
  • The framework broadens prior settings by dynamically synthesizing executable stateful environments, state-grounded tasks, and viable injection points.
  • Prior alignment methods mainly address direct attacks, while ToolSafety studies indirect injection associated with harmful content generation.
  • ToolHazard instead studies environment-embedded instructions that hijack agents into unintended or risky tool-use actions under benign queries.

3 Threat Model

The threat model considers API-based tool-use agents interacting with executable stateful environments, where attackers modify observable environmental content to trigger unsafe actions.

  • An environment is an executable, stateful tool-interactive system containing entities, state variables, tool APIs, transition rules, and operational constraints.
  • Browser-based web agents and webpage-level prompt injections are outside scope; the model focuses on API-based tool-use agent security.
  • Agents interact iteratively through reasoning and function calling, producing action-observation trajectories in response to benign user queries and environments.
  • Environment-side attacks inject instructions at attacker-writable states that can propagate into observations during task execution.
  • Viable injection points include emails, documents, database records, and tool outputs, while attackers cannot alter queries, system instructions, tools, or agent parameters.
  • An attack succeeds when an injected instruction hijacks decision-making and triggers an unintended or unsafe tool action.

4 ToolHazard

ToolHazard automates synthesis and verification of executable, stateful tool-interactive environments, adversarial attacks, and state-grounded long-horizon tasks. Its modules support environment construction, injection discovery and execution, task generation, and terminal-state evaluation.

  • ToolHazard automates executable environment construction, state-grounded task generation, environment-side attack planning, and programmatic verification.
  • Environment Simulator: The Environment Simulator infers environment types, state schemas, rules, and executable operations from seed agent tasks, then converts blueprints into programs and standardized APIs.Blueprints represent entities and state schemas E, rules and constraints R, and tools T; generated environments use classes, attributes, and callable methods.
  • Automated Quality Inspection: A dual-agent verification pipeline uses a Testing Agent to invoke tools and a Checking Agent to validate execution correctness and rule consistency.Environments below a predefined quality threshold are discarded, while the remaining environments enter the environment pool.
  • User Simulator: The User Simulator initializes states from environment skeletons and constraints, then synthesizes diverse long-horizon tasks grounded in the available tools and rules.
  • Attacker Agent: The attacker identifies modifiable free-form state attributes, analyzes tool read/write operations, filters trajectory-reachable injection points, and appends payloads through state updates.An attack point comprises a modifiable state, write operations that can modify it, and read operations that expose it to the agent.
  • Verification Function Generation: Task evaluation decomposes each task into verifiable conditions and checks them against the terminal environment state, supporting multiple valid execution paths.The resulting metric is agnostic to execution trajectories because all checks depend only on the final state.

5 ToolHazard Data Ecosystem

ToolHazard produces a benchmark and alignment dataset from quality-checked, trajectory-reachable adversarial environments. The resulting data supports complex security evaluation and adversarial training with programmatic outcome checks.

  • Environment Construction: 191 valid environments are initially produced, then filtering yields 60 training environments for ToolHazard-Align and 28 disjoint test environments for ToolHazard-Bench.Only environments with valid injection points reachable along task execution trajectories are retained.
  • ToolHazard-Bench: ToolHazard-Bench contains 512 tools and 87 state-grounded tasks, with an average execution horizon of 15.56 steps.The benchmark uses six predefined environment-side attack wrappers for each task.
  • Evaluation: Generated task and attack outcomes are evaluated by check functions over final environment snapshots, with BR and ASR computed programmatically without an LLM judge at evaluation time.Human validation of environments, tasks, and check functions is reported separately.
  • ToolHazard-Align: ToolHazard-Align contains 1,040 retained adversarial-training candidates after filtering invalid constructions and attacks unobserved along benign trajectories.The dataset begins with 300 environment–task instances and 1,800 attack candidates; 329 samples are used for RL and 711 for SFT.
  • Adversarial Alignment: ToolHazard provides infrastructure for adversarial training intended to improve safety and robustness in realistic adversarial tool-use environments.

6 Evaluation

Evaluation across seven LLM agents using ReAct shows substantial vulnerability to environment-side prompt injection, with high attack success rates across models and strategies.

  • The evaluation measures benign task completion rate and attack success rate using verification-function pass rates.
  • Nearly all evaluated models exhibit high ASR in adversarial environments, demonstrating limited security guarantees against environment-side interference.
  • Four attack strategies exceed 40% ASR on GPT-5, while three exceed 30% on Gemini-3.1-Pro.
  • Decision hijacking, tool selection, and reasoning criteria consistently achieve high ASR across models.
  • More capable model generations generally achieve higher BR and lower ASR, but these gains remain insufficient for reliable prompt-injection defense.

7 Analyses

Analyses show that attack effectiveness depends on when and where injections appear, while output formatting and attacks themselves can also degrade benign task execution.

  • Injection Timing: Earlier injection points consistently yield higher ASR when the tool-selection strategy is held fixed.Timing settings compare top-1, top-2, and random injection points.
  • Injection Placement: Injections placed in later writable fields achieve higher ASR, indicating a positional bias toward tail-end tool-response content.Placement settings compare top-1, top-2, and random fields.
  • Tool Call Output Formats: Free-form tool outputs yield substantially higher ASR than structured JSON and YAML outputs under identical settings.Structured formats may provide partial semantic isolation through explicit syntax.
  • Capability Impact: Environment-side prompt injections consistently reduce benign task completion rates across environments.The comparison uses benign task success rate under clean and attacked settings.
  • Capability Impact: These findings extend agent vulnerability beyond alignment failures to broader robustness degradation affecting both safety and capability preservation.
  • Alignment Analysis: ToolHazard-Align improves security and benign task performance on both ToolHazard-Bench and AgentDojo without empirically observed over-refusal.The two benchmarks use disjoint environments with substantially different state and tool distributions.

8 Conclusion

ToolHazard provides a scalable framework for synthesizing stateful environments and long-horizon tasks, discovering verifiable injection points, and evaluating and improving tool-use agent security.

  • ToolHazard synthesizes stateful environments and long-horizon tasks while discovering viable injection points for verifiable attacks.
  • ToolHazard-Bench reveals substantial LLM-agent vulnerabilities and shows that injection timing and placement significantly affect attack effectiveness.
  • ToolHazard-generated alignment data improves security on ToolHazard-Bench and AgentDojo while preserving benign task utility.

Limitations

The framework's evaluation realism and attack-strategy coverage remain bounded by synthetic environments and six predefined prompt-injection strategies.

  • Synthesized environments may not capture proprietary implementations, deployment-specific interactions, or long-tail production failure modes.The authors frame ToolHazard as reproducible security stress testing rather than exact production replication or direct production-risk estimation.
  • ToolHazard currently considers six predefined prompt-injection strategies and does not automatically discover novel attack strategies.Automated discovery of new strategies and wrappers remains outside the current scope.

Ethics Statement

Access to harmful-content data is restricted to authorized researchers under strict ethical guidelines. The paper also documents reproducibility resources, LLM assistance, and the synthesis pipeline’s reported costs.

  • Ethical access: Access to harmful-content data is restricted to authorized researchers who follow strict ethical guidelines to mitigate risks from sensitive material.
  • Reproducibility: The authors provide an open science package and publicly available core resources intended to support replication, validation, and future research.
  • LLM disclosure: LLMs supported manuscript polishing, code development, and literature review, while the authors retained responsibility for research ideas, experiments, analysis, and conclusions.
  • Resource use: Under the GPT-4.1/GPT-4.1-mini setting, synthesizing one environment costs about $0.59, one user scenario about $0.03, and one attack instance about $0.05.Environment quality inspection is the main cost because it requires iterative testing and checking of generated tools.

F Benchmark Quality Validation

ToolHazard-Bench was validated through automated inspection and human evaluation of synthesized environments and candidate tasks. The final benchmark retains 87 tasks after invalid or ambiguous cases are removed, while its attack and synthesis pipelines support systematic adversarial testing.

  • Environment validation: All 28 synthesized environments passed inspection for consistency among state attributes, tool interfaces, transition logic, and environment blueprints.
  • Task validation: More than 99% of candidate tasks were judged executable and grounded in initialized states, while check functions agreed with human judgments above 95%.Inter-annotator agreement exceeded 99%.
  • Benchmark curation: After adjudication and removal of invalid or ambiguous cases, the final test set retained 87 tasks, supporting ToolHazard-Bench’s reliability for large-scale security evaluation.
  • Cross-Attack Generalization: Training on only three attack strategies reduced ASR from 37.19% to 26.92% while improving BR from 65.57% to 73.31% on three unseen strategies.The reported results suggest robustness transfers to unseen attack formulations, while naturally occurring attacks remain future work.
  • Attack Construction: Environment-side injections are inserted into writable textual attributes and enter agent observations when later retrieved through normal tool calls.The attack pipeline identifies injectable attributes, analyzes read/write dependencies, selects locations, generates plans, and constructs hijacking tasks.
Loading 2608.11878v1…