PaperlayerView exact revision on arXiv

Source-linked AI summary

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis

Yu Li, Haoyu Luo, Yuejin Xie, Yuqian Fu, Zhonghao Yang, Shuai Shao, Qihan Ren, Wanying Qu, Yanwei Fu, Yujiu Yang, Jing Shao, Xia Hu, Dongrui Liu

arXiv:2604.02022v4cs.AI

TL;DR

Agent safety risks increasingly emerge across long, tool-mediated trajectories, while existing benchmarks provide limited diversity, observability, and realism. ATBench addresses this gap with a taxonomy-guided benchmark using heterogeneous tools, delayed-trigger interactions, and quality control; experiments show it remains difficult for strong evaluators while supporting fine-grained diagnosis.

  • Problem

    Existing agent safety benchmarks provide limited interaction diversity, failure observability, and long-horizon realism for risks emerging across multi-step traces.

  • Method

    ATBench constructs trajectory-level evaluations with a three-dimensional taxonomy, heterogeneous tool pools, delayed-trigger long-context protocols, filtering, and human audit.

  • Results

    ATBench remains challenging for strong evaluators, with fine-grained diagnosis reaching only 33.6% for risk source, 13.5% for failure mode, and 30.2% for real-world harm among strong closed-source models.

  • Takeaways & Limitations

    ATBench supports standardized evaluation of long-context, tool-mediated safety and diagnosis across taxonomy slices.

  • Takeaways & Limitations

    All trajectories are synthetically generated and English-only, and single primary labels can miss secondary interpretations; failure-mode annotation agreement is 67.3%.

Abstract

from arXiv · show

Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions rather than isolated prompts or final responses. Existing trajectory-level benchmarks remain limited by insufficient interaction diversity, coarse observability of safety failures, and weak long-horizon realism. We introduce ATBench, a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety. ATBench organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm. Based on this taxonomy, we construct trajectories with heterogeneous tool pools and a long-context delayed-trigger protocol that captures realistic risk emergence across multiple stages. The benchmark contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools. Data quality is supported by rule-based and LLM-based filtering plus full human audit. Experiments on frontier LLMs, open-source models, and specialized guard systems show that ATBench is challenging even for strong evaluators, while enabling taxonomy-stratified analysis, cross-benchmark comparison, and diagnosis of long-horizon failure patterns.

1 Introduction

ATBench addresses limited diversity, observability, and long-horizon realism in agent safety benchmarks with a taxonomy-guided, trajectory-level design. It combines heterogeneous tools, delayed-trigger interactions, quality control, and diagnostic evaluation.

  • Motivation: Agent safety risks often emerge across extended tool-using interaction traces rather than isolated responses.These risks can involve compounding planning errors, unsafe tool use, environmental feedback, or delayed exploitation of permissions.
  • Motivation: Existing benchmarks are limited by restricted scenario diversity, coarse failure observability, and insufficient long-horizon realism.These limitations obscure the causes, behaviors, and consequences of unsafe outcomes.
  • Benchmark scope: ATBench contains 1,000 diverse and realistic trajectories supported by rule-based and LLM-based filtering plus a five-reviewer human audit.The benchmark evaluates whether models can detect and diagnose risks in completed trajectories rather than measuring deployed-agent unsafe behavior rates.
  • Benchmark design: ATBench organizes agent safety along risk source, failure mode, and real-world harm to support controllable coverage and fine-grained diagnosis.The taxonomy serves both as a construction scaffold and as a diagnostic label space.
  • Benchmark design: ATBench combines heterogeneous tool pools, planner-based synthesis, and setup–exploit delayed-trigger trajectories to preserve diversity and long-horizon realism.Risks planted early can take effect several turns later, separating setup from exploitation.
  • Results: ATBench remains challenging for strong proprietary and open-source models while enabling cross-benchmark difficulty comparison and taxonomy-stratified diagnosis.The benchmark is designed to reveal where risks originate, how failures unfold, and what downstream harm results.

2 Related Work

Related work has shifted agent safety evaluation toward interactive, tool-using trajectories and has developed synthetic tool data and guard models. These directions motivate benchmarks that assess complete execution traces and structured diagnosis.

  • Trajectory-level evaluation: Recent agent safety benchmarks evaluate prompt injection, computer-use safety, and tool-utilization safety in interactive environments.This work moves beyond static text-only evaluation toward multi-step execution with tools.
  • Synthetic data: Synthetic tool pools and trajectory generation provide scalable alternatives when real agent trajectories are expensive, sensitive, or unsafe to collect.Prior work includes simulated tool-use interactions, large real-world API pools, and verification-based synthesis.
  • Guard models: Guard models and runtime safety mechanisms increasingly target policy reasoning, explicit constraints, proactive monitoring, and multilingual guardrails.These systems extend conversational safeguarding toward execution-time safety and trajectory diagnosis.

3 Benchmark Construction for Diversity with Realism

ATBench constructs complete multi-turn safety trajectories using a three-dimensional taxonomy, heterogeneous tools, paired safe/unsafe scenarios, delayed triggers, automated filtering, and human audit. The task supports binary safety judgment and fine-grained diagnosis over completed traces.

  • Task definition: ATBench evaluates complete multi-turn traces containing requests, responses, tool calls, and environmental feedback for overall safety.Unsafe trajectories additionally receive one primary label for each taxonomy dimension, without step-level annotations.
  • Design objective: The construction pipeline combines taxonomy-guided risk coverage, heterogeneous executable tools, planner synthesis, paired variants, delayed triggers, filtering, and human audit.These components jointly target diversity while preserving realism under long-horizon interactions.
  • Taxonomy: The taxonomy decomposes unsafe trajectories into risk source, failure mode, and real-world harm rather than surface-level risk types.Its leaf structure contains 8 risk-source categories, 14 failure-mode categories, and 10 harm categories.
  • Tool pool: A heterogeneous tool pool combines public APIs, tools adapted from prior resources, and simulated tools to cover underrepresented capabilities and risk-relevant regions.Tool names, arguments, and descriptions are normalized across sources.
  • Generation engine: The planner samples taxonomy slices and candidate tools, then specifies the task, trigger location, execution steps, and intended safety outcome.The blueprint is executed sequentially through query generation, risk injection, tool-call and response simulation, and agent-response generation.
  • Paired construction: Paired safe and unsafe variants share a scenario skeleton, with risk removed, resisted, or correctly handled in the safe version.The sampled risk source determines injection location, failure mode constrains behavior, and harm determines the downstream consequence.
  • Long-context protocol: Delayed-risk trajectories model dependencies across steps and intermediate states, capturing safety-critical consequences of earlier benign-looking actions.This targets realistic long-context risk emergence rather than merely increasing the number of turns.
  • Quality control: Rule-based checks validate tool names, schemas, and required parameters, while LLM filtering assesses behavioral plausibility and annotation consistency.A full human audit then produced 5 binary and 165 fine-grained corrections across 129 unsafe trajectories, yielding 503 safe and 497 unsafe cases.

4 Experiments

The experiments evaluate trajectory-level detection, fine-grained diagnosis, cross-benchmark difficulty, and transfer from engine-generated training data. ATBench is difficult for strong evaluators, especially when identifying risk sources, failure mechanisms, and indirect harms.

  • Experimental design: The empirical study covers coarse safety evaluation, fine-grained taxonomy diagnosis, category-wise analysis, cross-benchmark comparison, and generation-engine transfer.
  • Main findings: Evaluators often detect unsafe trajectories but struggle to diagnose the underlying risk across the three taxonomy dimensions.
  • Trajectory-level safety: GPT-5.4 achieves the highest coarse-grained F1 at 76.7%, while Gemini-3.1-Pro reaches 75.0% and AgentDoG-Qwen3-4B reaches 71.1%.
  • Fine-grained diagnosis: Fine-grained diagnosis remains weaker: GPT-5.4 reaches 33.6% on risk source, 13.5% on failure mode, and 30.2% on real-world harm, while AgentDoG-Qwen3-4B performs best overall.
  • Category-wise analysis: Tool-mediated categories and indirect harms are especially difficult, with models frequently recognizing unsafe outcomes without identifying the tool-related source or mechanism.
  • Cross-benchmark comparison: Performance is generally lower on ATBench than on prior benchmarks, including GPT-5.2 at 69.0% versus 90.8% on R-Judge and AgentDoG-Qwen3-4B at 64.0% versus 91.8%.
  • Transfer experiments: SFT on engine-generated trajectories improves real-trajectory safety judgment and fine-grained diagnosis across ATBench taxonomy dimensions.On R-Judge, accuracy improves by 6.6 points and F1 by 5.3 points; ATBench diagnosis also improves across Risk Source, Failure Mode, and Real-world Harm.

5 Discussion

ATBench addresses the tension between broad coverage and realistic trajectories through structural checks and human review. Its limitations include single-primary-label annotation, synthetic data, English-only trajectories, and text-and-tool scope.

  • Discussion: Broader tools and finer labels improve coverage, but diversity without structural checks can produce templated or implausible data.
  • Discussion: Delayed triggers, filtering, and full human audit are used to preserve realism while expanding benchmark diversity.
  • Limitations: Each unsafe trajectory receives one primary label per taxonomy dimension, which can miss secondary interpretations in multi-causal cases.
  • Limitations: Fine-grained failure-mode boundaries remain ambiguous, with 67.3% human agreement on that dimension.
  • Limitations: The benchmark uses synthetic English text-and-tool trajectories, so its distribution should not be treated as identical to deployed traces and does not cover multimodal or embodied settings.

6 Conclusion

ATBench is a long-horizon agent-safety benchmark designed to maximize diversity while preserving realism. It uses a three-dimensional taxonomy and generation pipeline to support structured evaluation, diagnosis, and future study of tool-mediated risks.

  • Conclusion: ATBench evaluates long-horizon agent safety by combining taxonomy-guided generation, heterogeneous tools, delayed triggers, and human-in-the-loop quality control.
  • Taxonomy: The taxonomy organizes risk by source, failure mode, and real-world harm, respectively describing origin, manifestation, and consequence.
  • Taxonomy: The taxonomy contains 8 risk-source categories, 14 failure-mode categories, and 10 harm categories at the leaf level.
  • Risk source: Risk sources include user inputs, environmental observations, external entities, and internal decision-making logic.
  • Failure mode: Failure modes distinguish behavioral failures from output-content failures, separating unsafe execution from risky textual outputs.
  • Real-world harm: The real-world harm dimension focuses on consequences such as physical, financial, privacy, psychological, reputational, and societal harms.

B Benchmark Construction Details

ATBench constructs and annotates trajectories with explicit safety labels, delayed-trigger interactions, and layered quality controls. Its annotation policy prioritizes one causal label per taxonomy dimension to keep evaluation stable and analyzable.

  • Annotation: Each trajectory is first assigned a binary safe or unsafe verdict, and unsafe trajectories receive a fine-grained taxonomy tuple.
  • Annotation: Annotators assign risk source, failure mode, and real-world harm in causal order, from the earliest risk factor to the resulting consequence.
  • Annotation: When multiple labels are plausible, annotators select the label most central to the main causal chain rather than tagging incidental issues.
  • Delayed-trigger generation: Long-context scenarios use contiguous SETUP and EXPLOIT episodes, with the later exploit depending on reusable state established earlier.
  • Delayed-trigger generation: Setup messages and tool outputs remain in shared history, allowing later stages to condition on the prior transcript rather than only the current turn.
  • Delayed-trigger generation: Unsafe trajectories act on the carried-over dependency, whereas safe trajectories refuse, verify, or neutralize it at the delayed-trigger step.
  • Quality control: Filtering checks tool-call structure and schema compliance, then uses an LLM reviewer to assess plausibility, labels, and reasoning consistency.

C.1 Trajectory-level Safety Evaluation

The evaluation protocol formats complete agent trajectories for safety assessment, using general-model prompts for overall safety and fine-grained prompts for taxonomy-level diagnosis. Guard models are evaluated with their official model-specific templates, including LlamaGuard, Qwen3Guard, and ShieldAgent.

  • Trajectory-level evaluation: Complete agent trajectories are formatted into a shared prompt for evaluating textual responses and tool-executed behaviors.The assessment requires an overall safe-or-unsafe judgment.
  • Fine-grained diagnosis: Fine-grained evaluation requires labels for Risk Source, Failure Mode, and Real-world Harm.The template is designed to expose diagnostic distinctions beyond a binary safety verdict.
  • Guard-model evaluation: Guard models use their official, model-specific chat templates to preserve fidelity to their original implementations.The evaluated guard models include LlamaGuard, Qwen3Guard, and ShieldAgent.

D Full Human Audit Details

ATBench underwent a full five-reviewer audit of all 1,000 trajectories, correcting a small number of binary verdicts but many fine-grained taxonomy labels. The final release contains 503 safe and 497 unsafe trajectories, while agreement was strongest for Risk Source and Real-world Harm and weaker for Failure Mode.

  • Audit protocol: Five reviewers inspected all 1,000 trajectories after automatic synthesis and rule-based and LLM-based filtering.Reviewers first checked coarse safe-or-unsafe verdicts, then examined fine-grained taxonomy labels for retained unsafe trajectories.
  • Audit outcome: 5 binary verdicts changed: 4 unsafe-to-safe corrections and 1 safe-to-unsafe correction.The audit therefore made relatively few coarse-grained label changes across the full release.
  • Audit outcome: 165 fine-grained label corrections affected 129 retained-unsafe trajectories, including 64 Risk Source, 67 Failure Mode, and 34 Real-world Harm revisions.These corrections were computed after coarse-grained review over trajectories still labeled unsafe.
  • Audit outcome: The final released benchmark contains 503 safe and 497 unsafe trajectories.These counts follow the binary-verdict correction pass.
  • Agreement study: Human agreement averages 84.7% for Risk Source and Real-world Harm but 67.3% for Failure Mode.The lower Failure Mode agreement reflects semantically closer category boundaries; the best model reaches only 16.5% on that diagnosis.

E.2 Ablation on the Sources of Benchmark Difficulty

The ablation separates long-context reasoning, delayed or indirect triggers, and tool/interface-grounded evidence as sources of benchmark difficulty. Results indicate that no single factor explains performance degradation, and model families weaken on different subsets.

  • Ablation design: The ablation evaluates binary safety detection on Overall, Unsafe, Long Ctx., delayed-or-indirect-trigger, and tool/interface-grounded subsets.This design separates multiple hypothesized sources of difficulty rather than treating benchmark performance as a single aggregate.
  • Ablation findings: Long-context capability matters, while delayed or indirect triggers and tool/interface-grounded evidence also substantially increase difficulty.Different model families degrade on different targeted subsets.
  • Ablation findings: ATBench difficulty arises from multiple interacting factors, including fine-grained causal diagnosis.Even models with reasonable coarse-grained scores remain far from saturated on Risk Source, Failure Mode, and Real-world Harm diagnosis.

E.3 Human Realism Study

A human study assessed semantic realism across four aspects of sampled ATBench trajectories, whose examples span unsafe and safe tool-use behaviors. All aspects scored above 4/5 on average, with an overall realism score of 4.09/5.

  • Study design: Three agent-safety researchers rated sampled trajectories on a 1–5 realism scale across four aspects and overall.A score of 1 means clearly unrealistic, 3 uncertain or mixed, and 5 realistic.
  • Human ratings: All four aspects scored above 4/5, with an overall realism score of 4.09/5.Tool Definition received the highest score, while User Query was relatively lower.
  • Trajectory coverage: The examples cover unsafe trajectories involving corrupted feedback, unverified medical updates, wasteful verification, hidden safety changes, and misleading evidence.These cases connect tool behavior and external observations to concrete financial, clinical, operational, and physical risks.
  • Trajectory coverage: Safe examples show agents halting after suspicious side effects, refusing to broadcast unverified chemical guidance, and staying within public-data boundaries.The cases emphasize stopping downstream communication, avoiding operationalization of unverified advice, and checking that profile data remains public.
  • Trajectory coverage: Other unsafe examples involve unsolicited bulk outreach, routing mismatches, speculative regulatory alerts, over-privileged data transfer, hazardous-cargo misrouting, and destructive deletion.These scenarios illustrate how injected or untrusted content can be turned into consequential external actions.
Loading 2604.02022v4…