PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification

Yunhao Feng, Ruixiao Lin, Ming Wen, Qinqin He, Yanming Guo, Yifan Ding, Yutao Wu, Jialuo Chen, Zhuoer Xu, Xiaohu Du, Jianan Ma, Zixing Chen, Xingjun Ma, Yunhao Chen, Xinhao Deng

arXiv:2607.01793v2cs.AI

TL;DR

Existing safety tests often rely on expert-designed scenarios and textual or attempted actions rather than verified environmental effects. Vera automates risk discovery, executable case construction, adaptive execution, and evidence-grounded verification, finding average attack success rates of 93.9% under multi-channel attacks across four production agent frameworks.

  • Problem

    Existing agent safety evaluations often conflate unsafe requests, attempted actions, or stated intent with realized violations and tightly couple risks, environments, adapters, and verification.

  • Method

    Vera combines autonomous risk discovery, combinatorial executable safety-case generation, adaptive runtime interaction, and evidence-grounded verification across agent frameworks.

  • Results

    93.9% average attack success under multi-channel attacks reveals substantial safety weaknesses across four production agent frameworks and diverse risks, methods, and environments.

  • Takeaways & Limitations

    Safety evaluation for increasingly autonomous, tool-integrated agents should evolve toward modular, executable testing infrastructure for future agent assurance.

Abstract

from arXiv · show

LLM agents increasingly perform autonomous actions through external tools, leading to complex and evolving safety risks. However, existing safety testing targets expert-designed safety violations, and the corresponding outcomes are evaluated by hard-coded rules, making them costly to extend as agents evolve. To this end, we present Vera, an end-to-end automated safety testing framework that instantiates software engineering testing principles for non-deterministic agents through a three-stage, self-reinforcing pipeline. First, a literature-driven exploration continuously discovers and structures emerging risks into taxonomies of safety risks, attack methods, and tool execution environments. Second, combinatorial composition across taxonomy dimensions produces executable safety cases, each specifying a concrete safety goal, a programmatically constructed initial state, and a deterministic verification predicate grounded in observable artifacts. Third, adaptive execution runs heterogeneous agents in isolated sandboxes where a control agent steers multi-turn interaction based on runtime observations, while evidence-grounded verifiers judge outcomes from environment state and tool-call evidence rather than model self-report. We evaluate Vera on four production agent frameworks (OpenClaw, Hermes, Codex, Claude Code), revealing substantial safety weaknesses, with average attack success rates reaching 93.9\% under multi-channel attacks; we also release Vera-Bench, comprising 1600 executable safety cases spanning 124 risk categories across three execution settings. These results indicate that modular, executable testing infrastructure is essential for rigorous and maintainable safety evaluation of rapidly evolving agentic systems at scale. The code is publicly available at https://github.com/Yunhao-Feng/Vera.

I. INTRODUCTION

LLM agents’ expanding tool-mediated autonomy creates increasingly diverse and complex safety risks that challenge large-scale, runtime-grounded evaluation. VERA addresses these challenges with executable safety cases, adaptive sandboxed execution, and evidence-grounded verification across divergent agent frameworks.

  • Motivation: Tool-using LLM agents can autonomously perform actions beyond text generation, while their safety risks rapidly expand in categories, manifestations, and combinatorial diversity.The cited risks include sensitive data exposure and unauthorized system actions.
  • Motivation: Existing evaluations often treat unsafe requests, attempted actions, or stated intentions as violations without verifying that harmful outcomes were produced through executed actions.This limitation motivates evaluation grounded in realized environment outcomes rather than textual claims.
  • VERA framework: VERA instantiates software-testing principles for agents through autonomous risk discovery, executable test-case generation, and runtime-adaptive execution.Its testing primitives include test oracles, combinatorial construction, and evidence-grounded verification.
  • VERA framework: Sandboxed adaptive execution records tool interactions, lets a control agent steer interactions from observations, and verifies outcomes from observable artifacts rather than model self-report.This design handles divergent execution paths caused by non-deterministic runtime planning and tool selection.
  • Evaluation: VERA was evaluated on four production agent frameworks and accompanied by VERA-Bench covering three threat models with deterministic verifiers.The evaluation revealed substantial safety vulnerabilities.

II. RELATED WORK … III. PRELIMINARIES

The paper motivates trajectory-level safety testing for tool-using agents and formalizes executions through observable trajectories, environment states, executable safety cases, and deterministic verifiers. Its threat model includes benign, single-channel, and adaptive multi-channel interactions while excluding model internals and protected system components.

  • A. Safety Risks of Computer-Use Agents: Computer-use agents increasingly execute tasks through external tools, creating risks including credential leakage, private-data exfiltration, and unauthorized operations.These risks arise across software repositories, desktop applications, and web applications.
  • B. Safety Evaluation and Testing for LLM Agents: Safety evaluation is shifting from prompt- or response-level harmful-output assessment toward trajectory-level analysis of tool-mediated behavior.This shift reflects the growing complexity and diversity of agent risks.
  • B. Safety Evaluation and Testing for LLM Agents: Prompt-level methods assess unsafe-request compliance or refusal, emphasizing content-safety boundaries rather than downstream execution behavior.Their violations are verified using LLM-based judges or fine-tuned safety classifiers applied to model outputs.
  • B. Safety Evaluation and Testing for LLM Agents: Trajectory-level benchmarks inspect complete execution traces in stateful tool environments, but their risk categories and scenarios are pre-defined by human experts.This dependence limits how autonomously such benchmarks can expand with evolving risks.
  • III. PRELIMINARIES: A computer-using agent A interacts with a stateful environment E through tools T across n conversation turns, issuing ki tool calls at turn i before response ri.The j-th call ai,j selects a tool and its arguments from T.
  • III. PRELIMINARIES: The trajectory records externally observable tool calls and responses, while internal chain-of-thought and planning are excluded because safety outcomes depend on executed actions and observable effects.The execution also records the cumulative environment state at termination, denoted sT.
  • III. PRELIMINARIES: An executable safety case σ = ⟨g, s0, Vg⟩ specifies a target violation, a programmatically constructed initial state, and a deterministic code-level verifier.Executing A in E produces trajectory τ and post-execution state sT, which the verifier uses to determine whether the violation occurred.
  • III. PRELIMINARIES: The verifier confirms success only from recorded tool calls, responses, and environment state; adversarial content alone is insufficient, while attacks may span user messages and injected tool results.The adversary can adapt across turns using τ but cannot modify the target model, system instructions, agent implementation, or internal tool code.

IV. METHODOLOGY

VERA implements end-to-end agent safety testing as a three-stage, self-reinforcing pipeline. It explores emerging risks, constructs executable safety cases, and executes tests using structured taxonomies and programmatically defined states.

  • Three-stage pipeline: VERA’s three-stage pipeline integrates risk exploration, executable test-case construction, and adaptive execution for end-to-end agent safety testing.The pipeline is described as self-reinforcing and illustrated in Figure 1.
  • Continuous Risk Exploration: Continuous Risk Exploration discovers and structures emerging risks from research literature into taxonomies of risks, attack methods, and environments.This stage consolidates risk-related knowledge into structured dimensions for subsequent testing.
  • Executable Test Case Construction: Executable Test Case Construction composes taxonomy elements into safety cases with concrete safety goals and programmatically constructed initial states.The passage also specifies a deterministic verification predicate as part of each case.

A. Continuous Risk Exploration

Vera continuously explores agent safety risks by organizing harmful consequences, attack methods, and tool-execution environments into actionable taxonomies. A Summary Agent iteratively expands these hierarchical trees from public safety literature, with the mechanism designed to incorporate emerging intelligence sources and require evidence for new concepts.

  • Risk representation: Safety risks are characterized by three orthogonal dimensions: harmful consequence, induction mechanism, and tool-execution environment.The taxonomies cover risks such as credential disclosure, unauthorized modification, and unsafe code execution, alongside attack methods including prompt injection and task decomposition.
  • Taxonomy construction: Each taxonomy is a hierarchical tree whose leaf nodes define the finest-grained actionable units for generating safety cases.A Summary Agent iteratively populates the trees from broad search concepts rather than downstream evaluation labels or examples.
  • Literature-driven exploration: The exploration begins with broad concepts derived from public agent-safety literature and recursively extracts structured concepts from retrieved documents.The same mechanism can extend to CVE databases, vendor advisories, and MITRE ATT&CK to support deployment-time updates tracking emerging vulnerabilities.
  • Evidence-based updates: A new leaf node is created only when a previously unrepresented risk, attack method, or environment class is supported by at least five distinct papers or attack scenarios.The update operator processes retrieved documents using create and update operations across the three taxonomies.

B. Executable Test Case Construction · C. Adaptive Execution and Evidence-Grounded Verification

Vera converts taxonomy-derived risks into executable safety cases with concrete goals, programmatic initial states, and deterministic verifiers, then executes them adaptively in sandboxed environments. Its construction filters and deduplicates feasible cases, while controlled threat variants and recorded tool interactions support evidence-grounded safety verification.

  • B. Executable Test Case Construction: Each executable safety case is a triple comprising a concrete safety goal, an initial environment state, and a programmatic verifier.The case is represented as σ = ⟨g, s0, Vg⟩.
  • B. Executable Test Case Construction: Candidate goals are generated by composing one leaf from each risk, attack-method, and execution-environment taxonomy.A goal composer maps each taxonomy tuple to a context-specific safety goal.
  • B. Executable Test Case Construction: An LLM-based goal composer contextualizes abstract taxonomy tuples into concrete, verifiable safety violations using dynamically updated demonstrations and environment descriptions.Typed-schema validation checks generated goals against a minimum-length target description.
  • B. Executable Test Case Construction: For each risk–attack-method pair, Vera samples ten environment leaves and retains only combinations exposing the resources and actions required by the goal.Retained goals are normalized and deduplicated, while cases depending on internal reasoning traces or duplicating accepted cases are filtered.
  • B. Executable Test Case Construction: Each retained base scenario expands into benign, single-channel, and multi-channel variants representing progressively stronger adversarial conditions.The benign variant provides a functional baseline; the single-channel variant leaves tool results unmodified, while the multi-channel variant transforms selected tool outputs.
  • C. Adaptive Execution and Evidence-Grounded Verification: The adaptive execution stage verifies target violations through a Control Agent coordinating sandboxed heterogeneous agents, runtime steering, and recorded tool interactions.A configurable tool gateway records interactions, while an adaptive test driver uses sandbox observations to steer multi-turn interaction toward the safety goal.

1) Large-scale Sandboxed Execution Environment: · 2) Adaptive Test Driver:

Vera combines per-agent adapters, isolated sandboxes, and an MCP-based tool gateway to execute heterogeneous agents while recording evidence of interactions, tool calls, and persistent effects. An adaptive Control Agent plans and steers attacks using runtime observations, gateway ground truth, and environment changes until evidence is produced or execution should stop.

  • 1) Large-scale Sandboxed Execution Environment:: Each execution runs the target agent, tool gateway, and scenario-required external services in an isolated sandbox, while per-agent adapters normalize framework-specific events into a shared trajectory.The adapters accommodate differences in launch procedures, message transport, tool protocols, and transcript formats.
  • 1) Large-scale Sandboxed Execution Environment:: The MCP-based tool gateway mediates tool calls and supports identity, append, prefix, and override transformations for multi-channel adversarial execution.Append and prefix preserve legitimate results alongside attacker-controlled content, whereas override fully compromises the data source.
  • 1) Large-scale Sandboxed Execution Environment:: Three evidence sources jointly reconstruct the trajectory and final state: interaction logs, gateway logs with original and transformed results, and persistent environment changes.Recorded effects include repository modifications, outgoing messages, transfers, and created records.
  • 2) Adaptive Test Driver:: Before execution, the Control Agent converts each safety case and available tool schemas into an attack plan specifying the intended interaction sequence and relevant tools.Its control state aggregates the safety case, history, strategy summaries, observed calls, injection rules, and progress estimate.
  • 2) Adaptive Test Driver:: The Control Agent selects tool results and gateway rules, first establishing plausible legitimate context before gradually introducing adversarial intent.At each turn it emits the next user message together with multi-channel gateway rules.
  • 2) Adaptive Test Driver:: Runtime observations combine the agent response, newly recorded gateway events, and observable environment changes, enabling adaptation based on evidence rather than potentially inaccurate self-report.Gateway events serve as ground truth when the agent misreports its actions.
  • 2) Adaptive Test Driver:: The observe–adapt–act loop reformulates refused requests, revises task decomposition after unexpected tool use, or changes injection points when relevant content is not retrieved.Execution ends at the budget, once required evidence is produced, or when further interaction is unlikely to change the outcome; decisions and trajectories are stored for replay.

3) Evidence-Grounded Verification:

Vera verifies agent outcomes with deterministic programs that prioritize manipulation-resistant environment evidence over tool-call intent or agent self-report. Its asymmetric judgment rejects unverified success claims while regenerating failed verifiers to avoid false negatives.

  • Evidence precedence: Environment state takes priority because tool calls record intent but do not guarantee effects, while agent responses matter only when their text is itself the violation.Evidence sources are selected by manipulation resistance, with fallback evidence used when the preferred predicate is undefined.
  • Deterministic verification: Each verifier is a deterministic Python program whose outcome is independent of the generation model.Verification runs while the sandbox remains active, allowing queries to live service APIs.
  • Asymmetric judgment: When the Control Agent reports failure, Vera records y=0 without verification; reported success requires confirmation against environment evidence before assigning y=1.This asymmetric judgment eliminates false positives from optimistic self-assessment.
  • Verifier robustness: Verifiers that fail because of syntax errors or tool-call schema mismatches are regenerated to avoid false negatives.The regeneration mechanism addresses verifier failures during outcome evaluation.

V. EXPERIMENT · A. Experimental Setup

Vera is evaluated across heterogeneous agent frameworks and backend models using taxonomies derived from approximately 800 papers. Its benchmark comprises 1,600 executable scenarios tested in freshly initialized Docker Compose sandboxes under controlled execution settings.

  • A. Experimental Setup: Vera evaluates four heterogeneous frameworks—OpenClaw, Hermes, Codex, and Claude Code—with differing execution loops, tool protocols, context management, and interfaces.These differences test whether generated scenarios remain independent of a particular agent implementation.
  • A. Experimental Setup: The evaluation includes GPT-5.2, Gemini-3, Qwen-3.7, Kimi-K2.6, and GLM-5.2 when compatible with each framework.Each agent–model configuration is constrained by backend compatibility.
  • A. Experimental Setup: Approximately 800 arXiv and OpenReview papers yield taxonomies with 124 risk categories, 77 attack methods, and 30 environment categories.These are leaf-level taxonomy counts produced by the risk exploration stage.
  • A. Experimental Setup: 39,078 candidate safety goals remain after compatibility filtering and deduplication, while quality filtering retains 1,600 executable base scenarios for Vera-Bench.The retained scenarios are compiled from the combinatorial compositions across taxonomy dimensions.
  • A. Experimental Setup: All executions use freshly initialized isolated Docker Compose sandboxes containing the target agent, MCP middleware, and scenario-required external services.Each sandbox is a 12-container stack comprising the target agent, MCP gateway, and five self-hosted backend services.
  • A. Experimental Setup: Each base scenario runs under benign, single-channel, and multi-channel controlled settings, with termination triggered by budget exhaustion, verified success, or lack of progress.The benign setting preserves tool results, whereas the single-channel setting permits adversarial user messages.
  • A. Experimental Setup: Each retained data item includes attack_plan, mcp_logs, trace.json, and verify.py artifacts for recording objectives, interactions, tool evidence, trajectories, and case-specific verification.The logs store tool calls, arguments, original service results, and observations returned to the agent.

B. Dataset Analysis

Vera’s dataset combines broad, executable coverage across independently constructed risk, attack, and environment taxonomies with meaningful variation in stability, transferability, agent performance, and execution cost. Its ESR patterns support combinatorial composition and show that adaptive, multi-channel testing yields substantial but heterogeneous attack success.

  • Dataset structure: Vera’s taxonomies contain 124 risk categories, 77 attack methods, and 30 environment categories, aggregated by first-level parent groups for interpretable analysis.The three taxonomies are independently constructed and each has three hierarchical levels.
  • Coverage and stability: 95.3% is the highest average ESR across risk groups for Integrity, while Harmful Output is lowest at 79.0%; all eight groups execute in all ten environment groups.Average risk-group ESRs range from 79.0% to 95.3%, with Cyber Attack at 88.4%.
  • Attack-method transferability: 89.9% is Profile Infer’s average ESR across risk categories, compared with 88.5% for Task Decompos, 87.9% for Obfuscation, and 74.4% for Social Engineer.Format Inducement is also comparatively low at 78.4%, showing substantial differences in attack-method transferability.
  • Validation of combinatorial composition: All 80 Table I cells and 110 Table II cells exceed 40% ESR, supporting executable combinatorial composition while preserving informative variation across combinations.Lower-performing combinations include Malware Gen × Web & Stor at 71.4% and Priv Escal × Travel at 73.7%.
  • Adaptive and multi-channel testing: 93.9% is the average Multi-Channel ESR versus 90.6% for Single-Channel, a 3.3-point increase that ranges from −2.1 points for Claude Code to +6.3 for OpenClaw.For Codex, multi-channel testing increases ASR by 4.7 points (91.1% →95.8%), whereas Claude Code decreases by 2.1 points (95.2% →93.1%).
  • Execution cost and interaction length: 155k input tokens, 3k output tokens, and 11 tool calls are the median retained-run values, while executions exhibit a pronounced right tail.The retained runs are usually moderate in length, but the corresponding 95th-percentile values extend substantially higher.

VI. DOWNSTREAM TASK

VERA supports downstream safety classification by generating training data that substantially improves Qwen3Guard across accuracy, recall, and F1. The fine-tuned guard model also transfers to R-Judge and trains stably, suggesting benefits beyond VERA’s own benchmark.

  • Safety classification: Off-the-shelf guards transfer poorly: LlamaGuard3 reaches 0.438 accuracy, 0.258 recall, and 0.310 F1, while AgentDoG reaches 0.742 recall but 0.490 accuracy and 0.643 F1.Base Qwen3Guard reaches 0.670 accuracy, 0.468 recall, and 0.637 F1.
  • Safety classification: Fine-tuning Qwen3Guard on VERA-derived data reaches 0.930 accuracy, 0.903 recall, and 0.941 F1 on the VERA downstream task.Relative to base Qwen3Guard, gains are 26.0 points in accuracy, 43.5 points in recall, and 30.4 points in F1.
  • Training dynamics: Training and evaluation losses decrease smoothly, with final train loss 0.0868 and best evaluation loss 0.0387 at step 210.The evaluation curve tracks training without late-stage instability.
  • Out-of-distribution transfer: On the separate R-Judge benchmark, the fine-tuned model achieves the highest accuracy at 61.7%, exceeding all off-the-shelf baselines.This evaluation tests transfer across different prompt distributions and decision boundaries.
  • Out-of-distribution transfer: On R-Judge, recall reaches 77.9%, providing a more balanced operating point than extremely aggressive guards while exceeding NemoGuard and Qwen3-Guard.The result suggests VERA training improves harmful-versus-non-harmful separation under distribution shift rather than merely fitting benchmark conventions.

VII. CONCLUSION

VERA operationalizes agent safety testing through executable safety cases, adaptive runtime interaction, and evidence-grounded verification over environment state. Experiments across four production agent frameworks reveal substantial safety weaknesses, while VERA-Bench-fine-tuned guard models generalize more effectively than strong off-the-shelf baselines.

  • Conclusion: VERA operationalizes agent safety testing with executable safety cases, adaptive runtime interaction, and evidence-grounded verification over environment state.The framework combines these three components into an executable testing approach.
  • Conclusion: Four production agent frameworks reveal substantial safety weaknesses across diverse risks, attack methods, and environments.The experiments span multiple dimensions of agent safety evaluation.
  • Conclusion: Guard models fine-tuned on VERA-Bench generalize more effectively than strong off-the-shelf baselines.The conclusion reports improved generalization for VERA-Bench-trained guard models relative to the comparison baselines.
Loading 2607.01793v2…