PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents

Hanjun Luo, Shenyu Dai, Chiming Ni, Xinfeng Li, Guibin Zhang, Kun Wang, Tongliang Liu, Hanan Salam

arXiv:2506.00641v3cs.AI

TL;DR

Reliable evaluation of LLM-agent safety and security is difficult because existing evaluators can miss interaction-level, subtle, cumulative, or ambiguous risks. The paper introduces AgentAuditor’s memory-augmented reasoning framework and ASSEBench, reporting consistent improvements, state-of-the-art results, and human-level performance.

  • Problem

    Existing evaluators struggle to assess complex agent interactions, including subtle meanings, cumulative risks, and ambiguous safety or security judgments.

  • Method

    AgentAuditor retrieves structured semantic features and chain-of-thought reasoning experiences from memory to guide training-free LLM-based evaluation.

  • Results

    AgentAuditor consistently improves evaluation across datasets and LLMs, achieving state-of-the-art and human-level performance in agent safety and security evaluation.

  • Takeaways & Limitations

    ASSEBench provides a joint safety-and-security benchmark with nuanced strict and lenient standards for evaluating LLM-based agent evaluators.

  • Takeaways & Limitations

    AgentAuditor’s effectiveness depends on the label quality of reasoning-memory shots, and label defects reduce its improvement.

Abstract

from arXiv · show

Despite the rapid advancement of LLM-based agents, the reliable evaluation of their safety and security remains a significant challenge. Existing rule-based or LLM-based evaluators often miss dangers in agents' step-by-step actions, overlook subtle meanings, fail to see how small issues compound, and get confused by unclear safety or security rules. To overcome this evaluation crisis, we introduce AgentAuditor, a universal, training-free, memory-augmented reasoning framework that empowers LLM evaluators to emulate human expert evaluators. AgentAuditor constructs an experiential memory by having an LLM adaptively extract structured semantic features (e.g., scenario, risk, behavior) and generate associated chain-of-thought reasoning traces for past interactions. A multi-stage, context-aware retrieval-augmented generation process then dynamically retrieves the most relevant reasoning experiences to guide the LLM evaluator's assessment of new cases. Moreover, we developed ASSEBench, the first benchmark designed to check how well LLM-based evaluators can spot both safety risks and security threats. ASSEBench comprises 2293 meticulously annotated interaction records, covering 15 risk types across 29 application scenarios. A key feature of ASSEBench is its nuanced approach to ambiguous risk situations, employing "Strict" and "Lenient" judgment standards. Experiments demonstrate that AgentAuditor not only consistently improves the evaluation performance of LLMs across all benchmarks but also sets a new state-of-the-art in LLM-as-a-judge for agent safety and security, achieving human-level accuracy. Our work is openly accessible at https://github.com/Astarojth/AgentAuditor.

1 Introduction

LLM agents introduce complex safety and security risks that existing evaluation methods struggle to assess. AgentAuditor addresses this gap with memory-augmented reasoning, while ASSEBench enables joint evaluation of agent safety and security.

  • Motivation: LLM agents’ autonomous, tool-using interactions create complex safety and security risks that require assessing behaviors beyond generated content.These risks are especially relevant in high-stakes or dynamic scenarios.
  • Motivation: Existing agent benchmarks and automated evaluators largely test standalone risk management, predefined patterns, or general semantic judgments.Rule-based methods struggle with implicit harms, while LLM-based methods face inconsistent criteria, bias propagation, and reduced interpretability.
  • AgentAuditor: AgentAuditor uses structured semantic features, experiential memory, adaptive representative-shot selection, retrieval-augmented generation, and chain-of-thought reasoning to guide new-case assessments.Its design aims to support robust, interpretable, and human-aligned evaluation of complex agent behaviors.
  • ASSEBench: ASSEBench is the first large-scale benchmark jointly covering agent safety and security, with 2293 annotated records spanning 15 risk types and 29 application scenarios.It also includes 528 interaction environments, 26 behavior modes, and strict versus lenient standards for ambiguous cases.
  • Results: AgentAuditor consistently improves evaluation across datasets and LLMs, achieving state-of-the-art and human expert-level performance.On R-Judge, the reported example reaches up to 96.3% F1 and 96.1% accuracy with Gemini-2.0-Flash-thinking.

2 Preliminaries

Automated agent evaluation uses rule-based or LLM-based evaluators, each with distinct strengths and limitations. Safety concerns unintended harmful behavior, whereas security concerns deliberate external attacks.

  • Rule-based Evaluators: Rule-based evaluators assess outputs with predefined keywords, patterns, or logical rules, making decisions efficient and interpretable.They have limited flexibility and generalizability and struggle with implicit or ambiguous harmful content.
  • LLM-based Evaluators: LLM-based evaluators use prompt engineering or fine-tuning to apply contextual and semantic analysis to potentially harmful content.They are easier to develop and maintain than rule-based methods but may show inconsistent criteria, bias propagation, and reduced interpretability.
  • Agent Safety and Security: Agent safety concerns unintended harmful behavior arising from internal factors such as flawed design or limited risk awareness.Examples include unintentional information leakage and autonomous actions causing property damage.
  • Agent Safety and Security: Agent security focuses on protecting agents from deliberate external attacks and malicious behavior.The passage distinguishes security from safety by emphasizing the external and intentional nature of the threat.

3 AgentAuditor: Our Framework

AgentAuditor equips LLM evaluators with structured experiential and reasoning memories, then retrieves relevant experiences to guide context-aware assessments of new agent interactions.

  • 3.1 Stage 1: Feature Memory Construction: AgentAuditor builds feature memory from interaction records using structured semantic features and dual human-readable and vectorized representations.The representations capture content, scenario, risk, and behavior information for interpretable and similarity-based memory access.
  • 3.2 Stage 2: Reasoning Memory Construction: Reasoning memory stores representative interaction samples alongside LLM-generated Chain-of-Thought traces that connect case features to evaluation conclusions.Representative shots are selected by clustering processed embeddings and choosing samples near cluster centroids.
  • 3.3 Memory-Augmented Reasoning: AgentAuditor retrieves relevant reasoning-memory entries through staged content and feature-similarity ranking for each target interaction.Feature similarities are weighted and aggregated after an initial Top-n retrieval to select the final k shots.
  • 3.3 Memory-Augmented Reasoning: The selected shots and their CoTs form a tailored few-shot prompt that the LLM uses to evaluate the target interaction.This retrieval-augmented prompt is intended to support precise, human-like evaluation through relevant prior reasoning experiences.

4 ASSEBench: Agent Safety & Security Evaluator Benchmark

ASSEBench is a comprehensive benchmark for evaluating LLM-based evaluators across agent safety and security, including structured annotations and separate standards for ambiguous cases.

  • 4.1 Benchmark Overview: The benchmark organizes cases by scenario, risk type, and behavior mode to support granular analysis of evaluator performance.These categories describe operational context, potential harms or threats, and characteristic agent response or action patterns.
  • 4.1 Benchmark Overview: ASSEBench jointly evaluates agent safety and security across four subsets containing annotated interaction records.The benchmark includes one security subset and three safety subsets, with binary safe/unsafe labels.
  • 4.2 Benchmark Construction: ASSEBench is constructed through data standardization, expert annotation with LLM assistance, and iterative balanced screening.Records originate from reproduced agent safety and security benchmarks and are refined into the final benchmark subsets.
  • 4.2 Benchmark Construction: Ambiguous safety cases are re-evaluated under Strict and Lenient criteria, enabling comparison of evaluator behavior across judgment standards.The two derived subsets are formed from ASSEBench-Safety by applying different labeling criteria to borderline entries.

5 Experiments and Results

Experiments evaluate AgentAuditor across models, datasets, ablations, and human judgments. Results show broad performance gains, human-level accuracy, adaptive handling of ambiguous standards, and strong component synergy.

  • Experimental Setup: Experiments span diverse models, ASSEBench, R-Judge, and manually annotated screening datasets to assess generalizability and applicability.The model set includes open-source and proprietary systems, reasoning models, and safety-specialized evaluators.
  • Empirical Results: 48.2% increase in F1-score is achieved by Gemini-2 on ASSEBench-Safety with AgentAuditor.Llama-3.1-8B accuracy on ASSEBench-Security rises from 49.69% to 81.03%.
  • Empirical Results: 96.1%, 99.4%, and 93.2% accuracy is achieved by Gemini-2 with AgentAuditor on R-Judge, AgentHarm, and ASSEBench-Security, respectively, approaching or surpassing single-human averages.On R-Judge, the combination also outperforms ShieldAgent by 15.1%.
  • Empirical Results: AgentAuditor narrows performance gaps between ASSEBench-Strict and ASSEBench-Lenient without manual intervention.GPT-4.1’s gap narrows from 10.3% to 5.6%, while Gemini-2’s narrows from 13.2% to -2.8%.
  • Ablation Studies: Feature Tagging, Clustering, Few-shot learning, and CoT are independently evaluated as core components of AgentAuditor.Top-K selects three similar reasoning-memory samples, while Random selects three samples at random.
  • Ablation Studies: CoT or Few-shot alone underperform the baseline, whereas their combination improves performance through complementary guidance and reasoning.Feature Tagging, Clustering, and CoT together provide semantic relevance, representative examples, and reasoning templates.
  • Human Evaluation: Human evaluation assesses both binary judgment accuracy and the human-likeness of generated risk reasoning.Gemini-2 with AgentAuditor reaches the average accuracy level of a single human evaluator on R-Judge, AgentHarm, and ASSEBench-Security.

6 Limitation

The paper identifies limitations involving label quality, reasoning and resource demands, heuristic parameters, binary-label evaluation, and English-only interaction records.

  • AgentAuditor’s effectiveness depends on the quality of labels in its reasoning-memory shots.Defective labels reduce the improvement achieved by the method.
  • CoT-based evaluation requires reasoning ability and consumes more resources.
  • Heuristic parameters perform well in current experiments, but their universality and interpretability require further verification.
  • Although AgentAuditor theoretically supports more than binary labels, the paper evaluates it only with binary labels.
  • ASSEBench interaction records are entirely in English, limiting application to multilingual agent safety.Different languages may introduce unique risks.

7 Conclusion & Future Work

The paper concludes by positioning AgentAuditor and ASSEBench as responses to the challenges of evaluating agent safety and security, while outlining mechanisms and future extensions.

  • 7 Conclusion & Future Work: ASSEBench jointly evaluates agent safety and security through systematically organized interaction records and ambiguity-aware classification.
  • 7 Conclusion & Future Work: Future work targets broader AgentAuditor generalizability, more comprehensive ASSEBench coverage, and dynamic-memory agent defense.
  • A Challenges for Automated Evaluation: Automated evaluators face four challenges: action-based risk, accumulated harmless operations, ambiguous harm boundaries, and non-universal criteria.These challenges involve environmental consequences, long interaction histories, uncertain context, and application-specific utility–risk trade-offs.
  • C First Integer Neighbor Clustering Hierarchy (FINCH): FINCH selects representative experiences for AgentAuditor’s reasoning memory through hierarchical, parameter-free clustering.It links first-neighbor relationships, recursively merges clusters, and supplies varied examples for evaluation guidance.
  • 7 Conclusion & Future Work: AgentAuditor equips LLM evaluators with structured interaction understanding and human-like reasoning supported by memory.Its framework uses representative-shot selection, structured memory, retrieval augmentation, and generated CoT traces.
  • B.1 Chain-of-Thought (CoT): AgentAuditor generates CoT traces from representative shots and retrieves relevant reasoning experiences for subsequent evaluations.The prompts combine interaction content, ground-truth labels, and fixed templates before memory-augmented reasoning.

F.2 Benchmarks for Agents

Existing agent benchmarks cover safety or security but often use rigid or narrow evaluation schemes. ASSEBench addresses this gap through broad coverage, collaborative annotation, balanced sampling, and strict versus lenient handling of ambiguity.

  • Existing benchmarks: AgentDojo evaluates prompt-injection robustness across 124 safe and 629 attack tasks, but repetition and rigid tool-invocation rules limit nuanced evaluator assessment.Its rule-based classifier can label an interaction unsafe even when the risky action is never executed.
  • Existing benchmarks: AgentHarm covers harmful-request resistance with 11 risk types, 110 behavior modes, and 500 reported tasks, but its public dataset currently contains 176 tasks.It uses continuous safety scores and refusal rate, while refusal decisions can be easier to judge than overall interaction safety.
  • Existing benchmarks: AgentSecurityBench spans 10 scenarios, more than 400 tools, 400 tasks, and 23 attack or defense methods across multiple agent operation stages.Its scope includes system prompting, user prompts, tool use, and memory retrieval.
  • Existing benchmarks: AgentSafetyBench contains 349 environments and 2,000 cases covering eight safety-risk categories and ten failure modes, but its fine-tuned evaluator showed lower reproduced accuracy than reported.The reproduced model tended to classify safe interactions as unsafe.
  • ASSEBench development: ASSEBench development standardizes interaction generation, uses three LLMs with Balanced Allocation, and applies human-computer collaborative classification.The resulting process includes differentiated Strict and Lenient treatment of ambiguous safety records.
  • Metric boundary: Binary safety and security metrics simplify comparison but cannot represent risk severity, judgment confidence, or unequal harm across risk types.This limitation motivates caution when interpreting safe/unsafe classifications.

J.2 Metrics in the Human Evaluation of AgentAuditor

The human evaluation rates AgentAuditor and baseline risk descriptions for reasoning quality using expert Likert judgments. It also evaluates annotation consistency with Krippendorff’s Alpha, while benchmark metrics retain distinct binary-classification trade-offs.

  • Human rating protocol: Six AI safety and security experts independently rate each reasoning trace on logical structure, reasoning soundness, and completeness using a 1–5 Likert scale.The dimensions assess followability, validity, and coverage of relevant safety or security aspects.
  • Score aggregation: Each item’s score on each dimension is the average of all annotators’ ratings, with the three dimensions analyzed independently.These averages support subsequent comparisons between AgentAuditor and baseline methods.
  • Reliability analysis: Krippendorff’s Alpha measures inter-rater reliability for the Likert ratings, with values above 0.67 typically acceptable and above 0.8 indicating good agreement.The reliability analysis validates using average ratings for system comparison.
  • Benchmark metrics: Attack success rate measures successful harmful elicitation but depends strongly on the chosen attacks, limiting generalization to novel threats.ASR directly measures vulnerability within the evaluated attack set.
  • Benchmark metrics: Refusal rate is simple to assess because it focuses on explicit refusals, but it provides limited information about the safety of complex compliant responses.The passage contrasts refusal judgments with nuanced overall safety assessment.
  • Benchmark metrics: Safety Score uses LLM-generated safe or unsafe labels to provide scalable assessment, but binary labels create challenges for nuanced behavior judgments.The supplied passage introduces SS as a broader alternative to ASR or RR.

K Detailed Experimental Results

AgentAuditor is evaluated with and without its augmentation across eight datasets and twelve models. Reported results support broad improvements, while ShieldAgent’s reproduced performance falls short of its original claim and remains difficult to verify fully.

  • Experimental scope: Eight datasets and twelve models are included in the complete comparison of AgentAuditor and baseline evaluators.The datasets include four ASSEBench subsets plus R-Judge, AgentHarm, AgentSecurityBench, and AgentSafetyBench.
  • ASSEBench results: AgentAuditor automatically selects 73 representative shots for ASSEBench-Security and 72 for each other ASSEBench subset.The reported ASSEBench comparisons contrast augmented (+AA) and original (Ori) results.
  • Cross-benchmark results: AgentAuditor automatically selects 24, 19, 71, and 78 representative shots for R-Judge, AgentHarm, AgentSecurityBench, and AgentSafetyBench, respectively.The corresponding comparisons are reported as +AA versus Ori results.
  • Metric interpretation: Refusal-rate judgments by GPT-4o are more accurate than its judgments of complete interaction safety, but refusal rate remains limited as an overall safety metric.The comparison concerns operation refusal rather than the safety of entire interaction records.
  • ShieldAgent discrepancy: ShieldAgent’s reported 91.5% accuracy exceeds direct GPT-4o performance at 75.5%, but reproduction tests did not reach the claimed ShieldAgent level.The discrepancy persisted across multiple benchmarks, including tests on AgentSafetyBench.
  • ShieldAgent discrepancy: The authors suspect overfitting, data leakage, or test-set bias may explain ShieldAgent’s discrepancy, but unavailable internal test data prevents further investigation.Their reproduction followed the original model version, loading method, and prompt settings.

M Impact of Different Embedding Models

Embedding-model choice affects AgentAuditor’s performance, but embedding dimensionality matters more than simply increasing model size. The study selects Nomic-Embed-Text-v1.5 with 512 dimensions for its strong performance and low resource consumption.

  • Larger embedding models improve AgentAuditor performance only to a limited extent, with noticeable diminishing returns.
  • At 512 dimensions, Stella-en-1.5B-v5 provides only marginal accuracy improvement over Nomic-Embed-Text-v1.5 despite nearly 10 times greater parameter count and resource consumption.
  • Embedding dimensionality is a more critical factor than model size because an appropriate dimension can produce significant performance gains.
  • Nomic-Embed-Text-v1.5 with 512 dimensions is selected for the rest of the work because it combines exceptional performance with low resource consumption.

N Impact of Different Clustering Methods

AgentAuditor’s representative-shot selection benefits from clustering, with FINCH providing the strongest reported performance among the tested methods. The experiments also examine how label noise affects the resulting reasoning memory.

  • N Impact of Different Clustering Methods: 96.31 F1-score and 96.10 accuracy make FINCH the best-performing clustering method in the comparison.
  • N Impact of Different Clustering Methods: Clustering-based AgentAuditor variants significantly outperform the base model without memory augmentation, supporting representative-shot selection across methods.
  • N Impact of Different Clustering Methods: FINCH is preferred because it automatically determines hierarchical cluster structure and granularity without manual hyperparameter tuning.
  • O.1 Robustness to Label Noise: With up to 6 incorrectly labeled shots, AgentAuditor retains an F1-score of 0.8536 and accuracy of 0.8333 while outperforming the baseline.
  • O.1 Robustness to Label Noise: At 50% incorrect labels, AgentAuditor falls below the baseline, indicating a threshold for substantial performance degradation.

O.2 Robustness Against Adversarial Attacks

AgentAuditor is evaluated against progressively stronger poisoning and retrieval-distribution challenges. The reported results indicate robustness to adversarial attacks, cross-domain memory, and imperfect retrieval, although performance can degrade under distribution mismatch.

  • O.2 Robustness Against Adversarial Attacks: The evaluation covers three escalating attack tiers: dataset poisoning, direct reasoning-memory poisoning with misleading CoTs, and white-box replacement of a final CoT example.
  • O.2 Robustness Against Adversarial Attacks: FINCH filters all malicious samples at a 1.1% dataset-poisoning rate, while AgentAuditor retains a performance advantage even at 9.9% poisoning.
  • O.2 Robustness Against Adversarial Attacks: Under reasoning-memory poisoning, AgentAuditor still surpasses the unattacked baseline when 8 of 24 memory shots are malicious.
  • OOD and Retrieval Robustness: Cross-dataset memory transfer from R-Judge to ASSEBench-Safety and ASSEBench-Security tests performance when highly relevant memory matches are unavailable.
  • OOD and Retrieval Robustness: AgentAuditor maintains strong performance with lower-ranked or randomly selected shots, supporting robust generalization when retrieval quality is imperfect.
  • OOD and Retrieval Robustness: The authors conclude that deployment on new domains is practical before domain-specific memory is fully constructed.

P Resource Usage Analysis

AgentAuditor trades higher inference cost for substantially lower preparation costs than fine-tuning, requiring far fewer manually annotated records. The resource analysis compares direct LLM judgment, fine-tuning, and memory construction.

  • P Resource Usage Analysis: AgentAuditor requires 24 manually annotated records to build its reasoning memory, compared with 4000 records required by the ShieldAgent fine-tuning baseline.
  • P Resource Usage Analysis: AgentAuditor has a significant advantage in initial resource costs compared with fine-tuning.
  • P Resource Usage Analysis: The comparison measures manually annotated records and GPU hours separately for pre-inference and inference consumption.
  • P Resource Usage Analysis: Although AgentAuditor uses more GPU hours during inference than base models, the authors report exceptional performance returns from this investment.
  • P Resource Usage Analysis: AgentAuditor’s end-to-end computational demand is reported as considerably lower than fine-tuning-based methods because fine-tuning carries substantial training overhead.

Q.1 Judgment Performance Evaluation

The evaluation compares AgentAuditor with human evaluators on binary agent safety and security judgments, while also assessing whether its reasoning resembles human risk analysis. It additionally examines the quality of AgentAuditor-generated reasoning traces across multiple dimensions.

  • Judgment performance: AgentAuditor’s judgment evaluation compares its binary safe/unsafe decisions with human experts across R-Judge, AgentHarm, AgentSecurityBench, AgentSafetyBench, and ASSEBench subsets.The study samples 50 unique entries from each dataset and obtains independent expert annotations.
  • Reasoning-process evaluation: Human experts evaluate AgentAuditor CoT traces against direct LLM risk analyses on ASSEBench-Security and ASSEBench-Safety using Completeness, Reasoning Soundness, and Logical Structure.The experiment samples 50 cases from each subset and evaluates outputs generated with and without AgentAuditor.
  • Scalability: AgentAuditor’s scalability analysis considers both dataset size and domain diversity, with one-time memory construction and domain-agnostic components supporting broader application.After reasoning memory is built, evaluating each new case depends on the small-scale memory rather than the original dataset size.
Loading 2506.00641v3…