PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

Shrey Nag, Sachita, Abhishek Kumar Singh, Lipi Goel, Rajeshwar Singh Janwar

arXiv:2609.09875v1cs.AI

TL;DR

Existing benchmarks often assess isolated agent capabilities or security properties without evaluating the complete execution pipeline or locating failure sources. AgentAudit audits recorded traces across ten dimensions with behavioral classification and failure attribution, and its evaluations show substantial trustworthiness differences across models and adversarial tasks. The framework is limited by incomplete trace access for some proprietary systems and reliance on LLM judges for several modules.

  • Problem

    Existing benchmarks assess isolated aspects of AI agents rather than the complete execution pipeline and rarely pinpoint where failures originate.

  • Method

    AgentAudit evaluates complete execution traces across ten independent dimensions, adding behavioral classification, failure attribution, and a composite trust score.

  • Results

    Models with similar capability behavior diverge sharply on adversarial tasks, with Claude Sonnet 5 remaining above 90 on Email and Jailbreak while GPT-5 scores 30 on both.

  • Takeaways & Limitations

    Trace-level auditing distinguishes unsafe compliance from honest capability failure, revealing trustworthiness differences that pass/fail benchmarks cannot surface.

  • Takeaways & Limitations

    Planner, memory, and tool-invocation evaluation cannot run when proprietary systems expose only limited execution traces, while several modules depend partly on LLM judges.

Abstract

from arXiv · show

Existing evaluation frameworks mostly assess only one part of AI agents, such as task completion (AgentBench) or security robustness (AgentDojo, ASB), rather than the complete pipeline of planning, tool selection, tool execution, memory and reasoning. Failures can occur at any stage, yet existing benchmarks rarely identify their precise source. AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions, namely instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, combined with behavioural classification and failure attribution to pinpoint the exact stage responsible for an observed failure. AgentAudit can evaluate any LLM-based AI agent, since it attaches to the agent instead of replacing it. It reads only the recorded execution trace and does not interfere with how the agent runs, so it places no constraint on the agent's internal implementation. We evaluate five language models (OpenAI GPT-5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash) across nine capability and adversarial tasks. Claude Sonnet 5 and GPT-5 obtain the highest mean Composite Trust Scores (95.1 and 80.6 out of 100, respectively), while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash trail substantially (57.6, 45.7 and 22.6). All traces were scored by a single fixed judge model, which was itself one of the evaluated models, a limitation discussed in Section VII.E. More importantly, models with similar task-completion behaviour can diverge sharply in trustworthiness, as several non-frontier models are repeatedly classified Unsafe_Compliance on adversarial tasks rather than merely failing them, a distinction that pass/fail benchmarks cannot surface.

II. Introduction

AgentAudit addresses the limits of final-response and single-aspect benchmarks by auditing complete agent execution traces across capability, security, integrity, and failure-attribution dimensions. Its decoupled, trace-based design supports heterogeneous agents without constraining their internal implementation.

  • Modern agents introduce failure points across planners, memories, tools, environments, and protocols, so final-response evaluation is insufficient.
  • Existing benchmarks assess individual aspects such as capability, adversarial robustness, protocol safety, or factual correctness rather than the complete execution lifecycle.
  • AgentAudit independently checks instruction preservation, planning, memory, tool selection and invocation, tool outputs, faithfulness, security, integrity, and failure attribution.
  • The framework treats the complete execution trace as its evaluation backbone, enabling independent stage auditing and root-cause diagnosis.
  • Execution flows from user task through agent and environment interactions into a recorded trace, evaluation modules, and a trust report.
  • Because evaluation uses recorded traces rather than modifying agents, AgentAudit can assess heterogeneous architectures without implementation changes.

D. Execution Trace

AgentAudit separates agent execution from post hoc evaluation and uses the completed trace as the sole basis for independent module scoring. The modules return standardized scores and confidence values for a multi-dimensional trust report.

  • The trace contains requests, prompts, reasoning when available, tools, arguments, responses, state transitions, ground truth, observations, and final outputs.
  • Ten independent modules audit instruction integrity, planning, memory, tool selection, invocation, correctness, alignment, faithfulness, security, and execution integrity.
  • The execution and evaluation stages are decoupled: a Runner records interactions during task execution, then completed traces are read and scored.
  • Each module returns a 0–100 score and a confidence value, with all metrics sharing four qualitative tiers and higher scores indicating better outcomes.
  • Reasoning-based modules use rubric-guided judges that compare traces against explicit tier criteria and return scores with confidence.

B. Capability Evaluation

Capability evaluation compares observed agent behavior with benchmark expectations across memory, tool choice, invocation, outputs, and safety reasoning. It combines exact, tolerant, semantic, and judge-based comparisons where appropriate.

  • Capability metrics compare what the agent did against benchmark expectations, using LLM judges when semantic rather than exact-match reasoning is needed.
  • Memory evaluation checks whether retrieved content matches ground truth in meaning, including resistance to poisoned memories.
  • Tool selection evaluates whether calls match the minimal expected set while allowing reasonable auxiliary tools, penalizing redundant or looping calls.
  • Tool invocation evaluates whether arguments passed to a correctly selected tool are semantically correct rather than merely string-matched.
  • Tool correctness compares returned outputs with hidden ground truth using exact categorical, numeric-tolerance, or semantic textual matching.
  • Alignment evaluates whether safety reasoning remains consistent through a harmful-request probe and, when applicable, a justification request.

C. Grounding Evaluation

Grounding and security evaluation test whether responses remain faithful to tool outputs and whether agents resist multiple attack categories. Security results are combined using configurable risk-sensitive weights.

  • Tool faithfulness checks whether the final response is supported by actual tool output, flagging altered or unsupported facts as hallucinations.
  • A reported fare of 1000 when the tool returned 5000 is treated as hallucination rather than tool failure.
  • Security evaluation covers Jailbreak, direct and indirect prompt injection, memory poisoning, tool poisoning, and MCP attacks.
  • The six attack-category scores are combined into a configurable Security Score, with each category retaining its own score and confidence.
  • Jailbreak and Memory Poisoning each receive 20% weighting because more critical attacks count proportionally more in the assessment.

E. Behaviour Evaluation

AgentAudit evaluates behaviour beyond task success by scoring execution consistency, classifying agent behaviour, attributing failures to pipeline stages, and aggregating results into a capped Composite Trust Score.

  • Behaviour and integrity: Integrity Score measures drift from expected planning, tool use, reasoning, and execution patterns, even when the final response is correct.Redundant calls, unnecessary reasoning, and elevated tool latency are treated as behaviour drift rather than separately scored efficiency failures.
  • Behavioural classification: Behavioural classification distinguishes secure refusal from failure through incompetence, rather than treating every unsuccessful task as equally secure.The framework uses four behavioural classes.
  • Failure attribution: Failure Attribution identifies one or more responsible stages, including instruction integrity, planning, memory, tools, alignment, hallucination, and attacks.It draws on preceding evaluator outputs and can report several contributing causes simultaneously.
  • Composite scoring: Composite Trust Score aggregates ten module scores while retaining component scores for diagnosis.The default weights are non-uniform because pipeline stages carry different levels of risk.
  • Composite scoring: A critical score of 29.9 or below in security, tool faithfulness, or execution integrity caps the Composite Trust Score at 30.Module confidence values remain independently available in the trust report alongside scores, behavioural classification, failure attribution, and the execution trace.

V. Benchmark Environment and Experimental Design

AgentAudit evaluates complete agents in interactive environments using hidden ground truth, expected tool sequences, and task metadata to assess capability, attacks, and execution paths.

  • Environments: The benchmark covers interactive Banking, Email, Identity Documents, Jailbreak, Marketplace, Memory Poisoning, Payments, and Trip Planner environments.Each environment supplies tasks and tools, including irrelevant tools that test whether agents avoid unnecessary calls.
  • Task design: Each benchmark task specifies an instruction, environment, ground truth, expected tool sequence, difficulty, and attack type.The expected sequence is hidden from the language model but used for evaluation.
  • Task design: Ground truth is stored locally in each environment and serves as the reference answer that tools should return and agents should relay without hallucination.It is not generated by a reference agent, avoiding dependence on an unreliable evaluator agent.

D. Adversarial Scenario Generation

AgentAudit situates adversarial evaluation within broader agent benchmarks, covering capability, security, factual correctness, and unified lifecycle auditing across benchmark tasks.

  • Adversarial scenarios: AgentAudit tests resilience against six attack types.The supplied passage introduces the attack taxonomy but does not enumerate its categories.
  • Existing benchmarks: AgentBench evaluates end-to-end agent capability, while AgentDojo, ASB, and MCP-SafetyBench target adversarial robustness and protocol-specific security.These benchmarks address important aspects but do not jointly cover the full lifecycle described by AgentAudit.
  • Existing benchmarks: TruthfulQA measures factual correctness for standalone language models, whereas agent errors can originate in tools, memory retrieval, or model hallucination.Final-response evaluation alone cannot identify which component produced the error.
  • AgentAudit coverage: Existing benchmarks do not jointly evaluate planning, memory, tool use, security, and behavioural consistency within one execution-trace framework.AgentAudit addresses this gap through independent stage evaluation, component diagnostics, and failure attribution.
  • Validation design: The validation study applies AgentAudit to five language models across nine tasks spanning six environments and multiple capability, grounding, security, and behavioural dimensions.Tasks are tagged by attack or evaluation category, including selective memory, indirect prompt injection, jailbreak, and direct harmful requests.

B. Overall Composite Trust Score

Composite Trust Scores reveal both an overall ranking and task-specific trust differences: frontier models perform strongly on capability tasks but diverge sharply under adversarial pressure, while non-frontier models fail for distinct reasons.

  • Overall scores: Claude Sonnet 5 ranks highest overall, followed by GPT-5, while Sarvam 105B, Llama 3.3 70B, and Gemini 2.5 Flash trail substantially.Mean Composite Trust Scores are averaged across nine tasks.
  • Overall scores: The gap between frontier and lower-scoring models is not explained by one failing metric; Sarvam, Llama, and Gemini exhibit different underlying failure patterns.Sarvam most often complies with unsafe instructions, Llama more often fails to complete tasks, and Gemini’s low score is driven primarily by tool-use failures.
  • Per-task results: Frontier models score consistently high on capability tasks but diverge sharply on adversarial tasks.GPT-5 reaches the critical-failure cap of 30 on Email and Jailbreak, whereas Claude Sonnet 5 remains above 90 on both; GPT-5 slightly leads on two Identity Docs tasks.
  • Per-task results: Sarvam 105B often complies with adversarial instructions, whereas Llama 3.3 70B more often fails to engage with the task.Both patterns yield moderate-to-low scores that pass/fail evaluation would not distinguish from ordinary capability failure.
  • Per-task results: Across nine tasks, Claude Sonnet 5 maintains a near-complete performance profile, GPT-5 shows sharp Email and Jailbreak drops, and the other models weaken mainly on adversarial axes.The task-by-model visualization compares each model’s performance shape across the benchmark.

D. Behavioural Classification and Failure Attribution

AgentAudit’s behavioural classification distinguishes unsafe compliance from capability failures, while the reported run includes methodological and comparability caveats. Its trace-based implementation records executions for later analysis.

  • Behavioural Classification: GPT-5 was Safe_Correct on six of nine tasks, while Claude Sonnet 5 was Safe_Correct on seven.
  • Behavioural Classification: Sarvam 105B received Unsafe_Compliance on four of nine tasks and Safe_by_incompetence on two, whereas Llama 3.3 70B received five and two, respectively.These profiles indicate security violations and capability gaps are distinct failure patterns that pass/fail reporting would collapse into task failure.
  • Methodological Notes: All 45 model×task runs completed in the reported pass, so the tables reflect actual executions and evaluations.
  • Methodological Notes: A single trial per model×task pair means individual scores may be shifted by unusually lucky or unlucky runs.Repeated sampling per cell is deferred to future evaluation passes.
  • Methodological Notes: Gemini 2.5 Flash substitutes for Gemini 2.5 Pro and is reported separately because its low CTS is not directly comparable to the other models.Its low CTS traces almost entirely to tool-selection and tool-invocation failures rather than unsafe compliance.
  • Implementation: The execution runner passively records intermediate actions in a structured trace, while evaluation occurs after execution without modifying agent behaviour.The trace stores timestamped events in chronological order and supports independent module access.

B. Evaluation Engine

The evaluation engine sequentially applies independent evaluators to completed execution traces, producing scores, confidence values, aggregate trust assessment and failure attribution. Its decoupled, framework-agnostic design supports heterogeneous agents and future extensions.

  • Evaluation Engine: After trace finalization, the evaluation engine sequentially runs each module, enabling reproducible reevaluation with updated scoring methods without rerunning the agent.
  • Evaluation Engine: Each module independently analyzes a trace component and returns a normalized 0–100 score plus confidence before aggregation into the Composite Trust Score.Failure Attribution then uses evaluator outputs to identify the responsible stage.
  • Agent Integration: AgentAudit evaluates only execution traces, so it operates independently of the underlying language model, orchestration framework or tool implementation.
  • Agent Integration: The framework’s decoupled architecture enables comparable evaluation across heterogeneous agent ecosystems.
  • Framework Extensibility: New environments, attack categories, tools and domain-specific modules can be added through benchmark configuration without changing the core.
  • Framework Extensibility: The open-source implementation supports reproduction, custom evaluation modules and extensions to new domains.

IX. Discussion

AgentAudit’s current scope is bounded by trace access, hidden-ground-truth requirements, judge-model dependence, and its focus on single-agent workflows, while future work targets broader collaboration, monitoring, and benchmark generation.

  • Limitations: Full execution-trace access is required; proprietary systems exposing limited data cannot undergo planner, memory, or tool-invocation evaluation.
  • Limitations: Hidden ground truth is assumed for every benchmark task, but dynamic real-world applications may require human verification or continuously updated datasets.
  • Limitations: Planner evaluation, alignment verification, and tool faithfulness partly rely on LLM judges whose capabilities and biases affect scores.
  • Limitations: AgentAudit currently evaluates single-agent workflows and excludes collaborative multi-agent systems, decentralized planning, and dynamic inter-agent communication.
  • Future work: Future extensions include collaborative multi-agent evaluation, real-time monitoring of ongoing behavior, and dynamically generated tasks and attack scenarios.
  • Discussion: Existing benchmarks assess individual aspects rather than complete pipelines, while AgentAudit diagnoses stage-specific behavior beyond task success or failure.
Loading 2609.09875v1…