PaperlayerView exact revision on arXiv

Source-linked AI summary

$A^2E$ : An End-to-End Agent Auditing Engine

Haoning Wang, Mingxun Zhang, Chenyue Yu, Yingjun Shang, Xia Hu, Guanchu Wang, Na Zou

arXiv:2608.07346v2cs.AI

TL;DR

Existing evaluation frameworks cover only part of harness evaluation, although deployed-agent capability depends on the harness as well as the model. A^2E combines task integration, trajectory monitoring, and multidimensional evaluation; its experiments find no harness–model configuration that dominates across tasks.

  • Problem

    Existing frameworks typically address only part of harness evaluation, while evaluating the model alone does not fully capture deployed-agent capabilities.

    Source: §1 · p. 3

  • Method

    A2E unifies task composition through ATP, standardized trajectory monitoring, and multidimensional evaluation in an end-to-end engine.

    Source: §1 · p. 3, §2 · p. 3, §3 · p. 5

  • Results

    Harness–model configurations vary across benchmarks, and no configuration consistently performs best across all tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Harness comparisons should account for task and model context rather than assume a globally rankable configuration.

    Source: §1 · p. 3

    The harness–benchmark matrix is presented to demonstrate end-to-end execution and scoring, not to rank frameworks.

    Source: §6.1 · p. 11

Abstract

from arXiv · show

With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce $A^2E$ (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. $A^2E$ leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, $A^2E$ systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with $A^2E$ further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.

1 Introduction

A2E unifies benchmark execution, trajectory monitoring, and lifecycle-aligned evaluation for agent harnesses. Its study finds that harness–model performance varies by task and that process and operational metrics distinguish harnesses beyond endpoint correctness.

1 Introduction

  • Existing evaluation frameworks typically address only part of the workflow, motivating a lightweight, minimally invasive, harness-agnostic evaluation substrate.Inspect AI (AI Security Institute, UK, 2024) supports benchmark orchestration and scoring but relies on harness-specific integrations; Phoenix (Arize AI, 2024) provides observability but not end-to-end benchmark execution and verification.Source: §1 · p. 3
  • A2E combines benchmark–harness task composition, standardized execution-trace capture, and evaluation metrics aligned to agent lifecycle stages.Its Task Layer uses ATP to reduce the need for pairwise benchmark–harness implementations; its Monitor Layer captures OpenTelemetry-compatible spans, and its Evaluation Layer registers metrics by lifecycle stage and dimension.Source: §1 · p. 3
  • A2E stores trajectories, metric definitions, and results in a database, supporting incremental, persistent, longitudinal evaluation across runs.The paper also describes A2E’s core implementation as substantially smaller than general-purpose systems such as Inspect AI (AI Security Institute, UK, 2024) and Phoenix (Arize AI, 2024).Source: §1 · p. 3
  • A2E finds no universally dominant harness–model configuration, while process and operational metrics distinguish harnesses more finely than endpoint correctness.The success–efficiency frontier varies across GDPVal, MMLU-Pro, and τ3-bench, and the strongest configuration depends on both model and task.Source: §1 · p. 3

2 Overview

A^2E organizes agent evaluation into a closed workflow spanning benchmark preparation, monitored execution, multidimensional assessment, and result analysis. Its framework and runtime pipeline use centralized data handling to connect these stages.

2 Overview

  • The overview defines three components—Task, Monitor, and Evaluation—that connect benchmark preparation, agent execution, trajectory collection, multidimensional evaluation, and result analysis.Together they form a closed workflow.Source: §2 · p. 3, Figure 2 · p. 4

2.1 Framework

  • The Task Layer organizes benchmarks by time, category, and difficulty, and uses ATP execution bundles to provide reproducible environments and integrate benchmarks without changing the remaining pipeline.Each bundle contains a sandbox definition, task dataset, experiment configuration, and runtime environment.Source: §2.1 · p. 4
  • The Monitor Layer normalizes access to ready-to-run and SDK-based agents, then records each reasoning–action–observation cycle, including calls, transitions, errors, tools, latency, tokens, and artifacts.Runs are captured as ordered event sequences that preserve outcomes and intermediate decisions.Source: §2.1 · p. 4
  • Evaluation combines outcome and trajectory assessment, using rule-based metrics and LLM-as-judge scores to evaluate correctness and agent behavior.Rule-based measures include accuracy, task and tool success rates, latency, token usage, cost, and trajectory steps; judges assess result and reasoning quality, tool use, instruction following, relevance, completeness, and safety. Scores and annotations are stored as structured records for trajectory inspection and aggregated comparisons.Source: §2.1 · p. 5

2.2 Pipeline

  • The runtime pipeline connects Task, Server, Evaluation, and UI through sandboxed execution, automatic monitoring, centralized record writing, and trace- and outcome-level evaluation.The components are loosely coupled; the UI is read-only and displays traces, scores, and aggregated results from the server database.Source: §2.2 · p. 5

3 Monitor Layer

Monitor captures intermediate agent behavior as structured traces, making both execution outcomes and the processes behind them observable. Its instrumentation and span-based traces support analysis beyond the agent’s final response.

3.1 Instrumentation

  • Instrumentation combines framework adapters, a shared semantic vocabulary, and spans to represent what the agent did, when it happened, and which operation led to it.The SDK maps framework-specific activity into semantic categories, while spans preserve temporal boundaries and parent–child structure; new frameworks can reuse this foundation with specialized adaptations.Source: §3.1 · pp. 5–6

3.2 Span-Based Traces

  • Monitor adopts OpenTelemetry spans to organize an agent run as a hierarchy of operations, preserving both action sequence and causal relationships among reasoning, model calls, tools, and skills.The top-level span represents the run; its descendants record increasingly specific activities, including tools triggered by model calls and results returned to the surrounding chain.Source: §3.2 · pp. 6–7
  • Span durations, statuses, and context reveal delays and failures, while separate operation records enable comparisons of time use and behavioral divergence across runs.Traces also show the reasoning path, model consultations, and external capabilities used, enabling evaluation of both outcomes and execution processes.Source: §3.2 · p. 7

4 Task Layer

The Task Layer connects agent harnesses to benchmarks through ATP, a shared interface that separates benchmark adaptation from harness execution. It also generates normalized task traces linked to framework-level execution details.

4 Task Layer

  • ATP gives benchmarks and harnesses a shared boundary for task representation, agent-facing interaction, and execution records.It is an internal software protocol that separates benchmark adaptation from harness execution.Source: §4.1 · p. 7

4.1 Agent Task Protocol (ATP)

  • ATP uses TaskInput, AgentBinding, AgentRunner, and TaskTrace objects to connect benchmark adapters with harness execution.TaskInput holds task instructions and state; AgentBinding provides tool schemas, execution, and prompt construction; TaskTrace records run status, final answer, and ordered tool calls.Source: §4.1 · pp. 7–8

4.2 Agent Harnesses & Benchmarks

  • The registry integrates multiple agent frameworks through ATP runners, while benchmark adapters map source items into task inputs and bindings.The report groups 23 benchmarks into coding, conversational, research, and computer-use areas, with text, tool-use, and sandbox task formats; registry support does not guarantee end-to-end validation for every framework–benchmark pair.Source: §4.2 · p. 8

4.3 Trajectory Generation

  • Trajectory generation samples benchmark tasks, runs each through an ATP runner, and records normalized traces linked to framework-level span trees when available.The CLI samples 40 tasks without replacement by default and stores run metadata, including the sample seed and task identifiers; the native and CLI paths differ in orchestration and root-span ownership.Source: §4.3 · p. 8

5 Evaluation Layer

A^2E’s evaluation layer addresses outcome-only assessment’s diagnostic, extensibility, and aggregation limits with a lifecycle-aligned, extensible, database-backed architecture. Its heterogeneous catalog organizes semantic, behavioral, and deterministic trace metrics across agent capabilities and operational properties, without claiming to define all agent quality.

5.1 Lifecycle-Aligned Evaluation

  • The lifecycle taxonomy separates process-level reasoning and actions, outcome-level final results, and lifecycle-level operational properties, enabling failure localization by execution stage and capability dimension.Its four stages are Reasoning (Task, Flow, Logical), Action (Tool, Skill, Memory), Final Answer (Answer Correctness, Task Completion), and Runtime Quality (Efficiency, Safety).Source: §5.1 · p. 9

5.2 Extensible Evaluation

  • Metrics are registered by lifecycle stage and dimension independently of evaluator implementation, so developers can add properties without changing the benchmark runner, harness, or existing metrics.New metrics may be benchmark-specific while remaining comparable through the shared taxonomy; examples include tool selection under Tool and latency or resource indicators under Efficiency.Source: §5.2 · p. 10

5.3 Scalable Evaluation

  • Database-backed records link task, run, turn, tool-call, and metric data; indexed queries, transactional updates, and incremental recomputation support scalable, resumable evaluation.Stored trajectories can be re-evaluated with new metrics, judge models, or aggregation policies without rerunning agents or repeating expensive API calls, while structured metadata supports provenance and reproducibility.Source: §5.3 · p. 10

6 Experiments

Across controlled harness comparisons, results vary by task and execution cost as well as correctness. Trajectory-level metrics and case studies expose execution behaviors behind those differences.

6.1 Harness–Benchmark Evaluation Matrix

  • All nine harnesses use the same DeepSeek-V4-pro (FP4) model and matched inference, tool, step-limit, and timeout settings across 23 benchmarks and 1,035 scored runs.Each benchmark contributes five identical task IDs per harness; full trajectories are recorded for 19 non-sandbox benchmarks, which contribute 855 runs.Source: §6.1 · p. 10, Table 1 · p. 11
  • 0.68 vs. 0.64: agno leads the full 23-benchmark average, while llama-index leads the 19 non-sandbox average at 0.77 vs. agno’s 0.74.The five-task cells have 0.20 score resolution and high per-cell variance; the table demonstrates end-to-end execution and scoring, not a definitive framework ranking.Source: Table 1 · p. 11, §6.1 · p. 11
  • Across eight instrumented harnesses, correctness ranges from 0.568 to 0.663 while mean token cost spans 2,063 to 7,319; agno scores highest on correctness and is second cheapest.The 855 non-sandbox trajectories are profiled across 13 metrics spanning planning, tool use, answers, and operational quality; other process dimensions move little on this benchmark mix.Source: Figure 6 · p. 12, §6.1 · p. 12

6.2 Cross-Benchmark Harness Comparison

  • Harness leaders vary by benchmark: CrewAI, OpenAI Agents, and AutoGen-AgentChat rank in GDPVal’s top three; OpenAI Agents, AutoGen-AgentChat, and LangGraph in MMLU-Pro’s; and LangGraph, Claude-Agent-SDK, and Google ADK in τ3-bench’s.Figure 7 ranks harnesses by the trade-off between higher success and fewer completion tokens, using GLM-5.2 as the common model.Source: §6.2 · p. 12, Figure 7 · p. 13
  • With GLM-5.2 fixed, harnesses differ in both task success and token use; τ3-bench separates them more than MMLU-Pro, and similar GDPVal success can come at different costs.The evaluation jointly considers effectiveness and efficiency, with a higher normalized trade-off score indicating a better balance.Source: §6.2 · p. 13

6.3 Case Study

  • Success-rate gaps between harnesses reach 0.20 on GDPVal, 0.30 on MMLU-Pro, and 0.66 on τ3-bench, despite using the same API model.Token use also varies; the paper attributes system-level differences to harness choices affecting prompt construction, tool interaction, state management, and execution control.Source: §6.3 · p. 14
  • On the same τ3-bench task, LangGraph scores 1.0 using 10,122 tokens, four LLM calls, and three tool calls, while CrewAI scores 0.0 using 96,704 tokens.CrewAI uses about 9.6× LangGraph’s tokens and makes five tool calls; its device-level actions miss the account-level cause, while LangGraph identifies it.Source: §6.3 · pp. 14–15, Table 2 · p. 14
  • With model, task, and initial environment held constant, the trajectories reveal context accumulation, inefficient tool selection, failure to execute goal-relevant actions, and incorrect termination as sources of the gap.CrewAI makes nine increasingly expensive LLM calls and accumulates 94,615 prompt tokens; LangGraph follows a shorter trajectory to the relevant diagnosis.Source: §6.3 · p. 15

Related papers

Loading 2608.07346v2…