Source-linked AI summary
$A^2E$ : An End-to-End Agent Auditing Engine
Haoning Wang, Mingxun Zhang, Chenyue Yu, Yingjun Shang, Xia Hu, Guanchu Wang, Na Zou
TL;DR
A2E addresses the challenge of systematically evaluating rapidly evolving agent harnesses beyond model-only or correctness-only assessments. It provides an end-to-end stack built on ATP, automatic trace monitoring, and lifecycle-aligned multidimensional evaluation. Experiments show substantial task-dependent variation among model–harness configurations, with no single combination consistently best across tasks.
Problem
Existing evaluations often reduce agent trajectories to task success or final-answer quality, limiting fine-grained assessment of harness behavior and reproducible harness-level evaluation.
Method
A2E combines ATP-based task composition, automatically instrumented standardized trajectory traces, and lifecycle-aligned metrics spanning execution stages and evaluation dimensions.
Results
No single harness–model configuration consistently dominates across tasks, while multidimensional metrics reveal differences in correctness, efficiency, planning, tool use, and operational quality.
Takeaways & Limitations
Harness effectiveness is model- and task-dependent, so evaluating deployed agents requires systematic trajectory-aware comparison rather than correctness alone.
Takeaways & Limitations
With only five tasks per cell, score resolution is 0.20 and per-cell variance is high; the evaluation matrix is not intended to rank frameworks.
Abstract
from arXiv · showhide
With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce $A^2E$ (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. $A^2E$ leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, $A^2E$ systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with $A^2E$ further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.
1 Introduction
A2E addresses fragmented harness evaluation with an end-to-end stack for task composition, trajectory monitoring, and lifecycle-aligned assessment. Its study finds that harness–model effectiveness varies across tasks and that correctness alone provides limited resolution.
- Existing evaluation frameworks typically cover only parts of reproducible harness-level evaluation and fine-grained trajectory collection.Harnesses shape prompts, tools, context management, and execution policies, so evaluating the underlying model alone does not fully capture deployed-agent capabilities.
- A2E unifies task composition, standardized trajectory monitoring, and lifecycle-aligned evaluation in one lightweight evaluation stack.Its Task Layer uses ATP, its Monitor captures structured execution traces, and its Evaluation Layer registers metrics by execution stage and dimension.
- Harness–model configurations vary substantially across GDPVal, MMLU-Pro, and τ3-bench, with no configuration consistently dominating all tasks.The best-performing harness changes across models and tasks, indicating that harness effectiveness is model- and task-dependent.
- Lifecycle-aligned metrics distinguish planning, tool use, final answers, and operational quality more finely than endpoint correctness alone.The metrics examine the trajectory from deliberation through action to the answer, including cost and safety.
2 Overview
A2E forms a closed evaluation workflow connecting benchmark preparation, agent execution, monitoring, multidimensional assessment, storage, and visualization. Its three layers separate these responsibilities while sharing a centralized server.
- A2E connects benchmark preparation with agent execution, trajectory collection, multidimensional evaluation, and result analysis.The system overview identifies Task, Monitor, and Evaluation as its three major components.
- Task Layer: The Task Layer organizes heterogeneous benchmarks and packages reproducible execution support.Benchmark management uses a hierarchical tree indexed by time, category, and difficulty; execution bundles include the sandbox, dataset, configuration, and runtime environment.
- Monitor Layer: The Monitor Layer provides a common access interface for heterogeneous agents and instruments their iterative reasoning–action–observation cycle.It captures model calls, state transitions, errors, tool calls, latency, token consumption, and generated artifacts as ordered event sequences.
- Evaluation Layer: The Evaluation Layer combines outcome evaluation with trajectory evaluation to assess both correctness and how agents reach their answers.Evaluators retrieve traces and outcomes from the server, then write scores and annotations back as structured records.
- The runtime pipeline uses loosely coupled Task, Server, Evaluation, and UI components, with the server coordinating experiment data.Monitoring writes records continuously to the centralized server, while the UI reads stored traces and results without launching experiments or modifying records.
3 Monitor Layer
The Monitor Layer converts heterogeneous agent execution into structured, interpretable span-based traces. These traces preserve operation hierarchy, timing, status, context, and relationships for analyzing both outcomes and execution processes.
- The Monitor records intermediate reasoning, model interactions, tool use, and skill invocation rather than only final responses.Instrumentation and span-based traces support analysis of both execution outcomes and the processes producing them.
- Semantic layer: Its semantic layer defines consistent categories such as agent, chain, model call, tool, and skill for interpreting agent behavior.The vocabulary identifies each activity’s inputs, outputs, and identity while separating semantic meaning from low-level runtime events.
- The runtime workflow writes monitored runs and traces to a centralized server, where evaluation retrieves them for trace-level and outcome-level assessment.Evaluation results are written back for storage and visualization.
- Span layer: The span layer records each behavior as a bounded operation with timing, context, status, and parent–child relationships.An agent run becomes a collection of related activities rather than a flat log sequence.
- SDK layer: The SDK layer maps framework-specific model calls, tools, workflows, and asynchronous operations onto stable semantic and span representations.Dedicated adapters contain framework differences while allowing the monitoring model to remain stable as support expands.
- Span-based traces expose delays, failures, divergent behavior, reasoning paths, model consultations, and external capabilities that final responses cannot reveal.They preserve both action sequence and causal organization within the wider execution process.
4 Task Layer
The Task Layer uses ATP to separate benchmark adaptation from harness execution and to connect heterogeneous tasks and agent systems through shared interfaces. It generates normalized task traces while supporting reproducible, sampled benchmark runs.
- Agent Task Protocol (ATP): ATP provides a shared interface between benchmark adaptation and agent harness execution.It standardizes task representation, the agent-facing interface, and execution records, avoiding separate logic for every benchmark–harness pair.
- Agent Task Protocol (ATP): A2E calls this internal software protocol the Agent Task Protocol (ATP).ATP is not a network protocol.
- Task-layer objects: Four task-layer objects connect benchmarks and harnesses: TaskInput, AgentBinding, AgentRunner, and TaskTrace.TaskInput stores task data and optional sandbox specifications; bindings provide tools and prompt construction; runners execute tasks and return traces.
- Agent harnesses: The registry supports harnesses for multiple agent frameworks, but registry support does not imply that every framework–benchmark pair has passed end-to-end validation.Each harness follows ATP through an AgentRunner that converts tools, configures the model client, runs the control loop, and returns a TaskTrace.
- Trajectory generation: Trajectory generation begins from a benchmark key, harness key, and model name, then samples tasks and runs them through the selected AgentRunner.The default sample is 40 tasks without replacement, with optional sample-size and random-seed controls.
- Trajectory generation: A TaskTrace stores run status, turn count, ordered tool calls, final answer, elapsed time, and raw framework output, linked when possible to a span tree.The trace preserves both normalized results and framework-level execution detail.
5 Evaluation Layer
The Evaluation Layer organizes agent assessment across execution stages and supports extensible, database-backed metric computation. This design combines diagnostic coverage with incremental, reproducible evaluation of stored trajectories.
- Motivation: Outcome-only evaluation provides limited diagnostic signal, extensibility, and scalability across agent trajectories.Failures may arise in reasoning, tool use, memory, answer generation, or system operation; metric catalogs and isolated trajectory files further constrain evaluation.
- Evaluation Taxonomy: The taxonomy covers reasoning, action, final answer, and runtime quality across four execution stages.Reasoning includes task, flow, and logical dimensions; action includes tool, skill, and memory; final answer includes correctness and completion; runtime quality includes efficiency and safety.
- Evaluation Taxonomy: Process-, outcome-, and lifecycle-level scopes identify both whether an agent fails and the execution stage and capability dimension associated with failure.Process-level evaluation covers reasoning and actions, outcome-level evaluation covers final results, and lifecycle-level evaluation covers operational properties across the trajectory.
- Extensibility: The taxonomy separates where a property is evaluated from how it is measured, allowing heterogeneous evaluators to share a consistent structure.Metric implementations may use an LLM judge, deterministic rule, environment verifier, or statistical aggregation function.
- Extensibility: New metrics can be added under lifecycle dimensions without modifying the benchmark runner, agent harness, or existing metrics.Examples include tool-selection metrics under Tool and latency or resource indicators under Efficiency; metric catalogs can evolve with capabilities, benchmarks, and safety requirements.
- Scalable Evaluation: Database-backed trajectory records connect tasks, runs, turns, tool calls, and metric results for indexed aggregation and incremental evaluation.Stored trajectories can be reevaluated with new metrics, metric versions, judge models, or aggregation policies, while external large artifacts may remain in file or object storage.
6 Experiments
A2E evaluates harnesses under matched configurations and exposes substantial differences in correctness, efficiency, and execution behavior across benchmarks. The results show that harness performance is task-dependent, while trajectory-level analysis adds detail beyond final correctness.
- Evaluation setup: All nine harnesses share the same backbone model, inference settings, tools, step limit, timeout budget, and aligned task IDs, isolating harness differences.Each benchmark contributes five fixed-seed tasks, so runs are matched by task.
- Harness–benchmark matrix: 0.00–0.60 correctness spreads appear on τ-bench and GDPVal, while traject-bench ranges from 0.20 to 1.00 under the same backbone.Single-turn tasks show identical scores across harnesses, whereas separation appears in multi-turn tasks.
- Trajectory analysis: Across eight instrumented harnesses, correctness spans 0.568–0.663 while mean token cost spans 3.5×, from 2,063 tokens for Claude-Agent-SDK to 7,319 for Smolagents.Agno has the highest correctness at 0.663 and is also the second cheapest; process metrics otherwise move little on this benchmark mix.
- Cross-benchmark comparison: Figure 7 compares task success against completion tokens and identifies benchmark-specific effectiveness–efficiency frontiers rather than a universal winner.The top-ranked harnesses differ across GDPVal, MMLU-Pro, and τ3-bench, and the comparison uses a normalized trade-off score.
- Harness behavior: With the same model, harness design choices in prompts, tools, context, execution loops, error handling, and termination policies produce system-level variation.A case study contrasts LangGraph’s correctness 1.0 using 10,122 tokens with CrewAI’s correctness 0.0 after 96,704 tokens and five tool calls.