Source-linked AI summary
An Empirical Study of Automating Agent Evaluation
Kang Zhou, Sangmin Woo, Haibo Ding, Kiran Ramnath, Subramanian Chidambaram, Aosong Feng, Vinayak Arannil, Muhyun Kim, Ishan Singh, Darren Wang, Zhichao Xu, Megha Gandhi, Nirmal Prabhu, Soumya Smruti Mishra, Vivek Singh, Gouri Pandeshwar, Lin Lee Cheong
TL;DR
Agent evaluation must assess complex execution traces, but existing approaches leave evaluation-generation criteria and implementation unaddressed. The paper introduces EvalAgent, which uses evaluation skills in a trace-based pipeline to generate executable assessments and reports, achieving 79.5% expert preference and 65.0% Eval@1.
Problem
Agent evaluation must assess execution traces containing reasoning steps, tool invocations, error recovery, and state transitions, while existing judge-based approaches assume pre-defined criteria.
Method
EvalAgent encodes evaluation expertise as reusable skills that compose into a trace-based pipeline for planning criteria, generating scenarios, collecting traces, producing executable assessment code, and reporting results.
Results
EvalAgent achieves 79.5% expert preference and 65.0% Eval@1, the rate at which evaluations execute successfully and produce substantive first-attempt results.
Takeaways & Limitations
Evaluation skills focus generated assessments and stabilize executability as metric counts grow, while planning is effective when paired with domain-specific evaluation knowledge.
Takeaways & Limitations
The benchmark covers 20 agents, experiments use only the Claude model family, and approximately one-third of generated evaluations require manual debugging.
Abstract
from arXiv · showhide
Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task. Without domain-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over-engineered evaluations averaging 12+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation. We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline. EvalAgent encodes evaluation domain expertise as evaluation skills (procedural instructions, reusable code and templates, and dynamically retrieved API documentation) that compose into a trace-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports. To systematically assess generated evaluations, we introduce a meta-evaluation framework alongside AgentEvalBench, a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%.
1. Introduction
Agent evaluation must assess execution traces and behavioral processes, not only final outputs, creating a costly evaluation-generation problem. EvalAgent addresses this gap with evaluation skills and achieves stronger execution and expert-preference results than baseline approaches.
- Agent evaluation must assess reasoning steps, tool calls, error recovery, and state transitions because final outputs can hide flawed reasoning or robust failure handling.
- Existing judge-based approaches assume predefined criteria, whereas this work targets automatic generation of criteria, executable assessment code, and actionable reports from agent code and traces.
- Frontier coding assistants generate 12+ metrics, code 2–3× longer than necessary, and plans that drift into shallow keyword heuristics without evaluation-specific guidance.
- EvalAgent encodes evaluation expertise as procedural instructions, reusable code and templates, and dynamically retrieved documentation within a trace-based evaluation pipeline.
- 65.0% Eval@1 measures evaluations that execute and produce substantive results on the first attempt, while expert annotators prefer EvalAgent in 79.5% of comparisons.
- Evaluation skills stabilize executability as metric count grows, reaching 65% Eval@1 at five metrics versus 30–40% for baselines.
2. EvalAgent
EvalAgent transforms agent code and requirements into executable evaluation artifacts and actionable reports through a six-stage, skill-guided pipeline. Its skills constrain scope, provide implementation scaffolding, and supply current API knowledge across planning, tracing, code generation, and reporting.
- EvalAgent transforms agent source code and user requirements into executable evaluation code and actionable reports through a six-stage pipeline.
- Evaluation skills package procedural instructions, executable code, and reference materials that are loaded on demand and reused across pipeline stages.
- Procedural skills constrain planning and code generation by requiring focused metrics, distinct behavioral coverage, implementation, orchestration, self-review, and dependency updates.
- Reusable templates and code patterns standardize plans, reports, trace parsing, and metric integration while reducing variance and scope expansion.
- Dynamic Context7 retrieval supplies current API signatures and usage patterns, addressing deprecated-API errors that cause many unconstrained-system executability failures.
- The pipeline plans criteria and scenarios, generates JSONL test cases, instruments agents with OpenTelemetry, collects traces, generates metric code, and reports failure causes with recommendations.
3. Meta-Evaluation and AgentEvalBench
The paper introduces AgentEvalBench and a meta-evaluation framework for assessing generated agent evaluations, then compares EvalAgent with four baselines across quality, executability, efficiency, and complexity. EvalAgent combines evaluation skills with trace-based assessment and consistently outperforms baselines, especially as evaluation complexity increases.
- Meta-evaluation validation: 79.5% of human expert comparisons preferred EvalAgent, with 10.5% ties, while the meta-evaluator matched the human-majority winner in 97.5% of cases.Human experts achieved Fleiss’ κ=0.923; meta-evaluator agreement was lower for the more subjective Plan Quality and Plan-Code Alignment dimensions.
- Main results: 84–100% overall win-tie rates across baselines show that EvalAgent produces higher-quality evaluations, with especially strong advantages over skill-free B4.Against B4, EvalAgent achieved 87.5–90.0% overall win-tie, including 92.5% for Code Quality & Complexity and 83.8–87.5% for Metric Relevance.
- Main results: 62.5–65.0% Eval@1 makes EvalAgent the strongest method, surpassing B1 at 15.0–17.5%, B2 at 45.0–60.0%, B3 at 17.5–35.0%, and B4 at 30.0–32.5%.Eval@1 requires first-run execution without errors and meaningful, non-vacuous evaluation results.
- Efficiency: EvalAgent uses 31% fewer tokens and 58% less time than B4 while achieving higher quality.For Sonnet, EvalAgent used 2,095K versus 3,024K tokens and 4.18 versus 10.00 minutes.
- Ablations and findings: Evaluation skills preserve executability as complexity grows: at five metrics, EvalAgent reaches 65% Eval@1 while B3 falls to 30% and B4 to 40%.Dynamic API documentation retrieval also raises Sonnet’s Eval@1 from 20.0% to 65.0%, a 45pp gain.
A.1. Phase 1: Evaluation Planning
The evaluation pipeline transforms agent traces into structured artifacts: test cases, trace-derived inputs, executable metrics, aggregated results, and actionable reports.
- Test Scenarios: Test cases are stored as JSON Lines with identifiers, scenario categories, queries, descriptions, and expected behaviors.Alternative generators may be used if they emit the expected schema.
- Trace Collection: OTEL-compatible traces are instrumented with spans containing timestamps, event types, and structured payloads.Trace processing filters agent-relevant spans and extracts operation names, inputs and outputs, prompts, completions, tool metadata, and timing.
- Evaluation Code: The system generates metric implementations, an evaluation orchestrator, and JSON result storage.Metrics may use deterministic checks or LLM-based assessments, while the orchestrator loads traces, applies metrics, and aggregates outcomes.
- Reporting: Evaluation reports include an executive summary, results analysis, failure analysis with root causes, and prioritized recommendations.The report structure connects measured performance to evidence-based improvement actions.
B. AgentEvalBench Dataset Details
AgentEvalBench contains 20 agents spanning varied domains, frameworks, architectures, tools, and memory configurations to support diverse evaluation scenarios.
- Simple Complexity Agents: Simple-complexity agents cover game playing, travel assistance, web browsing, career planning, document QA, research assistance, news aggregation, and software engineering.The agents use frameworks including Agno, MCP/Bedrock, LangChain/LangGraph, Embedchain, and custom implementations.
- Medium Complexity Agents: The benchmark includes agents for question answering, trip planning, information retrieval, conversational memory, code generation, database optimization, financial analysis, and medical NLP.These examples span LangGraph, CrewAI, MCP, Strands, LlamaIndex, and other frameworks.
- Dataset Coverage: The dataset spans diverse agent implementations, including multi-agent workflows, tool-based systems, retrieval augmentation, and persistent memory.This diversity is represented across the listed domains, frameworks, tools, and architectures.
- Additional Agents: Additional benchmark agents address data labeling and recommendation through specialized skills, tools, simulated interactions, and memory systems.Adala supports multiple labeling skills, while Agent4Rec models preferences with time- and importance-weighted memory.
C. Meta-Evaluation Dimension Rubrics
The meta-evaluation framework compares evaluation artifacts across five weighted dimensions, emphasizing requirement alignment, focused metrics, implementation quality, coherent plans, and plan-code consistency.
- Framework: Meta-evaluators compare approaches using A Wins, B Wins, or Tie judgments, then calculate an overall winner through weighted aggregation.A dimension winner receives its full weight, while ties award both approaches half the weight.
- User Requirement Fulfillment: User Requirement Fulfillment measures how effectively an evaluation addresses explicit user requirements, with focused implementation preferred over exhaustive additions.The benchmark includes generic requirements and agent-specific requirements such as medical entity correctness or code quality.
- Metric Relevance: Metric Relevance carries the highest weight at 30% and evaluates metrics by their signal-to-noise ratio rather than by metric count.Meaningful metrics should assess the evaluation goal without being trivial, redundant, or distracting.
- Code Quality & Complexity: Code Quality & Complexity weighs 25% and prioritizes metric logic correctness, organization, clean implementation, and readability.The rubric evaluates both correctness and maintainability of the generated codebase.
- Plan Quality: Plan Quality weighs 15% and assesses completeness, definition completeness, conciseness, and ease of understanding.Plans exceeding 1000 lines require justification, while those exceeding 1500 lines likely lose to shorter equivalent plans.
- Plan-Code Alignment: Plan-Code Alignment weighs 15% and checks metric alignment, implementation mismatches, missing implementations, and unplanned features.The dimension tests whether the implementation faithfully follows the stated plan.
D. Prompt Specifications
EvalAgent prompts structure evaluation planning, code generation, reusable skills, and comparative meta-evaluation, with explicit guidance favoring focused, validated, maintainable artifacts.
- D.1. Baseline Approaches: Baseline approaches vary in context access and staging, from single-turn text completion to source-code-only, one-stage, and two-stage tool-using generation.The two-stage baseline plans before coding but lacks structured templates and reference implementations.
- D.2. EvalAgent Prompts: EvalAgent’s planning prompt analyzes architecture and traces, identifies 2–4 key metrics, designs critical test scenarios, and produces a structured plan.The planning workflow explicitly focuses on actionable content and distinct behavioral aspects.
- D.2. EvalAgent Prompts: The code-generation prompt implements the plan, builds trace extraction and metric classes, creates an evaluation entry point, reviews code, and updates dependencies.It targets a minimal working version, validates APIs, avoids over-engineering, and follows the plan exactly.
- D.3. Evaluation Skills Examples: Evaluation skills combine procedural instructions, reusable code and templates, and dynamic resources to standardize the workflow and improve Eval@1.Templates scaffold plans and reports, while code patterns support OTEL parsing and metric integration.
- D.3. Evaluation Skills Examples: API validation with Context7 is mandatory before implementing publicly available evaluation-library functionality.The requirement targets current constructors, metric patterns, and available built-in metrics to avoid deprecated usage.
- D.4. Meta-Evaluator Comparison Prompt: The comparative meta-evaluator judges five weighted dimensions, applies an anti-length bias, and reports dimension winners, weighted scores, and differentiating factors.Its workflow reads plans and code, determines A/B/Tie outcomes, and favors fewer focused metrics and maintainable implementations.
- E. Study Design: Human annotation compares paired plans and code across five dimensions after training, calibration, blinded assignment, and randomized A/B presentation.Annotators independently review the agent, artifacts, rubrics, and dimension winners before calculating the weighted overall result.
E.3. Inter-Annotator Agreement
Inter-annotator agreement was nearly perfect across dimensions, with Plan-Code Alignment showing the greatest subjectivity. Consensus used majority voting, while overall winner disagreement occurred in 20% of cases.
- κ > 0.80 across dimensions except Plan-Code Alignment, which reached κ = 0.700.The lower agreement reflects greater subjectivity in assessing implementation faithfulness.
- Relevance achieved perfect agreement at κ = 1.000, while Plan-Code Alignment had the lowest agreement at κ = 0.700.
- 20% of cases showed overall winner disagreement, with majority voting used to establish consensus.Disagreement occurred in 8 of 40 cases.
F. Detailed Efficiency Analysis
Efficiency varies with requirement specificity and model choice, but EvalAgent remains stable and efficient across conditions. Its code generation stage is substantially more efficient than B4, consistent with reduced implementation exploration from evaluation skills.
- EvalAgent maintains stable efficiency under both generic and specific requirements, while specific requirements reduce exploration overhead for trace-based methods B3 and B4.
- Haiku executes faster but consumes more tokens, whereas Sonnet is slower but more token-efficient; EvalAgent retains efficiency advantages with both models.
- 30–49% more efficient code generation than B4 was achieved by EvalAgent, using 2235K versus 3219K tokens with Haiku and 1073K versus 2104K with Sonnet.The gains are attributed to evaluation skills reducing exploration during implementation.
G. Meta-Evaluation Results with Sonnet 4.5 as Meta-Evaluator
Using Sonnet 4.5 as meta-evaluator produces the same broad baseline ranking as Opus 4.5, with EvalAgent achieving strong win-tie rates across baselines. Cross-model agreement further supports robustness to the underlying evaluator model.
- 76–95% overall win-tie rates under Sonnet-as-judge are slightly lower than Opus-as-judge’s 84–97%, while preserving baseline rankings.The largest advantages occur in Metric Relevance and Code Quality.
- EvalAgent achieves win-tie rates above 80% against every baseline under both meta-evaluators.
- 86.6% cross-model agreement indicates that Opus 4.5 and Sonnet 4.5 concurred on most individual judgments.Inter-model gaps ranged from 0.6 to 8.8 percentage points.
- Cross-model agreement complements human validation by showing that meta-evaluation outcomes are robust to the choice of underlying LLM.
H.2. Run-to-Run Consistency
The meta-evaluator shows substantial run-to-run stability across independent evaluations. Agreement is strongest for concrete code properties and lower, though still strong, for more interpretive dimensions.
- 76.3% three-way agreement and 84.2% average pairwise agreement were achieved across 40 configurations and three independent runs.
- 84.6% three-way and 89.7% pairwise agreement were achieved for Code Quality & Complexity, the most consistent dimension.Code-level properties provide more concrete, less ambiguous evidence for comparison.
- Metric Relevance and User Requirement Fulfillment showed moderately lower but still strong agreement than Code Quality & Complexity.These dimensions allow greater interpretive latitude.
- The 76.3% three-way agreement substantially exceeds the 33.3% random-choice baseline, while 84.2% pairwise agreement indicates disagreements usually involve one divergent run.
I.1. Directory Structure and Code Comparison
The comparison contrasts EvalAgent’s compact, trace-grounded evaluation artifacts with baselines that rely on synthetic data, keyword heuristics, or more elaborate implementations. EvalAgent’s approach supports richer runtime validation and documented LLM-as-judge integrations.
- Directory structure: EvalAgent produces a two-file, 207-LOC project, whereas B4 generates 22 files across five packages with unreachable dead-code modules.The figure describes this as an 11× reduction in project size.
- Trace-based versus source-code evaluation: Source-code evaluation relies on synthetic test cases inferred from static analysis and can miss runtime edge-case behavior.The example assumes factorial computation from function signatures without observing actual executions.
- Trace-based versus source-code evaluation: Trace extraction captures genuine user inputs, tool calls, outputs, and extracted entities, enabling precision/recall and code-correctness checks against ground truth.The medical-document example validates extracted diagnoses and medications rather than merely counting terms.
- Trace-based versus source-code evaluation: Without traces, keyword heuristics count hard-coded terms and cannot distinguish semantically incorrect statements from correct ones.The source-code approach analyzes implementation structure but cannot observe tool calls, error handling, or execution paths.
- Dynamic documentation retrieval: Context7 supplies current API documentation, preventing integration failures such as omitting the required bedrock/ model-provider prefix.The documented DeepEval wrapper specifies required methods and the correct model format.
- Dynamic documentation retrieval: With Context7, LLM-as-judge evaluation generalizes across locations, captures semantic similarity, and returns reasoning, unlike brittle hard-coded matching.The Hilton example contrasts documented LiteLLM completion usage with heuristics that fail on unlisted or semantically equivalent locations.