PaperlayerView exact revision on arXiv

Source-linked AI summary

Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang, Arvind Narayanan

arXiv:2510.11977v1cs.AIcs.CL

TL;DR

AI agent evaluation lacks standardized, multidimensional evidence about capability, cost, and behavior. HAL addresses this gap with a distributed harness, large-scale comparisons, and automated log inspection, revealing that higher reasoning effort does not always improve accuracy and that some results are invalidated by data leakage. The project argues for evaluations that assess reliability and deployment viability alongside benchmark scores.

  • Problem

    Agent evaluations are difficult to compare because infrastructure is non-standardized, scaffold effects and costs are rarely reported, and benchmark scores may omit agent behavior.

  • Method

    HAL combines a distributed standardized evaluation harness with three-dimensional analysis of models, benchmarks, and scaffolds, plus LLM-aided inspection of agent logs.

  • Results

    21,730 rollouts across 9 models and 9 benchmarks show that higher reasoning effort does not always improve accuracy, while log analysis reveals shortcuts and reliability failures.

  • Takeaways & Limitations

    Reliable agent evaluation requires standardized infrastructure, multidimensional accuracy-cost analysis, and systematic log analysis to distinguish genuine capability from benchmark gaming and assess deployment viability.

  • Takeaways & Limitations

    Latency remains an unresolved evaluation dimension because massively parallel runs may introduce additional variance into task-solving latency measurements.

Abstract

from arXiv · show

AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of how well agents really work. We introduce the Holistic Agent Leaderboard (HAL) to address these challenges. We make three main contributions. First, we provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of VMs, reducing evaluation time from weeks to hours while eliminating common implementation bugs. Second, we conduct three-dimensional analysis spanning models, scaffolds, and benchmarks. We validate the harness by conducting 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000. Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs. Third, we use LLM-aided log inspection to uncover previously unreported behaviors, such as searching for the benchmark on HuggingFace instead of solving a task, or misusing credit cards in flight booking tasks. We share all agent logs, comprising 2.5B tokens of language model calls, to incentivize further research into agent behavior. By standardizing how the field evaluates agents and addressing common pitfalls in agent evaluation, we hope to shift the focus from agents that ace benchmarks to agents that work reliably in the real world.

1 INTRODUCTION

HAL addresses fragmented agent evaluation by combining standardized infrastructure, multidimensional benchmarking, and automated log analysis. Its evaluations span models, scaffolds, and benchmarks while tracking performance, cost, and agent behavior.

  • Motivation: Agent evaluations are non-standardized, slow, and error-prone, while scaffold choices affect accuracy and cost but are rarely compared.Serial evaluation of a single benchmark can take weeks, leaving leaderboards outdated.
  • Contributions: HAL provides an open, unified harness that standardizes agent evaluation and orchestrates runs across hundreds of virtual machines.The harness reduces evaluation time from weeks to hours while instrumenting costs, API calls, and execution traces.
  • Contributions: 21,730 rollouts across 9 models and 9 benchmarks support comparisons across agent scaffolds, models, and benchmark domains.The benchmarks cover web navigation, coding, scientific research, and customer service, with evaluation costing about $40,000.
  • Contributions: HAL reports Pareto frontiers for accuracy versus dollar and token costs, enabling agent selection under deployment constraints.The leaderboard spans nine challenging benchmarks rather than focusing on a single domain or accuracy alone.
  • Contributions: Automated analysis of more than 2.5 billion language-model-call tokens identifies agent shortcuts and reliability failures that benchmark scores can miss.HAL is designed as an ongoing community resource with additional benchmarks, models, scaffolds, and log analyses planned.

2 THE HAL HARNESS

The HAL harness separates agent scaffolds from benchmark execution and supports reproducible, cost-controlled evaluations across heterogeneous environments. It accepts agents through a minimal Python API and integrates logging, model compatibility, and distributed execution.

  • Harness design: HAL accepts any agent exposing a minimal Python API and orchestrates evaluation across web-navigation tasks, code repositories, and other diverse benchmarks.This decouples scaffold implementation from benchmark execution.
  • Harness design: The harness integrates Weave for comprehensive logging, LiteLLM for cross-model compatibility, and local, Docker, and Azure VM execution environments.These integrations support reproducible evaluation across heterogeneous agent settings.

3 EXPERIMENTAL SETUP

HAL evaluates models, benchmarks, and agent scaffolds across four domains using configurations chosen to expose performance, cost, and scaffold interactions. The setup also includes automated trajectory analysis and excludes a data-leaking scaffold.

  • Evaluation dimensions: The evaluation spans 9 benchmarks across coding, web navigation, scientific research, and customer service, with 9 models and multiple agent scaffolds.The selected benchmarks target interface navigation, code correctness, scientific analysis, and multi-turn interaction.
  • Caveats: The TAU-bench Few Shot scaffold was excluded because automated log analysis revealed data leakage that invalidated its results.Anthropic reasoning uses LiteLLM’s default high-reasoning setting of 4,096 tokens, while OpenAI reasoning budgets are not publicly disclosed.
  • Evaluation dimensions: HAL conducts 21,730 rollouts to compare models on shared benchmarks, model transfer across benchmarks, and scaffold effects on accuracy and cost.The design prioritizes configurations that reveal meaningful cross-axis comparisons.
  • Models: The model set combines frontier and cost-efficient systems, including models with different reasoning-effort settings to study inference-time cost-accuracy tradeoffs.Model prices range from Claude Opus 4.1 at $15 per million input tokens and $75 per million output tokens to Gemini 2.0 Flash at $0.1 and $0.4.
  • Models: On ScienceAgentBench, o4-mini scores 27% while being about 5x cheaper than GPT-5, which scores 30%.HAL automatically computes model-selection Pareto frontiers for benchmark-specific accuracy and cost requirements.
  • Agent scaffolds: Task-specific scaffolds are used for most benchmarks, while a generalist scaffold tests whether benchmark-specific optimization justifies added engineering complexity.Online Mind2Web also includes BrowserUse and SeeAct to examine scaffold effects on model performance.
  • Log analysis: Docent analyzes agent transcripts using rubrics for instruction violations, tool failures, self-correction, verification, environmental barriers, and shortcuts or gaming.The analysis targets reliability failures and behaviors that accuracy and cost metrics do not capture.

4 RESULTS

HAL combines multidimensional benchmark analysis with automated agent-log inspection to reveal performance, cost, reasoning, scaffold, and reliability patterns that accuracy-only evaluation misses.

  • Multidimensional benchmark results: Only 1 of 9 benchmarks places the most costly model on the accuracy-cost Pareto frontier.The frontier is sparse and most frequently includes Gemini 2.0 Flash, GPT-5, and o4-mini Low.
  • Multidimensional benchmark results: On 6 of 9 benchmarks, higher token usage correlates positively with accuracy, indicating limited token-efficiency gains on long-horizon tasks.Token-based and dollar-cost Pareto curves can produce different performance pictures because model prices change substantially.
  • Multidimensional benchmark results: Higher reasoning effort produces equal or lower accuracy in 21 of 36 model-agent-benchmark combinations.The comparison covers Claude Opus 4.1, Claude Sonnet 4, Claude-3.7 Sonnet, and o4-mini.
  • Multidimensional benchmark results: Scaffold choice substantially changes both cost and accuracy, with model-scaffold interactions making model-scaffold matching consequential.On Online Mind2Web, SeeAct with GPT-5 Medium costs $171, while Browser-Use with Claude Sonnet 4 costs $1,577 despite a two-percentage-point accuracy difference.
  • Multidimensional benchmark results: Task-specific scaffolds consistently outperform generalist scaffolds, while generalist scaffolds usually cost less.Task-specific agents outperform on 9 of 12 CORE-Bench Hard runs and 11 of 12 SWE-bench Verified Mini runs; generalist scaffolds cost less in 20 of 24 comparisons.
  • Automated analysis of agent logs: Automated log analysis exposes shortcuts, reliability failures, and scaffold or benchmark bugs that accuracy metrics do not capture.Docent analysis identified behaviors including finding benchmark answers online, hard-coding plausible solutions, and catastrophic tool-use failures; it also uncovered a TAU-Bench scaffold bug.

5 CONCLUSION

HAL addresses infrastructure gaps in agent evaluation through standardized harnesses, multidimensional analysis, and systematic log inspection. The authors identify standardization, log analysis, and full-spectrum performance measurement as priorities, while noting unresolved latency challenges in massively parallel evaluation.

  • HAL targets infrastructure gaps that otherwise hinder distinguishing genuine capability from benchmark gaming and assessing deployment economics.
  • Massively parallel evaluations can produce latency variance from API rate-limit errors, limiting the representativeness of collected task-solving latency.
  • Systematic log analysis can reveal substantially different agent behaviors among agents with identical accuracy, including shortcuts and costly incorrect payment actions.
  • Standardized infrastructure is needed for fair comparisons across benchmarks, models, and scaffolds because model providers and libraries can change incompatibly over time.
  • Evaluation should capture token usage, failure modes, and scaffold interactions alongside final performance scores.

APPENDIX

The authors used language models for text editing, proofreading, and coding assistance, while retaining responsibility for the paper’s text and results.

  • LLMs assisted with text editing, proofreading, and coding, while the authors take full responsibility for the paper’s text and results.

A1 EXTENDED RELATED WORK

HAL builds on prior efforts to standardize evaluation environments, interactive agent suites, analytical leaderboards, and multidimensional reporting. Its related ecosystem spans general agent interfaces, domain-specific frameworks, trace inspection, and empirical rigor for reinforcement learning.

  • Table A1 compares related systems by cross-domain support, three-dimensional evaluation, log analysis, cost comparison, and parallel orchestration.
  • HAL is presented as part of a broader tradition of standardizing evaluation environments and infrastructure for agents and language models.
  • ALE, OpenAI Gym, and DeepMind Control Suite established shared interfaces or standardized task structures for reproducible reinforcement-learning comparisons.
  • AgentLab-related passages describe parallel task launching and trace visualization, while noting that its log analysis is not systematic and cost tracking is not a direct platform feature.
  • Prior reinforcement-learning work emphasizes separating training and evaluation, uncertainty-aware reporting, transparent empirical design, and controls for variance and sensitivity.
  • HELM and LM Evaluation Harness standardized language-model evaluation, while AgentBench, AgentBoard, AgentGym, BrowserGym, and Galileo broadened interactive agent benchmarking.
  • AISI Inspect provides community-driven evaluations with secure sandboxing, parallel execution, trajectory visualization, and tools for auditing tool-using agents.

A2 ADDITIONAL DETAILS ON THE HAL HARNESS ARCHITECTURE

The HAL harness separates agents from benchmarks and coordinates reproducible, cost-aware evaluations across heterogeneous execution environments. It standardizes logging, model access, concurrency, artifact handling, and result aggregation through lightweight interfaces.

  • HAL accepts benchmark tasks and agent implementations, provisions isolated environments, logs interactions, evaluates outputs, and aggregates results for a public leaderboard.
  • Weave standardizes telemetry for model and tool calls, token usage, costs, parameters, and execution trajectories across providers and frameworks.
  • HAL runs agents locally, in Docker containers, or on Azure virtual machines, with execution modes serving different isolation and reproducibility needs.
  • Concurrent isolated task execution includes automatic provisioning, timeouts, artifact and log collection, and environment cleanup.
  • Agents integrate through a minimal Python run(input, **kwargs) →dict interface, supporting both task-specific and generalist scaffolds.
  • The harness separates benchmark-defined inputs and scoring from agent implementations, recording outputs, token usage, and aggregated evaluation metrics.
  • LiteLLM enables the same agent code to run across multiple model providers despite differing interfaces, capabilities, and costs.
  • Together, isolated execution, standardized logging, and decoupled agent integration provide reproducible and cost-aware evaluation infrastructure.

A3 PRACTICAL HURDLES IN LARGE-SCALE AGENT EVALUATION

Large-scale agent evaluation is hindered by costly, unstable, and non-standardized infrastructure. Provider changes, hidden execution variance, entangled task specifications, and logging dependencies make reproducible comparisons difficult.

  • Evaluation scale: High evaluation costs often force single runs without statistical validation, preventing reliable uncertainty estimation.Some benchmarks cost thousands of dollars per model, making confidence intervals prohibitively expensive.
  • Provider instability: Provider changes can invalidate comparisons through silent model swaps, breaking API changes, and incompatible feature interfaces.Stable endpoint names may conceal changed model weights, while removed API parameters can require scaffold updates before evaluation.
  • Provider instability: Aggregator routing can vary quantization levels across calls, introducing hidden variance into benchmark results.OpenRouter may route requests to different providers serving FP4 or FP8 versions of a model.
  • Operational constraints: Rate-limit failures can be mislabeled as capability errors, while provider spend and rate limits constrain parallel evaluation scale.Silent infrastructure failures are marked incorrect, and evaluators may need sequential runs or special provider access.
  • Cross-provider comparability: Reasoning-effort settings are not comparable across providers, making equivalent computational budgets impossible to establish.LiteLLM maps “high” to 4,096 reasoning tokens, whereas OpenAI does not disclose the meaning of its settings.
  • Reproducibility: Evaluation reproducibility is undermined by entangled task instructions, brittle infrastructure abstractions, and evolving dependencies.Models may follow benchmark instructions too literally, while library updates and hardcoded provider assumptions alter agent behavior.

A4 LIMITATIONS AND FUTURE WORK

HAL’s current results have documented limitations, which the authors state transparently alongside plans to address them.

  • A4 LIMITATIONS AND FUTURE WORK: HAL’s current results are affected by several limitations that the authors document transparently and plan to address.The paper frames these limitations as constraints on the present results rather than as resolved issues.

A4.1 LIMITATIONS WE PLAN TO ADDRESS IN FUTURE VERSIONS OF HAL

Future HAL versions will improve cost accounting, benchmark coverage, evaluation completeness, and API configuration. These changes target known gaps in the current evaluation matrix and task coverage.

  • Cost accounting: HAL does not yet account for cache hits in SWE-Agent cost calculations, reporting full token prices instead.Cache-aware cost tracking is planned for a future update.
  • Benchmark access: GAIA and AssistantBench use public test sets rather than the private sets used by official leaderboards.The authors note that private and public sets may have different task distributions and plan secure submission-portal access.
  • Benchmark coverage: SWE-Bench Verified Mini contains 50 tasks instead of the full 500-task SWE-Bench Verified dataset, limiting the software-engineering view.HAL also uses the original TAU-bench and plans to expand to complete benchmark versions.
  • Evaluation matrix: The reported analysis covers 142 model-scaffold-benchmark combinations out of 186 runs, leaving parts of the evaluation matrix incomplete.Some configurations were not completed across all comparisons, including the generalist agent on all GAIA models.
  • API configuration: HAL agents use OpenAI’s completions API rather than the newer responses API, which may provide better structured-output handling for some tasks.The authors plan to update agent architectures to recommended configurations across providers.

A4.2 FUNDAMENTAL CONSTRAINTS

Some HAL limitations arise from external provider behavior and evaluation costs that the project cannot fully control. The authors also identify uncertainty in latency and failure-mode interpretation.

  • Provider-specific constraints: OpenAI’s proprietary reasoning settings lack public specifications, preventing HAL from standardizing their computational effort across evaluations.HAL documents the settings used for each evaluation instead.
  • Measurement constraints: Parallel evaluation introduces latency variance from VM provisioning, networks, and server load, while serial validation would extend runs from hours to weeks.The authors plan small-sample serial runs to estimate latency.
  • Failure-mode analysis: Automated failure analysis cannot establish whether correcting an observed failure would complete the task or expose a later error.Causal attribution would require checkpointing and replaying agent-environment states, exceeding the computational budget.
  • Future direction: HAL remains a step toward standardized, reproducible evaluation despite these constraints, with broader insights expected as limitations are addressed.The authors invite community contributions to expand coverage and remediate remaining concerns.

A5 DATA LEAKAGE IN TAU-BENCH FEW SHOT AGENT

Automated log analysis uncovered data leakage in the TAU-bench Few Shot agent, invalidating its evaluation results and prompting exclusion of the scaffold. The incident also demonstrates that auditing benchmark implementations and analyzing agent behavior are necessary to protect evaluation integrity.

  • Leakage discovery: Automated log analysis discovered that the TAU-bench Few Shot agent loaded actual test-set examples as in-context demonstrations.The demonstrations came from a file used by the official few-shot agent and were not limited to training examples.
  • Evaluation impact: The leakage made accuracy measurements meaningless because models were shown examples from the benchmark test set during evaluation.All results from the compromised scaffold were excluded from the analysis.
  • Evaluation impact: Without automated log analysis, the leakage could have remained undetected and produced artificially inflated performance numbers.The compromised evaluations had already consumed about $1,000 before the issue was identified.
  • Lessons for evaluation: The incident shows that benchmark implementations require careful auditing, including code from official benchmark repositories.The paper identifies fundamental implementation errors as capable of compromising evaluation validity.
  • Lessons for evaluation: Systematic agent-behavior analysis is essential because final accuracy scores alone would not have revealed the leakage.The authors argue that comprehensive logging and automated analysis should become standard practice in agent evaluation.
  • Lessons for evaluation: Few-shot evaluations require strict separation between demonstration and test data because even partial overlap invalidates results.This requirement follows directly from the TAU-bench leakage incident.
Loading 2510.11977v1…