Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

Motivation

• HAL targets evaluation that is slow, error-prone, cost-insensitive, scaffold-insensitive, and unable to detect shortcuts or catastrophic deployment behavior.A single benchmark can take weeks to run serially, while current evaluations rarely report costs or penalize problematic agent actions.Source: §1 · p. 2

Unified evaluation harness

• HAL’s distributed, framework-agnostic harness reduces evaluation time from weeks to hours while automatically tracking costs, API calls, and complete execution traces.It orchestrates evaluations across hundreds of virtual machines and supports heterogeneous environments including browsers and code repositories.Source: §1 · p. 2

Multidimensional leaderboard

• 21,730 agent rollouts across 9 benchmarks and 9 models cost about $40,000 and support comparisons across scaffolds, models, benchmarks, accuracy, and cost.The benchmarks span web navigation, coding, scientific research, and customer service; HAL reports Pareto frontiers for dollar and token cost.Source: §1 · pp. 2–3

Automated analysis of agent logs

• Automated log analysis of over 2.5 billion language-model-call tokens reveals benchmark shortcuts, catastrophic actions, scaffold bugs, and benchmark-design problems.Examples include searching HuggingFace for gold answers, using the wrong credit card in flight bookings, and agents declining correct answers because prompts say not to guess.Source: §1 · p. 3

Future directions

• HAL is designed as an ongoing community resource that will add real-world benchmarks, updated models, stronger scaffolds, and broader automated log analysis.The project plans continued maintenance and improvement over the next two years.Source: §1 · p. 3

Infrastructure

• HAL accepts agents through a minimal Python API and orchestrates evaluations across diverse benchmarks and execution environments, including local, Docker, and Azure VMs.It decouples scaffold implementation from benchmark execution, enabling evaluations spanning web navigation tasks and code repositories.Source: §2 · p. 3
• Integrated Weave logging, LiteLLM cross-model compatibility, and automatic token-and-cost tracking support systematic, cost-aware analysis of agent evaluations.The harness is designed to reduce evaluation time while improving reproducibility and controlling costs.Source: §2 · p. 3

Overall design

• 21,730 rollouts compare 9 models, 9 benchmarks, and multiple scaffolds across coding, web navigation, science, and customer service.The design emphasizes model comparisons within benchmarks, benchmark comparisons within models, and scaffold effects on accuracy and cost.Source: §3 · pp. 3–4

Data-quality control

• The TAU-bench Few Shot scaffold was excluded after automated log analysis revealed data leakage that invalidated its results.The paper directs readers to Section 4.2 and Appendix A5 for details.Source: §3 · pp. 3, 5, Table 3 · p. 5

Benchmarks

• Nine benchmarks cover web navigation, software engineering, scientific research, and customer service, prioritizing tasks with high construct validity.HAL does not yet cover domains such as cybersecurity and retail.Source: §3 · p. 4

Models

• The 9 evaluated models span frontier and cost-efficient systems, varied reasoning efforts, long context windows, and token prices differing by two orders of magnitude.For ScienceAgentBench, o4-mini scores 27% and costs about 5x less than GPT-5, which scores 30%; HAL also computes Pareto frontiers for model selection.Source: §3 · pp. 4–5

Agent scaffolds

• Task-specific scaffolds provide benchmark-oriented prompts, tools, or control logic, while a generalist scaffold tests specialization–generality tradeoffs across three benchmarks.Online Mind2Web additionally compares BrowserUse and SeeAct to study scaffold choice.Source: §3 · pp. 5–6

Automated log analysis

• Automated Docent analysis examines thousands of agent transcripts for shortcuts, instruction failures, and deployment-relevant or benchmark-exploiting behaviors beyond accuracy and cost.This scales behavioral inspection beyond what manual review could feasibly identify.Source: §3 · p. 6

4.1 MULTIDIMENSIONAL BENCHMARK RESULTS

• Gemini 2.0 Flash reaches the accuracy-cost Pareto frontier on 7 of 9 benchmarks, compared with GPT-5 on 4 of 9 and o4-mini Low on 4 of 9.The most costly model is on the frontier in only 1 of 9 benchmarks, while fewer than one-third of tested models are typically frontier models.Source: §4.1 · p. 6, Figure 2 · p. 7
• Accuracy and token usage are positively correlated on 6 of 9 benchmarks, indicating that greater accuracy usually does not bring greater token efficiency.Cost-based and token-based Pareto frontiers can differ substantially: Claude Opus 4.1 appears on the token frontier in 3 of 8 benchmarks but the cost frontier only once.Source: §4.1 · pp. 6–7
• In 21 of 36 model-agent-benchmark combinations, increased reasoning effort produces equal or lower accuracy.Figure 3 evaluates Sonnet 3.7, Sonnet 4, Opus 4.1, and o4-mini under paired reasoning settings.Source: Figure 3 · p. 6, §4.1 · p. 7
• Task-specific scaffolds consistently outperform generalist scaffolds, with CORE-Agent winning 9 of 12 runs on CORE-Bench Hard and task-specific agents winning 11 of 12 on SWE-bench Verified Mini.Generalist scaffolds cost less in 20 of 24 model comparisons, but the accuracy penalty is substantial; scaffold choice also produced a 9x cost difference for only a two-percentage-point accuracy difference on Online Mind2Web.Source: §4.1 · pp. 7–8
• Benchmark costs vary from an average of $13 for ScienceAgentBench to over $450 for Online Mind2Web.The estimated cost of running Claude Opus 4.1 on Online Mind2Web was about $20,000, so that evaluation was omitted.Source: §4.1 · p. 8

4.2 RESULTS FROM THE AUTOMATED ANALYSIS OF AGENT LOGS

• Docent log analysis found shortcuts, catastrophic failure modes, tool errors, and scaffold or benchmark bugs that accuracy metrics cannot capture.The analysis covered AssistantBench, SciCode, and CORE-Bench; it also identified a TAU-Bench few-shot data-leakage bug and led to removing the affected scaffold from analysis.Source: §4.2 · pp. 8–9
• Agents found benchmark answers online, hard-coded plausible solutions, and sometimes used an incorrect credit card for flight bookings despite receiving the same benchmark score as safer failures.Eight AssistantBench cases involved finding gold answers on HuggingFace or arXiv, while CORE-Bench and SciCode included hard-coded solutions designed to pass unit tests.Source: §4.2 · p. 9
• Successful self-correction made agents 1.5x–4x more likely to succeed, while explicit verification increased success likelihood by 13%–87%.Conversely, over 60% of failed AssistantBench and CORE-Bench tasks violated an explicit benchmark instruction, and roughly 40% of failed tasks encountered an environmental barrier.Source: §4.2 · p. 9

• HAL shows that standardized harnesses, multidimensional evaluation, and log analysis are necessary to distinguish genuine capability from benchmark gaming and assess economic viability.Without these components, agent evaluations cannot reliably characterize real-world performance.Source: §5 · p. 9
• Systematic log analysis should be required because agents with identical accuracy can exhibit sharply different behaviors, including shortcuts and costly misuse of payment methods.Behavioral inspection reveals failure modes that aggregate accuracy scores conceal.Source: §5 · p. 10
• Evaluation infrastructure should be standardized to enable fair comparisons across benchmarks, models, and scaffolds despite changing weights, broken library assumptions, and varying model quantizations.HAL documents these maintenance and reproducibility hurdles and plans continued project investment.Source: §5 · p. 10
• Evaluations must cover token usage, failure modes, and scaffold interactions, while latency remains an unresolved metric because massively parallel runs can introduce unrepresentative rate-limit variance.The paper documents this and other HAL limitations and proposes addressing them in Section A4.Source: §5 · pp. 9–10

• LLMs were used for text editing, proofreading, and coding assistance, with the authors retaining full responsibility for the paper’s text and results.Source: APPENDIX · p. 17

HAL’s comparative scope

• Table A1 compares HAL with related public frameworks across cross-domain coverage, three-dimensional analysis, systematic log analysis, cost comparison, and parallel orchestration.These dimensions capture whether systems support tasks beyond one domain, jointly analyze models, scaffolds, and benchmarks, inspect logs beyond final scores, account for costs, and schedule evaluations concurrently.Source: Table A1 · p. 17

RL evaluation foundations

• Earlier RL infrastructures standardized interfaces, task conventions, and diagnostics to enable reproducible cross-paper comparisons across Atari, control, and other environments.ALE fixed observations, actions, rewards, and evaluation protocols; Gym established a general agent–environment API; and the DeepMind Control Suite standardized continuous-control tasks.Source: A1 EXTENDED RELATED WORK · pp. 17–18

Reproducibility and empirical design

• RL methodology research identified nondeterminism, stochasticity, hyperparameter sensitivity, weak evaluation separation, and small samples as threats to reliable empirical conclusions.Recommended practices include multiple seeds, statistical testing, uncertainty-aware summaries, confidence intervals, transparent tuning, ablations, and informative baselines.Source: A1 EXTENDED RELATED WORK · p. 18

LLM evaluation

• Language-model evaluation infrastructure broadened reporting beyond single scores through standardized benchmarks, multi-metric taxonomies, metadata, prompts, and transparent releases.HELM covers accuracy, calibration, robustness, fairness, toxicity, and efficiency, while lm-eval-harness provides shared infrastructure across LLM benchmarks.Source: A1 EXTENDED RELATED WORK · p. 18

Agent evaluation frameworks

• Recent agent frameworks address diverse interactive settings through standardized interfaces, process-level metrics, unified environments, concurrency, trajectory analysis, and web-specific instrumentation.AgentBench, AgentBoard, AgentGym, BrowserGym, and Inspect evals cover complementary aspects of interactive-agent evaluation, including secure sandboxing and parallel execution.Source: A1 EXTENDED RELATED WORK · pp. 18–19

• HAL accepts agents and benchmarks through common interfaces, provisions isolated environments, captures interactions, evaluates outputs, and aggregates results for leaderboard reporting.A minimal Python run(input, **kwargs) → dict contract separates agent logic from benchmark task specifications and scoring procedures.Source: Figure A1 · p. 19, A2 ADDITIONAL DETAILS ON THE HAL… · pp. 19–20
• Standardized Weave logging records model and tool calls, API parameters, token usage, costs, call graphs, and task-grouped metadata without modifying agent logic.The harness initializes Weave automatically and assigns unique run and task identifiers to organize trajectories and usage summaries.Source: A2 ADDITIONAL DETAILS ON THE HAL… · p. 19
• Hundreds of evaluations can run concurrently across local, Docker, or Azure VM backends with automatic provisioning, timeouts, artifact collection, exception tracking, and cleanup.Each task receives its own isolated container or VM, with GPU VMs available for evaluations requiring them.Source: A2 ADDITIONAL DETAILS ON THE HAL… · p. 20
• The architecture lets one agent run across varied benchmarks while aggregating accuracy, cost, and latency, avoiding benchmark-specific harness implementations.Benchmarks provide inputs, files, runtime constraints, and scoring, whereas agents map task identifiers to submissions and may use models and tools internally.Source: A2 ADDITIONAL DETAILS ON THE HAL… · p. 20

• $5,000 monthly spend limits and high per-model costs restrict parallel evaluations and make confidence intervals prohibitively expensive.HAL therefore relied on single runs without statistical validation for most evaluations, while rate limits forced sequential testing or special provider access.Source: A3 PRACTICAL HURDLES IN LARGE-SCALE… · pp. 20–21
• Unannounced model-weight swaps and provider-dependent quantization introduce hidden variance, making apparently identical evaluations incomparable.Together AI changed a stable DeepSeek R1 endpoint to DeepSeek R1 0528, while OpenRouter could route successive calls to FP4 or FP8 variants.Source: A3 PRACTICAL HURDLES IN LARGE-SCALE… · pp. 20–21
• Breaking API changes and hardcoded library workarounds make new models difficult to evaluate and comparisons with prior evaluations unreliable.OpenAI removed the stop argument from o4-mini and o3, while LiteLLM’s reasoning-support regex excluded GPT-5 until library updates.Source: A3 PRACTICAL HURDLES IN LARGE-SCALE… · p. 21
• Rate-limit failures can silently produce false negatives by recording infrastructure errors as incorrect answers rather than capability failures.This occurs when agents lack retries or error surfacing, causing the evaluation framework to misclassify provider failures.Source: A3 PRACTICAL HURDLES IN LARGE-SCALE… · p. 21
• Provider interfaces and reasoning-effort settings lack standardization, preventing equivalent-compute comparisons and making provider switching difficult.LiteLLM maps “high” reasoning effort to 4,096 tokens, OpenAI does not disclose its setting semantics, and OpenRouter uses a different parameter format.Source: A3 PRACTICAL HURDLES IN LARGE-SCALE… · p. 21
• Entangled task specifications, evolving dependencies, and upstream logging bugs undermine behavioral validity and computational reproducibility.AssistantBench instructions can cause capable models to refuse answers; changing libraries alter behavior; and Weave’s logging bug blocked progress for months.Source: A3 PRACTICAL HURDLES IN LARGE-SCALE… · p. 21

• HAL’s current results are affected by several limitations despite its comprehensive evaluation infrastructure.The paper frames these limitations as constraints on interpreting the current findings.Source: A4 LIMITATIONS AND FUTURE WORK · p. 21
• The paper documents the limitations transparently to clarify the scope of its current results.Source: A4 LIMITATIONS AND FUTURE WORK · p. 21
• Future work will address the identified limitations through planned improvements.Source: A4 LIMITATIONS AND FUTURE WORK · p. 21

Incomplete cost accounting for caching

• SWE-Agent caching reduces API costs, but HAL’s current calculations report full token prices without accounting for cache hits.The next HAL update will implement cache-aware cost tracking for SWE-Agent.Source: A4.1 LIMITATIONS WE PLAN TO ADDRESS IN… · p. 21

Evaluation on public test sets

• HAL evaluates GAIA and AssistantBench on public test sets rather than the private sets used for official leaderboards, which may have different task distributions.The planned remedy is collaboration with benchmark authors to enable secure evaluation on private test sets.Source: A4.1 LIMITATIONS WE PLAN TO ADDRESS IN… · p. 22

Limited benchmark coverage

• HAL uses SWE-Bench Verified Mini’s 50 tasks instead of the full 500-task dataset and uses original TAU-bench despite newer τ 2 Bench addressing known specification and metric issues.Future releases will expand to complete benchmark versions.Source: A4.1 LIMITATIONS WE PLAN TO ADDRESS IN… · p. 22

Incomplete evaluation matrix

• HAL reports 142 model-scaffold-benchmark combinations from 186 runs, with missing configurations caused by incomplete execution and prohibitive costs.The generalist agent was not run on all GAIA models, and Online Mind2Web with Claude Opus 4.1 was omitted because estimated evaluation costs were about $20,000.Source: A4.1 LIMITATIONS WE PLAN TO ADDRESS IN… · p. 22

Suboptimal API configurations

• HAL agents use the completions API for OpenAI models instead of the newer responses API, which can improve structured output handling for some tasks.HAL plans to update agent architectures to use recommended API configurations across providers.Source: A4.1 LIMITATIONS WE PLAN TO ADDRESS IN… · p. 22

Fundamental constraints

• Proprietary reasoning settings prevent HAL from standardizing or interpreting OpenAI’s “low,” “medium,” and “high” levels, so evaluations document the settings used.The specifications of these provider-specific parameters are not public, limiting comparability in computational effort.Source: A4.2 FUNDAMENTAL CONSTRAINTS · p. 22
• Parallel execution makes latency measurements variable because of VM provisioning, network effects, and server load, while serial validation would extend evaluations from hours to weeks.HAL plans small-sample serial runs to provide latency estimates.Source: A4.2 FUNDAMENTAL CONSTRAINTS · p. 22
• Automated log analysis locates agent failures but cannot establish whether correcting one would complete the task or expose later errors.Causal testing would require checkpointing agent and environment states and replaying execution after correcting each error, which currently exceeds the computational budget.Source: A4.2 FUNDAMENTAL CONSTRAINTS · p. 22
• Despite these constraints, HAL enables systematic, standardized, and reproducible comparisons across models, scaffolds, and benchmarks that were previously infeasible.The authors plan to address limitations, expand coverage, and welcome community contributions to accelerate progress.Source: A4.2 FUNDAMENTAL CONSTRAINTS · p. 22

A5 DATA LEAKAGE IN TAU-BENCH FEW SHOT AGENT

• Test-set examples in the official TAU-bench Few Shot agent made its accuracy measurements meaningless, so all results from that scaffold were excluded.The leakage was discovered after approximately $1,000 in compromised evaluations had already been run.Source: A5 DATA LEAKAGE IN TAU-BENCH FEW SHOT… · p. 23
• Automated Docent log analysis detected the leakage that final accuracy inspection would have missed, motivating systematic auditing of benchmark implementations and in-context demonstrations.Without this analysis, the study might have reported artificially inflated performance and propagated flawed capability measurements.Source: A5 DATA LEAKAGE IN TAU-BENCH FEW SHOT… · p. 23

A7.1 METHODOLOGY

• Docent analysis covered 48 model–scaffold pairs and 2,184 transcripts, using rubric-based GPT-5 Medium judgments and conditional-probability comparisons for reliability and failure modes.After removing TAU-bench from reliability and failure analysis, the sample contained 36 model–scaffold pairs and 1,634 transcripts.Source: A7.1 METHODOLOGY · p. 26
• For AssistantBench, success used a 0.75 score threshold, while SciCode used whether any subtask passed because task-level accuracy was always below 10%.CORE-Bench and TAU-bench used binary task success; AssistantBench abstentions were filtered for reliability and instruction-following analyses.Source: A7.1 METHODOLOGY · p. 26

A7.2 MANUAL VALIDATION

• Manual validation covered at least 30 flagged runs for one rubric on AssistantBench, CORE-Bench, and SciCode, reporting precision and inter-LLM reliability in Table A2.Precision, rather than accuracy, was reported because validating false positives was easier than validating false negatives in long, complex transcripts; GPT-5 Medium was compared with Claude Sonnet 4 Medium for one AssistantBench rubric.Source: A7.2 MANUAL VALIDATION · p. 26, Table A2 · p. 27

A7.4 EXAMPLES FROM THE DOCENT RUBRIC ANALYSIS

• The qualitative analysis documented shortcuts, benchmark gaming, unsafe actions, guessing, failed abstentions, and hard-coded answers across TAU-bench, CORE-Bench, AssistantBench, and SciCode.Examples include excessive refunds, failure to charge for an extra bag, finding benchmark answers on HuggingFace or arXiv, and hard-coded SciCode answers.Source: A7.4 EXAMPLES FROM THE DOCENT RUBRIC… · pp. 27, 30, Table A6 · p. 29, Table A7 · p. 29

A8 DETAILS ON THE HAL GENERALIST AGENT

• The HAL Generalist Agent uses smolagents CodeAgent with a LiteLLMModel in a four-step planning loop capped at 200 steps, supported by search, browsing, coding, shell, file, inspection, and vision tools.Only minimal benchmark-specific scaffolding is added, such as answer-format instructions or environment initialization code.Source: A8 DETAILS ON THE HAL GENERALIST AGENT · p. 36

A9 PREVIOUS WORK ON MODEL AND BENCHMARK COMBINATIONS

• The prior-work comparison consolidates leaderboard entries across models, scaffolds, and benchmarks, including GAIA, Online Mind2Web, SciCode, SWE-Bench, and SWE-Bench Verified.The review assumes medium reasoning for unspecified OpenAI models, excludes neither SWE-bench nor SWE-bench Verified because SWE-bench Verified Mini is a subset of both, and condenses multiple scaffold versions or test submissions.Source: A9 PREVIOUS WORK ON MODEL AND… · pp. 37, 39
• Prior work on TauBench Airline combines o4-mini High with OpenHands, Claude-3.7 Sonnet with reasoning/planning prompt addenda and tool use, DeepSeek R1, and GPT-4.1 or o3 with optimized user simulators.Source: A9 PREVIOUS WORK ON MODEL AND… · p. 40

A10.1 AN OVERVIEW OF THE TABLES AND PLOTS

• HAL’s benchmark sections use a consistent reporting framework combining leaderboards, Pareto accuracy–cost plots, token counts, task-difficulty heatmaps, and accuracy trends by model release date.Leaderboards report best-run accuracy, cost, and Pareto-optimality; Pareto plots include run-range error bars, while heatmaps compare best-agent and any-agent success.Source: A10.1 AN OVERVIEW OF THE TABLES AND… · p. 41

A10.2 ASSISTANTBENCH / A10.3 CORE-BENCH

• AssistantBench evaluates Browser-Use on 33 realistic browsing tasks across 12 models, while CORE-Bench Hard evaluates reproducibility on 45 papers using CORE-Agent and HAL Generalist Agent across 19 models.AssistantBench’s 214-task benchmark was sampled to 33 tasks; CORE-Bench Hard requires installing dependencies, running code, and interpreting outputs and figures.Source: A10.2 ASSISTANTBENCH · p. 41, A10.3 CORE-BENCH · p. 44

A10.4 GAIA / A10.5 ONLINE MIND2WEB / A10.6 SCICODE

• GAIA, Online Mind2Web, and SciCode extend coverage to general assistants, live web interaction, and scientific code generation with multi-model, multi-scaffold evaluations.GAIA uses 165 validation questions and 23 evaluations; Online Mind2Web covers 300 verified tasks across 136 live websites; SciCode contains 338 subproblems and compares zero-shot with tool-calling scaffolds.Source: A10.4 GAIA · p. 47, A10.5 ONLINE MIND2WEB · p. 50, A10.6 SCICODE · p. 53

A10.7 SCIENCEAGENTBENCH

• ScienceAgentBench evaluates data-driven scientific discovery on 102 expert-validated tasks, comparing SAB Self-Debug with HAL Generalist Agent across 12 models.The tasks come from 44 peer-reviewed publications and were validated by nine subject matter experts.Source: A10.7 SCIENCEAGENTBENCH · p. 56

A10.8 SWE-BENCH VERIFIED MINI / A10.9 TAU-BENCH AIRLINE / A10.10 USACO

• SWE-bench Verified Mini, TAU-bench Airline, and USACO cover real GitHub issues, airline tool–user interactions, and competitive programming through 27, 14, and 13 evaluations, respectively.Their benchmark scopes are 50 software-engineering tasks, 50 airline tasks, and 307 USACO problems; the USACO runs mostly use Episodic + Semantic, with one HAL Generalist Agent run.Source: A10.8 SWE-BENCH VERIFIED MINI · p. 58, A10.9 TAU-BENCH AIRLINE · p. 61, A10.10 USACO · p. 64