PaperlayerView exact revision on arXiv

Source-linked AI summary

Establishing Best Practices for Building Rigorous Agentic Benchmarks

Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, Daniel Kang

arXiv:2507.02825v5cs.AI

TL;DR

Agentic benchmarks evaluate complex real-world tasks, but flaws in task setup and outcome evaluation can distort reported agent capabilities. The paper introduces ABC, a checklist synthesized from prior issues, best practices, and benchmark-development experience, and applies it to benchmark assessment and improvement, reducing CVE-Bench’s performance overestimation by 33% in absolute terms.

  • Problem

    Many agentic benchmarks have task-validity and outcome-validity issues, making it important to ensure their reported capability measurements can be trusted.

  • Method

    The authors synthesize an Agentic Benchmark Checklist from benchmark-building experience, surveyed best practices, and previously documented evaluation issues.

  • Results

    33% in absolute terms: applying ABC to CVE-Bench reduced performance overestimation, while assessing ten benchmarks identified issues causing estimation errors of up to 100% in relative terms.

  • Takeaways & Limitations

    ABC provides actionable guidance for designing, assessing, interpreting, and reporting agentic benchmarks across task validity, outcome validity, and benchmark reporting.

Abstract

from arXiv · show

Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench Verified uses insufficient test cases, while TAU-bench counts empty responses as successful. Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%.

1 Introduction

Agentic benchmarks evaluate complex tasks through task outcomes, but flaws in task setup and outcome evaluation can substantially distort reported capabilities. The paper identifies these validity threats, proposes ABC, and applies it to benchmark assessment and improvement.

  • Motivation: Agentic benchmarks track AI-agent progress on realistic tasks by evaluating whether agents achieve successful end-to-end outcomes.Their growing use makes trustworthy evaluation results important for research and decision-making.
  • Validity threats: SWE-bench-Verified can accept incorrect patches because its manually vetted unit tests may miss important edge cases, making 24% of the top 50 leaderboard positions incorrect.The example illustrates an outcome-validity failure.
  • Validity threats: Outcome validity requires evaluation results to genuinely indicate task success, whereas task validity requires success to correspond to possessing the target capability.The paper treats these as two major conditions for valid benchmark results.
  • Validity threats: τ-bench allows a trivial agent without airline-ticketing knowledge to pass 38% of tasks, demonstrating a task-validity failure caused by task design.The paper also reports that empty responses can be counted as successful.
  • Contributions: ABC organizes best practices for assessing task validity, outcome validity, and benchmark reporting.The checklist is intended for critically assessing existing benchmarks and guiding future development.
  • Contributions: Applying ABC to ten widely used benchmarks identified evaluation issues causing agent-performance estimation errors of up to 100% in relative terms.The paper also presents a case study applying ABC to improve an agentic benchmark during development.

2 Related Work

Prior work has examined many dimensions of AI-benchmark quality and documented issues in individual agentic benchmarks. Agentic benchmarks are especially complex because they evaluate non-categorical outcomes produced through multistep, real-world interactions.

  • Assessing AI Benchmarks: Benchmark-quality research has addressed label quality, evaluation protocols, construct validity, data contamination, reproducibility, and practical usage.These dimensions provide context for evaluating benchmark reliability beyond raw scores.
  • Benchmarking of AI Agents: Agentic benchmarks span coding, environment interaction, mathematics, and other domains while typically requiring non-categorical outputs and multistep execution.This design emulates real-world challenge resolution and makes evaluation more complex than traditional benchmarks.
  • Issues in Evaluating AI Agents: Prior analyses report outcome-evaluation problems including performance underestimation from implicit answer-format assumptions and successful evaluations without correct patches.Reported examples include 5.3% underestimation and incorrect-patch passes on 7.7% of SWE-bench-Lite tasks and 5.2% of SWE-bench-Verified tasks.

3 Overview

The paper frames rigorous agentic evaluation around task validity and outcome validity, then develops and applies ABC through benchmark collection, checklist construction, and systematic assessment. Agentic settings make these criteria difficult because they combine complex environments with unstructured outcomes.

  • Taxonomy: Agentic evaluation is rigorous when the target capability matches task success and task success matches a positive evaluation result.These equivalences define task validity and outcome validity, respectively.
  • Taxonomy: Unstructured outcomes such as text, code, and file edits require specially designed evaluation methods whose failures can violate outcome validity.SWE-bench-Verified illustrates how incomplete tests can accept patches that remain incorrect.
  • Taxonomy: Complex task setups can violate task validity because environments, tools, legacy state, or implementation flaws may create shortcuts or impossible tasks.The paper gives τ-bench’s empty-response success behavior and intentionally unattainable tasks as examples.
  • Checklist development: ABC translates the two validity criteria into actionable checks and provides guidance for interpreting and reporting scores when a criterion cannot be fully satisfied.The checklist also includes benchmark reporting as a third part.
  • Study design: The authors collected 17 popular agentic benchmarks, developed ABC from literature, best practices, and benchmark-building experience, and assessed ten selected benchmarks.Satisfied checklist items received one point, while identified issues were tested experimentally for quantitative validation.

4 ABC: Agentic Benchmark Checklist

ABC provides checks for task validity, outcome validity, and benchmark reporting across tools, environments, implementations, outcome types, and evaluation methods. Its guidance targets shortcuts, impossible tasks, invalid grading, and inadequate transparency about unavoidable limitations.

  • Task validity: Task-validity checks address design and implementation flaws that create shortcuts causing false positives or impossible tasks causing false negatives.The checklist covers tools, environments, and implementation correctness.
  • Task validity: Tool checks require explicit self-hosted tool versions, reliable API services, rate-limit management, and termination with user notification when interruptions occur.These checks help keep the benchmark’s operating conditions specified and observable.
  • Task validity: Environment and implementation checks include cleaning legacy state, verifying ground-truth annotations and task setup, using an oracle solver, and inspecting pilot outliers.The examples target unintended information leakage and configuration vulnerabilities.
  • Outcome validity: Outcome-validity checks are organized by outcome type and evaluation method, including information acquisition, code generation, state modification, and multistep reasoning.The framework distinguishes methods such as string matching, LLM judges, testing, and state matching.
  • Outcome validity: Matching-based evaluation should account for equivalent expressions, negation, format assumptions, guessing, and listing all possible answers.The checklist separately recommends pilot experiments to assess LLM-judge accuracy and self-consistency.
  • Outcome validity: Testing-based evaluation should verify test correctness and quality with objective measures, while state matching should cover all successful outcomes and relevant versus irrelevant states.These checks aim to reduce false positives from incomplete tests or insufficient state spaces.
  • Benchmark reporting: Because evaluation issues may be unavoidable, ABC requires transparent communication of their impact, mitigation efforts, and qualitative and quantitative evidence.The reporting guidance emphasizes openness, validity documentation, and disclosure of limitations.

5 Assessment of Agentic Benchmarks

Applying ABC to ten selected agentic benchmarks revealed widespread flaws in task validity, outcome validity, and benchmark reporting. Detailed cases show that these flaws can substantially overestimate or underestimate agent performance, while ABC also guides improvements to CVE-Bench.

  • Assessment Scope: ABC assessment covered ten open-source agentic benchmarks spanning all capability categories and evaluation methods.The assessment calculated average scores for applicable checklist items and reported final results in Figure 5.
  • Task Validity: More than half of the benchmarks exhibited implementation flaws, especially those providing tools to agents.These flaws concern whether tasks are solvable if and only if agents possess the target capability.
  • Assessment Results: Seven benchmarks violated task validity, seven violated outcome validity, and all ten had limitations in reporting.The benchmark-level assessment identified problems across all three ABC components.
  • Outcome Validity: More than half of the benchmarks failed to address inherent limitations of their evaluation methods.This indicates widespread weaknesses in whether evaluation results truly indicate task success.
  • Benchmark Reporting: 80% of benchmarks failed to acknowledge weaknesses in design or implementation, and none satisfied every reporting criterion.The reporting assessment found incomplete disclosure of benchmark limitations across the selected set.
  • Assessment Findings: Specific issues produced large estimation errors, including 38% and 40% overestimation in τ-bench, 1.4–5.2% in WebArena, and 31% in KernelBench.Other findings included 100% SWE-Lancer scores without solving tasks and 28% OSWorld performance underestimation caused by changed websites.
  • Assessment Findings: In τ-bench, unsolvable tasks and substring-matched database text allowed agents to pass by doing nothing or dumping the database.The airline subset contained 38% intentionally unsolvable tasks, while 2% of airline tasks and 3.6% of retail tasks used verbatim database text as ground truth.
  • Revising CVE-Bench: CVE-Bench improvements addressed both outcome and task validity, including a time-based injection flaw that overestimated performance by 32.5%.The benchmark also exposed an outbound-server shortcut: denying external requests reduced agent success rates by 10%.

6 Conclusion

The paper introduces ABC as actionable checklists for outcome validity, task validity, and result reporting, then uses it to assess benchmarks and improve CVE-Bench.

  • ABC provides best practices for building rigorous agentic benchmarks, focusing on outcome validity, task validity, and reporting.

A Limitation and Impact Statement

The study’s conclusions are bounded by its benchmark sample, current understanding of agent capabilities, and the versions analyzed at writing.

  • The analysis covers only 17 agentic benchmarks used by top AI providers between January 2024 and March 2025.Benchmarks outside this period were not analyzed, so the checklist may not be exhaustive.
  • The taxonomy and analysis reflect the current understanding of AI-agent reasoning capabilities.Future capabilities may introduce evaluation challenges not addressed by this study.
  • The findings reflect benchmark versions available at the time of writing and may not fully apply to subsequent revisions.
  • The study aims to raise awareness and foster stronger standards and improved quality in agentic benchmarks.The authors anticipate more critical evaluation of benchmark results and reassessment of AI-agent leaderboards.

B Details of Benchmark Collection and Selection

The authors collected agentic benchmarks from major AI-provider releases and award-winning academic work, narrowed them by agentic criteria, and selected ten for detailed assessment.

  • The initial survey identified 78 benchmarks from top AI providers’ releases and award-winning peer-reviewed academic work.The collection focused on state-of-the-art models released between January 2024 and March 2025.
  • 25 collected benchmarks were classified as agentic because their tasks require multistep reasoning or command execution.Simple fact-seeking, straightforward question-answering, and straightforward programming tasks were excluded.
  • Ten benchmarks were selected for in-depth assessment based on open-source availability and coverage of evaluated capabilities and evaluation methods.
  • The benchmark corpus includes examples such as FACTS Grounding, MRCR, MBPP, Wild Bench, BBH, and Kernel-Bench.
  • ABC was synthesized from benchmark-development experience, existing best practices, and lessons from issues in existing benchmarks.
  • The appendix provides assessment reports for benchmarks including Bird-Bench, CyBench, SWE-Bench-Verified, τ-Bench, MLE-Bench, WebArena, GAIA, OSWorld, and KernelBench.Each report’s caption specifies the corresponding paper and codebase evaluated.

E.1 SWE-bench

SWE-bench evaluates agents by testing generated patches against repository unit tests, but insufficient tests can accept incorrect patches and distort reported performance.

  • SWE-bench gives agents GitHub issues and codebase summaries, then evaluates generated patches using existing repository unit tests.
  • Unit tests can produce false positives because their cases may be insufficient to establish patch correctness.
  • The PolyFit() test checks only scenarios where both x and y contain missing values, leaving other input conditions untested.
  • An incorrect PolyFit() patch generated by IBM SWE-1.0 was evaluated as correct because of the insufficient test case.
  • 5.3% of Verified tasks and 7.7% of Lite tasks can pass without resolving their GitHub issues.These tasks change the Verified and Lite leaderboards by 40.9% and 24.4%, respectively, and overestimate performance by 2.3% and 1.6%.

E.2 τ-bench

τ-bench evaluates agents on user interactions governed by domain-specific policies, but its outcome checks can reward agents that do not perform the required work. This permits trivial or random behavior to pass substantial portions of Airline and Retail tasks.

  • Benchmark overview: τ-bench evaluates agents’ interactions with users under domain-specific rules, using database-state changes and optional required response text.The benchmark targets airline and retail-style policy-following tasks.
  • Identified issue: When a task requires neither a database change nor meaningful response text, doing nothing can receive a positive evaluation.The evaluation does not require the agent to demonstrate the intended interaction in these cases.
  • Identified issue: A flight-cancellation task can be passed without checking the ticket or denying a non-refundable request, because the task has no required output and unchanged state is accepted.The example illustrates a false positive caused by the reward design.
  • Qualitative results: 38% of Airline tasks and 6.0% of Retail tasks are passed by a do-nothing agent for any k.These figures are reported as pass^k or pass@k rates.
  • Qualitative results: 40% of Airline tasks and 9.6% of Retail tasks are passed by a spamming agent that outputs all the data for any k.This shows that some evaluation outcomes can be achieved without appropriate task behavior.
  • Related evaluation issue: BIRD can also misjudge valid SQL answers because multiple queries may answer the same natural-language question despite producing different outputs.For the smallest-population example, Query 1 returns all tied countries while Query 2 returns one; both answer the question.

E.4 SWE-Lancer

SWE-Lancer evaluates agents that implement features and fix bugs by modifying a codebase and passing end-to-end tests, but its test archive can be altered without authentication. Consequently, an agent can obtain a perfect reported resolution rate without completing the software-engineering tasks.

  • Benchmark overview: SWE-Lancer evaluates agents’ ability to implement features and fix bugs independently by modifying a codebase through Python scripts.Agents receive task descriptions and interact with the filesystem.
  • Identified issue: SWE-Lancer uses end-to-end tests to evaluate implementation correctness, but agents can inspect and update the password-protected archive’s directory and files without its password.The archive’s protection does not prevent test-case modification.
  • Identified issue: Agents can replace the original tests with a trivial assertion such as “assert 1==1”.This creates a shortcut that bypasses the intended software-engineering challenge.
  • Qualitative results: 100% resolve rate is achievable by overwriting the archived test cases without completing the software-engineering tasks.The reported resolution rate therefore does not necessarily indicate successful implementation.
  • Related evaluation issue: WebArena illustrates a separate evaluation problem: exact matching rejects alternative expressions, substring matching permits exhaustive enumeration, and LLM judging can be unreliable.In one example, a 53-minute answer was judged correct despite a 63-minute ground truth.

E.6 KernelBench

KernelBench evaluates GPU kernels for functional correctness and efficiency, but its testing and execution setup leaves several shortcuts and blind spots. Additional adversarial tests found that generated-kernel correctness was overestimated by 31%.

  • Benchmark overview: KernelBench asks agents to write PyTorch code containing an inline GPU-kernel implementation that is functionally correct and more efficient than the original.The benchmark evaluates both correctness and efficiency.
  • Identified issue 1: Uniformly random fuzzing misses some incorrect behaviors because functions such as mean(softmax(x)) and relu(x-2) are insensitive to those inputs.The tested input distribution does not expose every relevant failure mode.
  • Identified issue 2: The evaluation can permit cheating because it runs the ground-truth kernel before the generated kernel, allowing generated programs to extract the reference result.This is an execution-order vulnerability in the evaluation implementation.
  • Identified issue 3: KernelBench’s fuzzer does not cover different memory layouts, tensor shapes, or hardware environments, so some thread-related errors are graded as correct.The cited kernel requires a thread-safe guard under certain thread-count conditions.
  • Implementation context: The kernel implementation assigns each block an outer instance and each thread a strided partial-sum range before reducing results in shared memory.The code comments describe flexible two-dimensional indexing, per-instance pointers, partial sums, and parallel reduction.
  • Quantitative result: 31% is the reported overestimation of generated-kernel correctness after manually verified test cases were added to generations.The authors sampled three generated functions per level-1 task, used o3-mini to propose tests, manually verified them, and applied them to all generations.

F An Example of Rigorous Benchmark Reporting

The BIRD example shows how rigorous benchmark reporting can connect evaluated capabilities to task constructs, expose and quantify unavoidable flaws, and report uncertainty and baselines. It also recommends interpreting success rates cautiously when ground-truth errors remain.

  • Construct validity: BIRD’s reporting example evaluates natural-language-to-SQL capability by providing a query, database schema, and SQL knowledge, then requiring executable SQL that returns correct answers.The example explicitly links the target capability to the benchmark’s task and outcome.
  • Evaluation subject: The reporting checklist requires the benchmark to identify its evaluation subject, including ML models and agent frameworks.This clarifies what entity the reported performance describes.
  • Mitigation: BIRD addresses two execution-based evaluation limitations—ties with LIMIT queries and incorrect annotations—by verifying a random sample of 500 tasks.The verification found that 11.65% of sampled ground-truth queries were incorrect.
  • Impact analysis: 11.65% of verified sampled ground-truth queries were incorrect, and such errors can produce estimation errors and incorrect agent rankings.The example treats the annotation problem as an unavoidable flaw whose impact should be documented.
  • Quantitative reporting: The reporting example models performance uncertainty under a normality assumption and derives confidence intervals from the estimated ground-truth incorrectness rate.It uses µ = e + (1 − 2e)p0 and σ2 = µ(1 − µ), then calculates intervals using the 11.65% plug-in estimate.
  • Baselines and checks: The example reports confidence intervals for each model, a SQL-expert success rate of 92.96%, and a trivial-agent sanity check with 0% success.These additions provide uncertainty, a non-AI baseline, and a check on harness rigor.
  • Interpretation: Because BIRD has potential flaws, the example advises users not to rely on success rate alone for model selection and instead to consult its confidence interval.This is guidance for interpreting results rather than a claim that the benchmark is unusable.
Loading 2507.02825v5…