Source-linked AI summary
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
Xinming Tu, Tianze Wang, Yingzhou, Lu, Kexin Huang, Yuanhao Qu, Sara Mostafavi
TL;DR
Execution-based agent benchmarks can mismeasure capability when their coupled artifacts contain inconsistent specifications, programs, evaluators, or environments. BenchGuard audits these artifacts with structured frontier-LLM protocols and optional execution evidence, finding author-confirmed ScienceAgentBench defects and matching 83.3% of expert revisions on BIXBench. The results support automated auditing as a complement to human review and as a potential quality gate in benchmark construction.
Problem
Execution-based benchmarks can contain cross-artifact inconsistencies that make valid agent behavior appear incorrect, while human review may miss these defects.
Method
BenchGuard cross-verifies instructions, reference solutions, evaluation logic, and environments with structured LLM protocols, deterministic checks, and optional agent traces.
Results
BenchGuard identified 12 author-confirmed defects in ScienceAgentBench and exactly matched 83.3% of expert revisions on BIXBench.
Takeaways & Limitations
Automated auditing can complement human review and inform benchmark revision and construction as a quality-control step.
Takeaways & Limitations
Findings can be hallucinated and require expert review, while the framework is scoped to task-oriented benchmarks with defined verification logic.
Abstract
from arXiv · showhide
As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all - they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through BenchGuard, the first automated auditing framework for task-oriented, execution-based agent benchmarks. BenchGuard cross-verifies all benchmark artifacts via structured LLM protocols, optionally incorporating agent solutions or execution traces as additional diagnostic evidence. Deployed on two prominent scientific benchmarks, BenchGuard identified 12 author-confirmed issues in ScienceAgentBench - including fatal errors rendering tasks unsolvable - and exactly matched 83.3% of expert-identified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely. A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review. These findings point toward AI-assisted benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself.
1 Introduction
Execution-based benchmarks can fail because their instructions, environments, reference solutions, and evaluation logic interact inconsistently. BenchGuard audits these cross-artifact relationships with structured protocols, identifying defects that human review can miss.
- Execution-based benchmarks couple instructions, executable ground truth, evaluation scripts, and environments, so correctness emerges from their interactions rather than a single label.
- A four-category, 14-subcategory taxonomy organizes benchmark defects by the constituent artifacts of execution-based tasks.
- BenchGuard audits 50 complex tasks with five frontier models for under $15.
- It uncovered author-confirmed, solve-rate-altering defects in ScienceAgentBench and achieved expert-level alignment on BIXBench, showing that careful human review can remain insufficient.
- BenchGuard frames benchmark auditing as cross-artifact consistency checking, surfacing misalignments as structured findings for expert review.
2 Related Work
Prior benchmark verification has exposed persistent quality problems, but execution-based agent benchmarks create additional failures because correctness depends on coupled executable artifacts. This motivates auditing the full evaluation pipeline rather than reviewing labels or checkers in isolation.
- Prior audits of static and execution-based benchmarks show that small specification, annotation, or evaluator errors can materially alter measured performance.
- Most earlier auditing pipelines treat the evaluation oracle as self-contained, whereas execution-based benchmarks distribute correctness across instructions, programs, evaluators, and environments.
- Execution-based tasks can appear reasonable component by component while still producing mis-measurement through specification–implementation mismatches.
- Human verification is useful but one-off review may not suffice because test suites, contamination, evaluator limitations, and model-judge reliability can distort reported scores.
3 Method
BenchGuard cross-verifies the four artifacts defining an execution-based task, combining structured LLM reasoning with deterministic checks and optional execution evidence. It produces confidence- and severity-annotated findings for human adjudication within a defined scope.
- BenchGuard ingests the instruction, reference solution, evaluation script, and environment, then returns categorized findings with severity, confidence, and cited evidence.
- The definition-level audit combines LLM-driven cross-verification with deterministic static checks and can incorporate agent solutions or execution traces as diagnostic evidence.
- BenchGuard targets tasks with defined inputs, outputs, and verification logic, excluding open-ended conversation, non-deterministically verifiable interaction, and irreversible real-world actions.
- A consolidated LLM call cross-verifies all four artifact types and deduplicates findings so one root cause is reported once.
- The six-phase protocol checks task understanding, gold-program correctness, evaluation logic, specification feasibility, environment assumptions, and consolidated atomic findings.
- Execution-level auditing appends generated programs, evaluation results, and logs, improving ScienceAgentBench recall for Opus 4.6 from 83.3% to 91.7%.
4 Experimental Setup
The evaluation uses two complementary benchmarks, five frontier LLM backends, reproducible audit configurations, and recall–precision metrics grounded in confirmed or expert-revised defects. Human experts validate the resulting findings.
- BenchGuard is evaluated on two agent benchmarks selected for complementary evaluation paradigms and quality-assurance histories.
- ScienceAgentBench contains 102 scientific data-analysis tasks evaluated by deterministic Python scripts and previously reviewed by its creators and nine subject-matter experts.
- Five frontier LLM backends assess robustness across model families and support cross-model agreement and ensemble analysis.
- Audits use definition mode with deterministic checks, reproducibility-oriented temperature settings, and one independent LLM call per task.
- Recall measures identified confirmed defects, while flagged-task precision measures aligned findings within tasks containing confirmed defects.
- Human experts validate all findings; ScienceAgentBench findings receive confirmation from the original benchmark authors, while BIXBench findings use verified ground truth and independent review.
5 Results and Analysis
BENCHGUARD exposed substantial benchmark defects and showed strong but imperfect alignment with expert revisions, while its models contributed complementary diagnostic coverage. The results also illustrate that automated and human auditors detect different classes of issues.
- ScienceAgentBench: 12 author-confirmed ScienceAgentBench defects included fatal specification errors, metric mismatches, and evaluation logic that rejects correct solutions.
- BIXBench: The most frequent BIXBench issue was underspecified instructions and evaluation rubrics anchored to a single reference implementation, which can penalize valid alternatives.
- BIXBench: 83.3% exact alignment matched 20 of 24 BIXBench expert issues, while partial matches raised recall to 95.8% (23/24).
- Collaboration boundary: BENCHGUARD’s partial match on an unnecessary decimal-format constraint illustrates restraint, whereas its detection of undocumented sample removal illustrates complementary blind spots.
- Collaboration boundary: All five models caught a gold program’s undocumented removal of samples KL3 and WL3, although experts missed it.
- Complementary diagnostic profiles: 83.3% exact alignment came from the five-model union versus 54.2% for the best single model, and the full 50-task audit cost under $15.
6 Discussion and Conclusion
The audits show that benchmark defects can survive careful human review, while automated findings can support benchmark repair and earlier quality control. The authors position BENCHGUARD as a complement to expert review and a possible component of benchmark construction.
- BENCHGUARD found 12 author-confirmed ScienceAgentBench defects and matched 83.3% of independent BIXBench expert revisions.
- Automated audit findings can provide actionable evidence for repairing instructions, gold programs, and evaluation logic.
- The authors propose integrating auditing into benchmark construction so implicit assumptions are surfaced before reaching published artifacts.
LLM Disclosure
The authors disclose using frontier LLMs as benchmark auditors and alignment judges, while retaining author responsibility for the scientific work.
- Frontier LLMs served as benchmark auditors and alignment judges, while the authors conducted the scientific claims, experimental design, and analysis.
A Full Error Taxonomy
BENCHGUARD organizes benchmark auditing around a structured taxonomy and protocol covering artifact correctness, evaluation logic, task specifications, and environments. Its rules prioritize specific, independently fixable findings and produce evidence-backed JSON reports.
- Taxonomy and audit areas: The taxonomy contains 14 subcategories spanning four audit areas grounded in benchmark artifacts.
- Severity and confidence: Severity ranges from Critical for impossible tasks to Low for minor issues, while confidence tiers span Confirmed, Likely, and Possible.
- Categorization rules: Instruction-level categories take priority over ground-truth categories when ambiguity causes the gold program to implement one interpretation of an underspecified specification.
- Finding granularity: Each bug must be reported once under its most specific subcategory, and each finding must describe exactly one independently fixable root cause.
- Output format: Findings are emitted as JSON records containing categories, severity, finding type, title, description, evidence, recommendation, and confidence.
- Prompt structure: The protocol presents all task artifacts in a structured context, including the instruction, gold program, evaluation script, data description, and environment information.
- Audit protocol: Six audit phases check task understanding, ground-truth correctness, evaluation logic, task specification, environment issues, and consolidated findings.
- Execution-level auditing: Agent programs are supporting evidence rather than ground truth and may expose assumptions or ambiguities through execution-level auditing.
C Benchmark Ingestion Details
BENCHGUARD standardizes benchmark ingestion around task directories containing metadata, instructions, tests, solutions, and optional environment files. It supports tiered inputs and records audit findings and released per-task evidence across the evaluated benchmarks.
- Task layout: Each task is recognized only when its directory contains a task.toml file.The file stores metadata and runtime configuration, but all fields are optional beyond the file’s existence.
- Task layout: The standardized layout includes required instructions and tests, a recommended reference solution, and optional environment and domain-knowledge files.A root-level benchguard_hints.yaml can inject benchmark-specific review guidance.
- Input tiers: Three input tiers range from instructions plus tests to full cross-artifact auditing with the gold program.Minimal input enables EVAL, INST, and ENV checks; adding the gold program enables definition-level auditing.
- Input tiers: BENCHGUARD can incorporate agent results or execution traces to enable trajectory-informed verification.Execution-level input is an optional diagnostic extension beyond definition-level artifact auditing.
- Audit outputs: The benchmark release includes complete findings for 102 ScienceAgentBench tasks and 50 BIXBench Verified-50 tasks, while Tables 5 and 6 summarize defects and findings.The released records include descriptions, evidence, and confidence scores.
G Ensemble Analysis and Model-Specific Patterns
Ensembling improves alignment with expert-identified BIXBench issues, while models exhibit complementary diagnostic profiles. Findings are dominated by incomplete instructions, with additional model- and evaluator-specific patterns.
- Ensemble strategies: Eleven of 24 issues achieve strong consensus from at least three models, including ambiguous filtering criteria, multi-issue tasks, and missing covariates.The overlap is substantial but imperfect, so individual models still contribute distinct detections.
- Ensemble strategies: 83.3% exact alignment (20/24) is achieved by the five-model union, versus 54.2% for the best single model.Majority vote covers 11/24 issues, while including partial matches raises coverage to 87.5% for at least three models and 95.8% for at least two.
- Model-specific patterns: INST-INCOMPLETE is the dominant subcategory across all models, with 31–78 findings per model.Secondary profiles diverge across GT-LOGIC, EVAL-STOCHASTIC, and EVAL-COVERAGE findings.
- Model-specific patterns: Only GPT-5.4 detects EVAL-JUDGE-BIAS, producing 7 findings, whereas other models flag none.This failure mode appears in BIXBench’s LLM-judge setting and is absent from deterministic script-evaluated ScienceAgentBench.
- Cost: Gemini 3.0 Flash audits all 50 tasks for $0.53 total, while Opus 4.6 costs $5.98.Gemini also has the highest partial-match recall at 95.8%, and Opus remains below $0.12 per task.
H Atomic Issue Decomposition for BIXBench
BIXBench’s expert revisions are decomposed into 24 independently fixable atomic issues, most of which concern instructions. Some logic defects co-occur with underspecified instructions or reflect factual errors in the gold program.
- Issue decomposition: 24 atomic issues are derived from 17 expert-revised questions using a split test for independent fixability.Two concerns are separate when one can be fixed while the other remains.
- Issue distribution: 87.5% (21/24) of atomic issues are instruction-level problems involving underspecified requirements, ambiguous definitions, or missing constraints.This distribution makes instruction-level defects the principal issue category in the Verified-50 subset.
- Issue distribution: GT-LOGIC defects can co-occur with instruction-level issues when an underspecified instruction leads the gold program toward one interpretation.One additional GT-LOGIC issue is a factual error in the gold program’s FAM138A categorization.
- Issue distribution: Twelve tasks contain one issue, three contain two, and two contain three independently fixable issues each.The multi-issue tasks are bix-31-q2 and bix-54-q7.
I Execution-Level Audit on BIXBench
Execution-level auditing appends agent-generated summaries to BIXBench audit contexts, but its benefit varies substantially by model. The five-model union improves only marginally, and the main BIXBench results therefore use definition-level auditing.
- Diagnostic value: Execution-level BIXBench results are mixed because the value of agent traces depends on their richness and informativeness.The main BIXBench tables use definition-level auditing only.
- Results: Execution-summary auditing raises the five-model union’s exact recall from 83.3% to 87.5%, while RecallA+P remains 95.8%.The effect is uneven across models: some improve, while Opus 4.6 and Gemini 3.1 Pro regress in exact recall.
- Results: Gemini 3.0 Flash improves exact recall from 45.8% to 66.7% while reducing total findings from 114 to 105.This is the largest reported model-level gain from appending execution summaries.
- Diagnostic value: The only newly exact-matched issue is bix-32-q2, where the execution summary exposes the agent’s concrete KEGG enrichment path.The ambiguous frequency-ratio definition in bix-54-q7 remains missed under both settings.
- Evaluation: BENCHGUARD’s alignment judge compares each finding with each gold issue and labels the relationship aligned, partial, or unrelated.The judge treats a shared root cause as aligned and uses partial for a coarser identification of the same functional problem.