Source-linked AI summary
Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation
Vishvesh Bhat, Jay Vaghasiya, Muhammad Ahmed Mohsin, Asad Aali
TL;DR
Tool-calling benchmark scores are increasingly used as proxies for agent capability, although evaluator validity and reproducibility require direct assessment. We audit four tool-calling benchmarks by collecting execution traces and evaluator outputs, comparing official labels with expert judgments, and analyzing disagreement causes. Among 496 audited tasks, 92 official labels disagreed with expert judgments, yielding an aggregate misalignment rate of 18.5%.
Problem
Tool-calling benchmark scores are increasingly used as proxies for agent capability, although evaluator validity and reproducibility require direct assessment.
Source: §1 · p. 1Method
We audit four tool-calling benchmarks by collecting execution traces and evaluator outputs, comparing official labels with expert judgments, and analyzing disagreement causes.
Source: §3.2 · p. 3, §4.1 · p. 6Results
Among 496 audited tasks, 92 official labels disagreed with expert judgments, yielding an aggregate misalignment rate of 18.5%.
Source: §4.2 · p. 7Takeaways & Limitations
Current benchmark scores can reflect evaluator properties rather than agent capability, supporting evaluation that is auditable, reproducible, and aligned with human judgments.
Source: §1 · p. 2, §5 · p. 9Tool-Veritas results use different task distributions and model configurations from the audited benchmarks, so they support evaluator-design evidence rather than a controlled benchmark-level ranking.
Source: §4.5 · p. 9
Abstract
from arXiv · showhide
Tool-calling benchmarks are increasingly used to rank language-model agents, yet their scores are often treated as ground truth without validating the evaluators themselves. We present a systematic validity and reproducibility audit of four major tool-calling benchmark families: BFCL v4, {\tau}2-Bench, LiveMCPBench, and MCP-Atlas. Across 496 expert-reviewed benchmark tasks, we find 92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate. The failures are not isolated annotation mistakes: deterministic benchmarks exhibit brittle state matching, trajectory lock-in, incorrect ground truths, substring-based communication failures, and reward-basis misalignment, while LLM-judge benchmarks exhibit rubric drift, hallucinated completion, answer-only scoring, and substantial run-to-run variance. In LiveMCPBench, 23 repeated evaluations of the same setup produce scores ranging from 57.9% to 76.8%, a spread of 18.9 percentage points, large enough to change leaderboard conclusions. These results show that current tool-calling scores can reflect evaluator artifacts rather than agent capability. We introduce a unified taxonomy of tool-calling evaluation failures, release trace-level audit artifacts and corrected evaluation components, and argue for decomposed metrics that separately measure tool invocation, task completion, and outcome verification. Our findings suggest that progress in tool-using agents requires benchmarks whose evaluators are themselves reproducible, auditable, and aligned with human judgments of task success. We further introduce Tool-Veritas, a configurable benchmark that combines deterministic state verification with optional qualitative judging, and Harness Lab, an open-source system for benchmark execution, trace inspection, repeated-run comparison, and evaluator debugging.
1 Introduction
Tool-calling benchmarks are widely used to measure agent capability and guide deployment, but that role assumes their evaluators reliably capture task success. A four-benchmark audit finds substantial evaluator-human disagreement and failures across deterministic and LLM-judge systems.
1 Introduction
- Benchmark-driven evaluation is increasingly used to measure progress, compare models, and guide deployment, implicitly treating scores as reliable measures of tool-calling capability.The benchmarks cited include BFCL, τ-bench, τ 2-Bench, LiveMCPBench, and MCP-Atlas.Source: §1 · p. 1
- A trace-level audit of 496 executions across BFCL v4, τ 2-Bench, LiveMCPBench, and MCP-Atlas identified 92 evaluator–human disagreements, alongside deterministic and LLM-judge evaluation failures.Deterministic benchmarks suffer from brittle state comparisons, exact-match constraints, incorrect ground-truth annotations, trajectory lock-in, and reward formulations that fail to capture task completion. LLM-judge benchmarks exhibit stochastic rubric generation, judge variance, hallucinated evaluations, and implementation-paper mismatches that introduce substantial score instability.Source: §1 · p. 2
- Repeated evaluations can change benchmark scores without changing the model, while false negatives and false positives mark successful tasks as failures and incomplete tasks as passes.The default LiveMCPBench pipeline produces score swings approaching twenty percentage points on reruns.Source: §1 · p. 2
- The paper contributes a failure taxonomy, decomposed measures of invocation, completion, and verification, Tool-Veritas, and Harness Lab for execution and debugging.Harness Lab preserves raw artifacts, supports repeated-run comparison and selective retry, and records human adjudications.Source: §1 · p. 2
2 Related Work
Prior tool-calling benchmarks standardized API selection and stateful interaction, but many scoring approaches leave user-intent satisfaction or verified tool use under-specified. Harness Lab addresses an infrastructure gap in cross-benchmark inspection and debugging.
Benchmark execution and debugging infrastructure
- Existing benchmark releases offer limited cross-benchmark trace inspection, repeated-run comparison, selective retry, and persistent human adjudication; Harness Lab supplies a shared execution and debugging layer.It preserves each benchmark’s original evaluator and raw artifacts.Source: §2 · p. 3
3 Methodology
The methodology treats evaluator agreement with expert trace-level judgments as the validity target, then audits four benchmarks through trace collection, manual adjudication, and failure analysis. It distinguishes false positives from false negatives and organizes disagreements into a benchmark-agnostic taxonomy.
3.1 Preliminaries
- A tool-calling benchmark is modeled as tasks, an evaluation harness, and a benchmark-specific success criterion; its evaluator verdict is assumed to match expert assessment of task success.The audit objective is to identify and characterize cases where evaluator and expert judgments diverge.Source: §3.1 · p. 3
3.2 Benchmark Audit Framework
- The audit collects complete execution traces and evaluator metadata from BFCL, τ 2-Bench, LiveMCPBench, and MCP-Atlas, reproducing evaluations before manual adjudication and root-cause analysis.The collected material includes tool invocation logs, environment states, evaluator outputs, and benchmark verdicts.Source: §3.2 · p. 3
3.3 Human Adjudication
- Human adjudicators inspect trajectories, tool execution, state transitions, and user-visible outcomes, labeling success by whether the user objective was achieved.Disagreements are divided into false negatives and false positives according to the benchmark label versus human assessment.Source: §3.3 · p. 3
3.4 Failure Taxonomy
- Trace-level analysis classifies deterministic failures such as exact-match constraints, state over-specification, trajectory lock-in, annotation errors, and reward misalignment, alongside judge failures including rubric drift and variance.The taxonomy also includes hallucinated completion and implementation-specification mismatches, and is intended to characterize failures independently of a particular benchmark implementation.Source: §3.4 · pp. 3–4