Source-linked AI summary
Noise Floor Audit for Agent Benchmarks
Yihang Chen, Pin Qian, Su Wang, Chong Peng, Huan Xu, Xiyang Wu, Yiqi Sun
TL;DR
Function-calling benchmark scores may obscure measurement variability and failure character. This audit uses matched AST grading across three native endpoints, finding nearly deterministic frozen reruns but much larger sensitivity to semantics-preserving prompt perturbations. It also shows that malformed-output shares vary substantially, so marginal accuracy alone is incomplete.
Problem
Benchmark headline scores can hide rerun variability, prompt sensitivity, and qualitatively different failure modes.
Method
The audit evaluates 3 native tool-calling endpoints across 2 providers on official BFCL categories using matched AST grading and repeated fixed-temperature runs.
Results
Semantics-preserving prompt perturbations produce a larger measurement floor than frozen reruns, with median paired SDs 11x to 58x larger; malformed-output shares are 30%, 7%, and <1%.
Takeaways & Limitations
Paired perturbation analysis and failure taxonomy provide more information than marginal accuracy alone for these benchmark comparisons.
Takeaways & Limitations
The estimates are conditional on the frozen study set and observed endpoint behavior, and Gemini’s provider and thinking configuration effects cannot be separated.
Abstract
from arXiv · showhide
We audit measurement variability for 3 native tool-calling endpoints across 2 providers on the official BFCL multiple and parallel categories, using matched AST grading. At temperature 0, reruns are nearly deterministic across Groq endpoints and a thinking-enabled Gemini setting: ever-flip fractions are 0.7%, 2.0%, and 2.7%, with mean run correlations of 0.997, 0.966, and 0.961. Semantics-preserving prompt perturbations create the larger floor on all endpoints, with median perturbation paired SDs 11x to 58x larger than rerun paired SDs. The failure character also shifts: malformed-output failures account for 30%, 7%, and <1% of task failures, so marginal accuracy hides not only stability but also failure mode.
1 Introduction
This audit examines whether function-calling benchmark scores are stable under frozen reruns and semantics-preserving prompt changes. It finds small rerun noise, substantially larger perturbation variability, and failure differences hidden by marginal accuracy.
- Single benchmark scores can hide rerun noise, prompt sensitivity, and qualitatively different failure modes in a compound tool-calling measurement pipeline.The pipeline includes endpoint output, provider-specific parsing, grading, and aggregation over matched instances.
- 0.7%, 2.0%, and 2.7% are the ever-flip fractions across the 3 endpoints, with mean run correlations of 0.997, 0.966, and 0.961.
- 11x to 58x larger median perturbation paired SDs show that semantics-preserving prompt changes create the larger measurement floor.The comparison is based on matched paired analysis.
- Opposite stability rankings across the two Groq sizes show that endpoint capability does not consistently predict measurement stability.The Gemini comparison confounds provider and thinking configuration rather than extending a clean scale axis.
- 30%, 7%, and <1% of task failures are malformed-output failures across the endpoints.The failure character shifts from structural failures toward well-formed-but-wrong calls.
2 Related Work
Prior work establishes that aggregate language-model evaluations require uncertainty reporting, reproducibility controls, and robustness analysis. This audit applies those concerns to tool-calling outcomes and distinguishes measurement sensitivity from prompting or adversarial-robustness goals.
- Evaluation research warns that final scores can hide instability and motivates paired comparisons and explicit uncertainty estimates.
- Reproducible language-model evaluation depends on endpoint versioning, decoding controls, retry policy, parser behavior, grader implementation, and raw-output retention.This audit holds the suite, prompt template, grader, and matched instances fixed.
- Prompt-robustness studies show that semantically equivalent or formatting-only changes can alter accuracy, absolute scores, and relative model rankings.
- This audit estimates measurement sensitivity under declared semantics-preserving prompt perturbations rather than optimizing prompts or measuring adversarial robustness.
- Adjacent agent-benchmarking work makes reliability visible through fine-grained analyses, pass-k consistency, and category-specific outcomes rather than one final score.The audit instead measures score-scale movement under frozen reruns and matched prompt perturbations.
3 Experimental Setup
The audit evaluates three native endpoints on a frozen, matched BFCL subset using AST exactness, repeated temperature-0 runs, and semantics-preserving prompt variants. Its paired design isolates rerun and prompt-surface movement from instance composition.
- 150 matched instances from the official BFCL AST-graded multiple and parallel categories form the frozen study set.The instance-id list is hash-frozen before reported runs, with no excluded instances or unrecoverable units.
- AST exactness requires the predicted function-name multiset and each expected argument to match the official ground truth.Parse failures and malformed payloads count as incorrect unless an unrecoverable instance is excluded under the paired-design policy.
- The endpoints are llama-3.1-8b-instant, llama-3.3-70b-versatile, and gemini-3.5-flash, with Gemini calls pinned to low thinkingLevel.Retained main-arm Gemini calls average 189 thinking tokens, while served identities are recorded and guarded against drift.
- N = 10 temperature-0 reruns cover each endpoint, while the perturbation arm uses N = 5 reruns and 4 semantics-preserving prompt variants.The variants change user-message surface form while retaining the schema, target function set, and ground-truth call.
- Paired-difference SD is computed from matched instance-level score differences and is the relevant uncertainty quantity for leaderboard gaps.The observed paired-to-√2-marginal ratios are 0.95, 0.91, and 1.00.
- Freezing the instance set removes instance-mix variance and makes perturbation comparisons within-instance robustness tests.Temperature 0 is treated as a conservative deployment measurement target, while higher temperatures add another randomness source.
4 Results
At frozen decoding, reruns remain close to deterministic, while semantics-preserving prompt perturbations produce substantially larger variability across endpoints. Stability rankings also differ by variability source, and failure taxonomy reveals shifts that aggregate scores conceal.
- Rerun near-determinism: 0.997, 0.966, and 0.961 mean pairwise run correlations accompany ever-flip fractions of 0.7%, 2.0%, and 2.7% across the three endpoints.Rerun paired-difference SDs are 0.28pp, 0.91pp, and 1.1pp, respectively.
- Prompt perturbation floor: 58x, 11x, and 16x are the approximate median perturbation-to-rerun paired-SD ratios for llama-3.1-8b-instant, llama-3.3-70b-versatile, and gemini-3.5-flash.Perturbation paired SDs span 9.6pp–19pp, 5.2pp–13pp, and 16pp–21pp across the endpoints.
- Bootstrap sample-size curve: 0.26pp, 1.1pp, and 1.3pp are the full-size bootstrap means for rerun paired SD across the three endpoints.Bootstrap intervals widen substantially at the low-instance end, making small matched subsets less reliable for estimating this floor.
- Capability and stability: 10pp, 16pp, and 19pp are the median perturbation paired SDs for llama-3.3-70b-versatile, llama-3.1-8b-instant, and gemini-3.5-flash.The Groq endpoints reverse rank between rerun stability and perturbation stability, so capability does not consistently predict stability here.
- Failure character: 30%, 7%, and <1% of task failures are malformed-output failures for llama-3.1-8b-instant, llama-3.3-70b-versatile, and gemini-3.5-flash.The taxonomy distinguishes structural failures from well-formed but wrong calls, including wrong arguments or call counts.
5 Discussion
The audit finds that frozen-decoding reruns add little measurement noise, while semantics-preserving prompt perturbations create a substantially larger floor and reveal distinct failure modes. These results make paired perturbation analysis and failure taxonomy more informative than marginal accuracy alone.
- 0.28pp, 0.91pp, and 1.1pp paired SDs show that frozen-decoding reruns add little precision relative to their cost.Ever-flip fractions are 0.7%, 2.0%, and 2.7%.
- 16pp, 10pp, and 19pp median perturbation paired SDs make prompt-template robustness the more visible uncertainty source.The audit reports these perturbation floors for the three endpoints.
- Malformed calls indicate interface reliability and decoding constraints, whereas well-formed but wrong arguments indicate semantic tool-use behavior.The failure taxonomy separates structural failures from semantic tool-use errors.
- Matched perturbation matrices and failure taxonomies are more informative than a single marginal accuracy for score differences on this suite.Marginal accuracy hides rerun stability, the perturbation floor, and failure character.
- Capability does not consistently predict stability: the two Groq endpoints rank oppositely for rerun and perturbation stability.The audit does not explain this reversal, and Gemini confounds provider with thinking configuration.
6 Limitations
The audit is limited to a frozen BFCL native-tool-calling snapshot and three endpoints, so its floors are conditional rather than general estimates of benchmark or deployment variability. Its descriptive AST-based analysis also cannot identify causal drivers or evaluate broader agent behavior.
- The benchmark covers only BFCL native tool calling on AST-graded multiple and parallel categories, excluding executable, retrieval, irrelevance, and multi-turn trajectories.It is a scope audit for one harness snapshot, not a stable public leaderboard.
- Three endpoints from two providers are too few to estimate provider-level variance, and endpoint identifiers are time-bound API products.Version-drift protection cannot ensure that future provider deployments reproduce the same floor.
- The perturbation results are conditional on a declared prompt family rather than an estimate over all possible prompts or deployment distributions.The matched design removes instance-mix variance but limits generalization.
- The descriptive analysis cannot identify whether training choices, routing policies, decoding constraints, or provider-side validators caused the observed patterns.The audit treats the provider endpoint as the measurement object and omits weights, decoder internals, log probabilities, controlled seeds, and serving-stack ablations.
A Diagnostic Tables
The diagnostic tables distinguish bootstrap rerun-SD estimates from plug-in estimates and distinguish instance-level paired-difference SDs from aggregate-score paired SDs. Their aggregation levels and estimators should not be conflated.
- Bootstrap rerun paired-SD means over 25 to 150 matched instances can differ from Table 2 plug-in estimates because SD is nonlinear.The full matched-size plug-in values are 0.28pp, 0.91pp, and 1.1pp.
- Within- and across-fingerprint SD columns report instance-level paired differences, not aggregate-score paired SDs, so their magnitudes are not directly comparable.The distinction is between per-pair, instance-level aggregation and aggregate-score aggregation.