AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

1 Introduction

• LLM judges lack systematic calibration for agentic tool-calling, where tool selection, parameter structure, sequence accuracy, and query coverage can fail independently.Without ground-truth traces—the common deployment scenario—judges must reconstruct correctness from queries and tool schemas. Existing benchmarks report only aggregate pass-rate agreement without varying these dimensions (Qin et al., 2024; Guo et al., 2024, 2025).Source: §1 · p. 1
• 3,808 BFCL-style records span six DAG topologies and three controlled difficulty tiers, each with a programmatically verified ground-truth trace.Five generators and six judges yield 321,648 paired with-GT/without-GT evaluations using four structural metrics and bootstrap confidence intervals. The contribution is a combined reliability protocol for evaluating judges rather than agent capability.Source: §1 · p. 2
• 77–82% alignment is the shared ceiling for all six judges on hard queries without ground truth, regardless of model capacity.Alignment degrades monotonically with difficulty, 1.5× faster without ground truth. Follow-up ablation identifies difficulty as the primary driver for capable generators, although ceiling height is somewhat prompt-dependent for weaker generators.Source: §1 · p. 2
• +4.8–+6.5 pp alignment gains from structured per-metric rubrics exceed other tested interventions on one pairing, but do not generalize uniformly.A second judge-generator pairing shows smaller gains that reverse on hard queries. Chain-of-thought adds at most 0.3 pp across 24 paired comparisons, while temperature produces ≤0.25 pp spread across two pairings.Source: §1 · p. 2
• Ground-truth exposure reduces alignment for GPT5.4 and Gemini-2.5-Pro, consistent with over-anchoring rather than independent judgment.Inter-judge agreement is moderate (κ ≈0.42 with GT), with systematic, capacity-correlated patterns not visible in aggregate scores.Source: §1 · p. 2
• QwQ-32B best matches the programmatic reference with ground truth, while GPT-OSS-120B is most human-aligned; without ground truth, frontier judges lead narrowly.The human-alignment finding comes from a validation study; the shared convergence ceiling limits practical differences between judges without ground truth.Source: §1 · pp. 2–3

2.1 Agentic Data and Tool-Calling Benchmarks

• FuncBenchGen (Maekawa et al., 2025) shares controllable DAG workflows but evaluates generators; AgentJudgeBench evaluates judges through paired ground-truth conditions and per-metric decomposition.TaskBench (Shen et al., 2024) likewise uses tool-dependency graphs but targets decomposition quality. Synthetic pipelines (Wang et al., 2023b; Xu et al., 2025b; Cui et al., 2024) inform data generation but target model training.Source: §2.1 · p. 3
• AgentJudgeBench adopts BFCL’s typed JSON schema while extending single-turn function calling to multi-step DAG workflows across three difficulty tiers.Existing agent benchmarks predominantly use execution feedback or deterministic trajectory scoring. τ-bench (Yao et al., 2024) emphasizes pass^k reliability rather than per-turn correctness, but evaluates agents rather than their judges.Source: §2.1 · p. 3

2.2 LLM-as-Judge

• AgentJudgeBench evaluates structural correctness with verdicts {0, 0.5, 1} and interdependent failure modes, unlike text-centric preference evaluation.Zheng et al. (2023) established strong GPT-4 agreement with human dialogue preferences while identifying positional, verbosity, and self-enhancement biases. JudgeBench (Tan et al., 2025) and Arena-Hard-Auto (Li et al., 2024b) extend evaluation to hard, verifiable response pairs.Source: §2.2 · p. 3
• Dedicated text-quality judges inform AgentJudgeBench’s bias analysis, while judge ensembles offer a separate strategy for reducing individual biases.JudgeLM (Zhu et al., 2023) distills GPT-4 verdicts and identifies position, knowledge, and format biases; Prometheus 2 (Kim et al., 2024) provides open-weight, rubric-conditioned assessment. These taxonomies inform the prompt-format ablation, while Verga et al. (2024) demonstrate bias reduction through ensembling.Source: §2.2 · p. 3

2.3 LLM Judges for Tool-Calling

• 321,648 completed evaluations underpin AgentJudgeBench’s systematic decomposition of judge reliability by metric, topology, difficulty, and ground-truth condition.Earlier tool-calling studies report aggregate reliability on small samples without varying complexity, difficulty, or ground-truth availability. ToolSandbox (Lu et al., 2024) instead addresses judge unreliability by replacing LLM judges with programmatic evaluation.Source: §2.3 · p. 3
• 87.1% aggregate human agreement in ToolLLM (Qin et al., 2024) and 88–96% agreement in GeoBenchX (Krechetova and Kochedykov, 2025) exemplify prior aggregate reliability reporting.StableToolBench (Guo et al., 2024) and MCP-AgentBench (Guo et al., 2025) similarly deploy LLM judges. Agent-as-a-Judge (Zhuge et al., 2024) and Auto-Eval Judge (Bhonsle et al., 2025) introduce more structured frameworks, but do not systematically vary the conditions examined by AgentJudgeBench.Source: §2.3 · p. 3

3.1 Data Generation

• 3,808 synthetic records span 15 enterprise domains, six dependency topologies, and 8–19 tools per record, providing verified reference traces and independent topology–difficulty control.Construction generates scenarios, typed tool schemas, dependency-linked executable pseudocode, user queries, and ordered traces recording inputs and outputs. Topologies comprise linear, fan-out, fan-in, diamond, optional enrichment, and loop-like workflows.Source: §3.1 · p. 4
• 93.9% of medium→hard rewrites validate as harder, versus 58.1% unanimous validation for easy→medium, making medium a robustness check rather than a calibrated midpoint.Rewrites increase query ambiguity while holding task structure and GT traces constant. Roughly 41.9% of medium variants are better characterized as paraphrases, so primary difficulty-degradation evidence comes from medium→hard comparisons.Source: §3.1 · p. 4
• 92.7% metric-level agreement with independent human judgment validates the scorer on 120 hard records stratified across all six topologies.Two non-LLM quality gates check schemas, argument types, trace consistency, argument sufficiency, grounding, and query naturalness, regenerating failures. The deterministic reference supplies reproducible scoring across 321,648 evaluations without relying on another LLM or an unscalable human-in-the-loop reference.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 Evaluation

• Four independent scores in [0, 1] measure tool selection, parameter structure, sequence accuracy, and query coverage.Tool selection penalizes set mismatch, while sequence accuracy checks positional identity normalized by expected length. Parameter structure compares argument-key sets, and query coverage measures the fraction of expected tool identifiers recovered, ignoring extras.Source: §3.2 · p. 5
• Spearman ρ ≥0.832 between equal-weight and alternative metric-weighting rankings indicates that judge rankings are relatively stable under reweighting.Generator accuracy uses an unweighted four-metric mean, while other analyses retain separate components. Equal weights reflect the metrics’ orthogonal roles and the absence of a principled preference among them.Source: §3.2 · pp. 5–6
• Paired with- and without-GT prompts evaluate identical generator outputs, measuring LLM judges’ agreement with a deterministic reference under different reference-access conditions.Judges receive the query, tool schemas, and predicted calls, returning four JSON scores on {0, 0.5, 1} with justifications and an overall assessment. Only with-GT prompts expose the reference trace; decoding settings remain constant.Source: §3.2 · pp. 4, 6
• 100% alignment denotes perfect reproduction of the programmatic score vector, while an independent judge attains 50% in expectation.Alignment is reported both in aggregate for each judge–generator–difficulty–condition configuration and separately by metric to identify difficult correctness dimensions.Source: §3.2 · p. 6

4.1 Experimental Setup

• 321,648 valid tuples cover 90 factorial cells spanning five generators, six judges, three difficulty tiers, and paired with- and without-GT conditions.The study reports alignment and GT lift; Prometheus-2 (Kim et al., 2024) is an additional judge-specialized baseline excluded from six-judge statistics.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Results and Analysis

• 77–82% alignment on hard queries without ground truth bounds all six judges across four of five generators, including GPT-5.4.All 30 generator–judge pairs degrade strictly from easy to hard under both conditions, with without-GT degradation roughly 1.5× larger. The medium-to-hard drop (≈5–7 pp with GT) is the primary difficulty evidence: hard rewrites receive 93.9% unanimous validation, versus 58.1% for easy-to-medium, which partly measures paraphrase robustness.Source: §4.2 · pp. 6–7, Table 2 · p. 7
• 10 of 15 generator–difficulty cells favor QwQ-32B with GT, but it never leads without GT, making judge selection configuration-dependent.GT lift is positive for QwQ-32B and GPT-OSS-120B but negative for GPT-5.4 and Gemini-2.5-Pro; sequence-order anchoring penalizes functionally equivalent alternatives. Corrupted GT leaves Gemini-2.5-Pro unchanged and QwQ-32B within 0.2 pp of without-GT alignment; without GT, Gemini-2.5-Pro leads on stronger generators and GPT-5.4 on weaker ones.Source: Table 2 · p. 7, §4.2 · p. 7, Table 1 · p. 7
• 79.1% mean inter-judge agreement with GT (κ = 0.419) rises to 92.6% without GT (κ = 0.559), reflecting verdict compression rather than genuine consensus.Without GT, κ declines with judge-tier separation (ρ = −0.8253, p < 0.01), unlike with GT (ρ = −0.171); 94–97% of off-diagonal disagreements concern the 0.5 partial-credit boundary. Prometheus-2 remains near chance (42–52% exact agreement, κ = 0.01–0.07), while a six-judge soft jury only matches the top individual on with-GT hard records (82.5%) (Verga et al., 2024).Source: §4.2 · pp. 7–8
• Sequence accuracy is the weakest with-GT metric, while DAG difficulty follows a judge-independent ordering: fan-out easiest and loop-like/fan-in hardest.Sequence accuracy shows substantial inter-judge spread with GT; without GT, judges compress into a narrow per-metric band reflecting the 1.0-default rubric.Source: §4.2 · p. 7, Figure 3 · p. 8
• +4.1–+5.6 pp alignment gains follow recalibrating hard without-GT prompts from a 1.0 default to 0.5 for weaker generators.Gains are ≤+1.0 pp for the three strongest generators; the ceiling reflects default-driven verdict compression, peak hard-record generator error, and uniform per-metric compression, while the 1.0 default over-credits weaker generators’ incorrect outputs. The six-judge ensemble reaches 79.5% versus GPT-5.4’s 79.8%, supporting correlated structural failure rather than independent judge noise.Source: Table 4 · p. 8, §4.2 · p. 8
• +4.8–+6.5 pp with-GT alignment gains favor structured per-metric JSON over free-form prompting for Qwen3-32B on Llama-3.3-70B.For QwQ-32B on SmolLM3-3B, structured prompting gains +3.9 pp on easy and +2.4 pp on medium, but reverses to −0.8 pp on hard. This largest tested configuration lever therefore does not generalize uniformly across judge–generator pairs or difficulty levels.Source: Figure 5 · p. 9, §4.2 · p. 9
• +0.11 pp mean GT alignment difference from enabling QwQ-32B thinking indicates negligible chain-of-thought benefit.Across 24 paired cells spanning four open-weight generators, no difference exceeds 0.3 pp under either condition; the authors attribute QwQ-32B’s advantage to training distribution rather than inference-time compute. Temperature changes across T ∈ {0.3, 0.7, 1.0} produce maximum spreads of 0.6 pp for Qwen3-32B on Llama-3.3-70B and 0.25 pp for GPT-OSS-120B on Llama-3.1-8B-Instruct.Source: §4.2 · pp. 8–9, Table 5 · p. 9, Figure 4 · p. 9

5 Conclusion

• 77–82% alignment on hard tasks without ground truth forms a shared ceiling regardless of judge capacity.The C2 recalibrated-prompt ablation supports this task-level ceiling, improving alignment by ≤+1.0 pp for capable generators.Source: §5 · p. 9
• 1.5× faster alignment degradation without ground truth accompanies a monotonic decline with increasing difficulty.The comparison is against alignment degradation with ground truth.Source: §5 · p. 9
• QwQ-32B best matches the programmatic reference with ground truth, while GPT-OSS-120B best matches human judgement; without ground truth, frontier judges lead only narrowly.The GPT-OSS-120B human-judgement comparison also applies to evaluation with ground truth.Source: §5 · p. 9
• −3.9 pp for Gemini-2.5-Pro and −1.5 pp for GPT-5.4 show that ground-truth exposure reduces frontier-judge alignment, consistent with over-anchoring.Source: §5 · p. 9
• +4.8–+6.5 pp gains make prompt structure the largest configuration lever, but benefits depend on the judge/generator pair; CoT reasoning and temperature have negligible effects.Source: §5 · p. 9
• AgentJudgeBench introduces difficulty-stratified rewrites, six-pattern DAG-topology annotations, and paired with-ground-truth/without-ground-truth evaluation for structured, dependency-driven tool-calling.Easy, medium, and hard rewrites preserve the ground-truth trace. The study reports 321,648 completed evaluations.Source: §5 · p. 9

• 3,808 records, generator outputs, and all seven judges’ with-GT/without-GT verdicts underlying every table and figure are available in the Hugging Face dataset.Source: Supplementary Material · pp. 9–10
• The SyGra code release includes data generation, difficulty rewriting, and the programmatic judge under a permissive open-source license.The programmatic judge implements Eqs. 1a-5a.Source: Supplementary Material · p. 10
• The code release also provides the LLM-judge runner with full prompt templates from Appendix W and a reproducibility script for the main results.Source: Supplementary Material · p. 10

Limitations

• GPT-5.4’s non-reproducible Azure snapshot limits independent verification, although all primary claims replicate independently on the open-weight generators.GPT-5.4 also serves as judge and rewrite meta-judge. Its generator self-bias remains within other-generator ranges except for sequence accuracy (+0.172 versus ≤+0.008), an unresolved post-hoc correlational signal.Source: Limitations · p. 10
• 92.5–98.3% human agreement supports three programmatic-scoring metrics, but parameter-structure agreement reaches only 82.5%, qualifying absolute scores.The 120-record, single-annotator study exposes penalties for schema-valid extra keys and analogous tool redundancy in 9 of 21 tool-selection disagreements. The deterministic reference scales to 321,648 evaluations; maximally generous corrections shift rankings ≤0.09 pp, but best-judge claims report both programmatic and human-verdict rankings.Source: Limitations · p. 10
• Synthetic records and exclusively general-purpose main judges constrain benchmark scope despite pipeline validation and a judge-specialised baseline.Synthetic data enables verified per-record references unavailable at scale from proprietary systems; quality gates and 92.7% human agreement validate the pipeline, but domain drift remains unresolved. Prometheus-2 (Kim et al., 2024) has 20–30 pp lower alignment and is excluded from six-judge statistics.Source: Limitations · p. 10
• ≤+1.0 pp without-GT ceiling shifts for the three strongest generators contrast with +4.1–+5.6 pp for the two weakest under alternative prompting.The alternative uses a 0.5-default prompt. Task difficulty is therefore the primary ceiling driver only for capable generators.Source: Limitations · p. 10
• Temperature insensitivity replicates on a second judge–generator pairing, but prompt-format advantages replicate only partially and reverse on hard queries.The extended RQ4/RQ6 ablations support treating format effectiveness as judge–generator-dependent rather than uniformly dominant.Source: Limitations · p. 10
• 58.1% unanimous validation for easy→medium rewrites, versus 93.9% for medium→hard, makes medium a robustness check rather than a fully calibrated tier.These validation rates qualify the interpretation of difficulty rewriting.Source: Limitations · p. 10
• Evaluation-time findings do not establish whether judge failures transfer to training signals, reward models, or model-selection gates.Future work must test ground-truth-exposure over-anchoring, without-GT verdict compression, and correlated cross-judge failures in those settings.Source: Limitations · p. 10

Ethical Considerations

• 77–82% without-GT ceilings and frontier-model over-anchoring make sole-judge certification risky, particularly for safety-critical tool-calling outputs.All six judges exhibit similar hard-query failure patterns, suggesting shared training-distribution biases and a risk of evaluation monoculture. Complement LLM judging with programmatic scoring and human review.Source: Ethical Considerations · p. 10
• Failure-mode disclosures could enable gaming through over-anchoring, 1.0-default bias, and prompt-format sensitivity; released prompts, scorer code, and raw outputs support countermeasures.Source: Ethical Considerations · pp. 10–11
• Synthetic records avoid real-user privacy concerns but inherit generation-pipeline biases and may under-represent non-English or marginalised workflows.Source: Ethical Considerations · p. 11
• 321,648 paired judge calls make the full evaluation compute-intensive; scenario-specific judge recommendations aim to reduce unnecessary evaluation.Source: Ethical Considerations · p. 11

A Practitioner Decision Guide

• 89.1% mean GT alignment makes QwQ-32B the recommended single judge across conditions, alongside competitive 85.6% without-GT performance and dominance across GT difficulty levels.With ground truth available, GPT-OSS-120B is the best open-weight alternative without reasoning-model inference overhead.Source: Table 6 · p. 13, A Practitioner Decision Guide · p. 13
• ≤1 pp separates the leading without-GT judges, Gemini-2.5-Pro and GPT-5.4, while any two judges differ by ≤2 pp on hard queries.Without-GT judging is the practical baseline because inference-time ground truth is usually unavailable. Rankings average alignment equally across five generators; count-weighted averaging shifts results by ≤0.3 pp.Source: A Practitioner Decision Guide · pp. 12–13, Table 6 · p. 13
• ρ = 0.03 between binary and three-point judge rankings accompanies the recommendation to prefer GPT-OSS-20B over QwQ-32B for binary pass/fail pipelines.Binary collapse remaps every 0.5 verdict toward the programmatic direction, inflating alignment for judges with frequent 0.5 verdicts: GPT-OSS-20B (26.5%) and GPT-5.4 (28.2%).Source: A Practitioner Decision Guide · p. 13
• GPT-OSS-120B is recommended when agreement with human annotators matters most, topping both programmatic and human-verdict orderings.The reported difference versus human verdicts is ∆ = −2.8 pp.Source: A Practitioner Decision Guide · p. 13

B Extended Limitations Discussion

• 82.5% human agreement on parameter structure, versus 92.5–98.3% on three other metrics, qualifies the programmatic reference used for judge rankings.The 120-record study used one annotator, so its 92.7% agreement and QwQ-32B 1st→4th ranking reversal are not consensus results; excluding parameter structure changes aggregate rankings by ≤0.2 pp, while an upper-bound correction changes alignment by ≤0.09 pp without changing rankings. The scorer also penalizes equivalent tool substitutions or defensible extra calls—9 of 21 tool-selection disagreements—with corpus-wide prevalence and schema-aware correction left untested.Source: B Extended Limitations Discussion · p. 13
• 20–30 pp lower alignment separates judge-specialized Prometheus-2 (Kim et al., 2024) from general-purpose judges across both ground-truth conditions.Across all five generators, Prometheus-2 shows near-chance pairwise agreement, weaknesses concentrated in parameter structure and query coverage, and negative GT lift; its published reference-free human-judgment comparison addresses a different axis. It is excluded from six-judge statistics and does not alter headline findings, while broader judge-specialized coverage remains future work.Source: B Extended Limitations Discussion · p. 14
• +0.172 sequence-accuracy self-credit for GPT-5.4, versus ≤+0.008 on other generators, is a metric-localized exception despite no aggregate self-preference.GPT-5.4 serves simultaneously as generator, judge, and rewrite meta-judge; the self-bias analysis is post-hoc and correlational rather than a controlled role ablation. Its Azure snapshot lacks a fixed version string, preventing independent re-verification of this signal.Source: B Extended Limitations Discussion · p. 14
• 58.1% easy→medium and 93.9% medium→hard unanimous rewrite validation make medium a robustness check rather than a fully calibrated independent tier.Medium→hard supplies the primary difficulty-degradation evidence. Records are synthetic and limited to single-shot, stateless planning: quality gates and 92.7% human agreement support the pipeline, but comparative findings remain exposed to deployment-domain drift, and real-trace validation awaits corpus access.Source: B Extended Limitations Discussion · pp. 13–14
• +4.1 and +5.6 pp without-GT ceiling shifts for Llama-3.1-8B and SmolLM3-3B reveal prompt dependence, versus ≤+1.0 pp for the three strongest generators.These shifts replace the default-to-1.0 rubric with a 0.5-default; difficulty remains the primary driver for capable generators, and count-weighting instead of unweighted topology averaging shifts headline figures by ≤1.5 pp without changing rankings. Second-pairing ablations replicate temperature insensitivity, but prompt-format gains replicate only partially and reverse on hard queries, leaving full-grid generalization untested.Source: B Extended Limitations Discussion · p. 14
• Evaluation-time findings do not establish training-time effects, but motivate testing reference over-anchoring, compressed rewards, and correlated judge failures.Reference over-anchoring would concern reward pipelines exposing reference traces, whereas without-GT verdict compression could supply near-constant rewards on hard queries and weaken corrective gradients. The evaluated six-judge ensemble only matches the best individual judge, suggesting correlated failures would persist under ensembling; controlled training runs are still needed.Source: B Extended Limitations Discussion · p. 14

C Dataset Statistics

• 3,808 unique records yield 11,424 difficulty-variant rows, with intentionally unequal topology frequencies reflecting patterns in 15 enterprise seed domains.Fan-in accounts for 27.5%, optional enrichment 21.6%, linear workflows 21.3%, and loop-like workflows 5.9%; linear records anchor the difficulty gradient without requiring dependency tracking. Structural statistics collapse the three difficulty variants because they share tool schemas and ground-truth traces.Source: Table 7 · p. 15, C Dataset Statistics · p. 15
• 12.9 available tools and 3.3 expected calls per record, on average, create deliberate over-provisioning that requires genuine tool selection.Inventories range from 8–19 tools and traces from 2–5 calls, using about 25% of available tools on average. Tools have 0–19 typed parameters (mean 2.4, median 2), including no-argument sentinels and complex configuration tools, collectively exercising tool selection, parameter structure, sequence accuracy, and query coverage.Source: C Dataset Statistics · p. 15

D Evaluation Corpus Statistics

• 93.8% overall success corresponded to 53,608 of 57,120 inputs, with attrition occurring at generator-output and judge-verdict stages.Table 9 reports valid-verdict record counts by generator and difficulty tier.Source: Table 9 · p. 16, D Evaluation Corpus Statistics · p. 16
• 3,808 unique records yield 11,424 possible inputs per generator through three difficulty variants, with identical judged-record sets across all six judges.Unparseable or empty generator tool-call sequences were excluded; multiplying Table 9 totals by six judges and two conditions gives evaluation-instance counts.Source: Table 9 · p. 16, D Evaluation Corpus Statistics · p. 16
• ≈5.7% of inputs produced unparseable generator outputs, the dominant attrition source, concentrated in SmolLM3-3B (−15.6%) and Qwen3-32B (−10.4%).SmolLM3-3B's medium-difficulty rate was 79.8%, versus 86.6% on easy/hard. Regeneration was not pursued because these failures were attributed to structured-output capability gaps expected to recur.Source: D Evaluation Corpus Statistics · p. 16
• ≈3.3% of main-grid judge calls remained null after two retries, excluding approximately 10,498 generator–judge–difficulty–record tuples.Nulls concentrated on long GPT-5.4 and Llama-3.3-70B outputs approaching judge context limits, particularly Claude Sonnet 4.5, causing truncation or malformed JSON. Further retries were deemed cost-prohibitive after <5% recovery; uniform distribution across topologies and difficulty levels suggested no systematic exclusion bias.Source: D Evaluation Corpus Statistics · p. 16

E Bootstrap Confidence Intervals for Table 2

• ≤0.3 pp half-widths characterized 95% bootstrap confidence intervals across all five generators, supporting statistically robust monotone difficulty effects.Each generator–judge–difficulty–condition cell used 2,000 stratified resamples. All six ground-truth-lift confidence intervals were non-overlapping across the positive-versus-negative divide.Source: E Bootstrap Confidence Intervals for… · p. 16

F Difficulty Degradation and Generator Accuracy

• Figure 8 summarizes alignment by difficulty across six judges and five generators; Table 11 summarizes generator programmatic accuracy.Generator accuracy is averaged across four metrics and six DAG topologies; the supplied passage provides no plotted or tabulated values.Source: F Difficulty Degradation and Generator… · p. 16

Validation design and limitations

• 120 hard records provide validation against one calibrated annotator per record, not a consensus reference; no inter-annotator agreement statistic was collected.Each topology contributes 20 records: 14 with at least one imperfect programmatic score and six perfect-score records. Annotators saw queries, schemas, generated and expected calls, and programmatic scores, but not paper hypotheses or LLM judge outputs.Source: G Programmatic Judge Validation · pp. 16–17

Programmatic scorer validation

• 92.7% overall human agreement across 480 metric-level verdicts supports the programmatic scorer as a reference signal, although agreement varies by metric.Sequence accuracy and query coverage achieve 97.5% and 98.3% agreement, respectively.Source: Table 12 · p. 17, G Programmatic Judge Validation · pp. 17–18

Scorer disagreement mechanisms

• 82.5% agreement makes parameter structure the weakest metric, with 21 disagreements exposing schema-versus-ground-truth mismatches and partial-match rounding errors.Approximately 16 disagreements involve schema-valid extra keys penalized by the scorer but accepted by humans; the remaining five involve structurally matching parameters under-penalized through rounding. Nevertheless, 74.1% of programmatic-versus-LLM metric-level mismatches concern tool selection, sequence accuracy, and coverage, where structural and semantic correctness largely coincide.Source: G Programmatic Judge Validation · p. 18

Tool-selection and topology ambiguity

• 85.0% agreement makes optional_enrichment the lowest-agreement topology, reflecting ambiguity over optional branches; tool-selection disagreements also expose semantic equivalence and reasonable extra tools.The nine tool-selection disagreements involve differently named but semantically equivalent tools, or reasonable extra tools penalized as outside the reference.Source: G Programmatic Judge Validation · p. 18

Judge alignment against human verdicts

• −6.9 pp is the mean human-minus-programmatic alignment gap across all six judges; GPT-OSS-120B is most human-aligned on the 120-record with-GT subset.GPT-OSS-120B has the smallest gap (−2.8 pp), while Gemini-2.5-Pro has the largest (−10.2 pp); ranking correlation is weak (Spearman ρ = 0.26), with QwQ-32B falling from first to fourth. Humans are more lenient, particularly on sequence accuracy and parameter structure, whereas judges and the scorer favor stricter structural matching.Source: Table 13 · p. 18, G Programmatic Judge Validation · p. 18

Parameter-structure weighting

• 0.2 pp is QwQ-32B’s maximum score shift when parameter structure is excluded; all six judges retain their rankings across three metric weightings.The full 321,648-evaluation with-GT grid was tested with equal weighting, parameter structure halved, and parameter structure excluded. This preserves rankings within the programmatic reference, unlike replacing that reference with human verdicts.Source: G Programmatic Judge Validation · pp. 18–19, Table 14 · p. 19

Bounding the Eq. 2 rescoring fix

• 0.09 pp bounds every judge’s with-GT alignment change under the maximally generous extra-key correction, with the full ranking unchanged.Across 53,619 records, extra-key-only patterns occur in 0.83% of 186,654 generated tool calls; crediting every such case raises scorer parameter-structure accuracy by 0.49 pp (90.5% → 91.0%). Judge verdicts remain fixed; a schema-aware correction remains future work, with its ranking-relevant risk bounded below 0.1 pp.Source: Table 15 · p. 19, G Programmatic Judge Validation · p. 19

H Over-Anchoring Case Studies

• 1,322 over-anchoring instances on Gemini-2.5-Pro hard-difficulty outputs underpin three representative cases illustrating dominant sequence anchoring and secondary coverage anchoring.The mechanism is demonstrated only for Gemini-2.5-Pro; GPT-5.4’s −1.5 pp overall GT lift does not independently verify the same mechanism.Source: H Over-Anchoring Case Studies · p. 19

Case 1: Sequence Anchor, Microbial-Growth Linear DAG

• 0.67 is the programmatic seq score in the microbial-growth case, where GT exposure makes the judge accept a two-call prefix despite a missing third call.GT exposure shifts attention from completeness to matching presented steps. Without GT, the judge still misses the omission, inventing a redundancy critique but withholding full credit.Source: Case 1: Sequence Anchor,… · p. 20

Case 2: Sequence Anchor, Cloud-Budget Linear DAG

• 0.33 is the programmatic seq score in the cloud-budget case, where GT anchoring suppresses detection of an unresolved predicted_trend dependency.With GT, the judge rationalizes the tools as independent; without GT, it correctly identifies that the first call requires an input unavailable from the query or prior steps.Source: Case 2: Sequence Anchor, Cloud-Budget… · p. 20

Case 3: Coverage Anchor, Power-Grid Linear DAG

• 0.67 is the programmatic cov score in the power-grid case, where an extra call is penalized but GT exposure induces coverage credit by association.With GT, the judge credits coverage of the query’s numerical data rather than independently evaluating the prediction. Without GT, it identifies partial coverage because a final synthesis tool is missing.Source: Case 3: Coverage Anchor, Power-Grid… · p. 20

I Worked Example

• 1.0 for tool, param, and cov is shared by both scorers in the worked example, despite symbolic placeholders in the generator’s fan-in plan.The final call is emitted before its inputs resolve, but param checks required argument names rather than placeholder values. QwQ-32B returns identical scores with and without GT.Source: I Worked Example · p. 21
• 83.3% alignment in the worked example reflects genuine sequencing ambiguity: the judge accepts interchangeable fan-in branches, while the programmatic scorer enforces exact reference ordering.The programmatic scorer assigns seq = 0.33 versus QwQ-32B’s with-GT seq = 1.0. Both readings are defensible rather than constituting a judge failure or scorer bug.Source: I Worked Example · p. 21

J C3 Corrupted-GT Control: Full Results

• 0-1.8 pp separates Gemini-2.5-Pro’s corrupted-GT and standard-GT alignment across all difficulties on Llama-3.3-70B outputs.The result supports the interpretation that Gemini anchors to any reference block regardless of its content.Source: J C3 Corrupted-GT Control: Full Results · p. 21
• 0.2 pp bounds the difference between QwQ-32B’s corrupted-GT and without-GT alignment across all difficulties on Llama-3.3-70B outputs.This supports independent reasoning when the reference is incoherent.Source: J C3 Corrupted-GT Control: Full Results · p. 21
• The C3 control replaces each judge’s reference with randomly sampled ground truth from a different record sharing the same DAG topology.The comparison includes standard-GT, without-GT, and corrupted-GT conditions for both judges on the Llama-3.3-70B generator.Source: J C3 Corrupted-GT Control: Full Results · p. 21

K Notation

• Table 16 catalogues the paper’s symbols in their order of appearance.Source: K Notation · p. 21

L Judge LLMs Comparison

• AgentJudgeBench provides all four comparison dimensions, while none of seven existing agentic tool-calling evaluation systems addresses more than one.The dimensions are per-metric decomposition, difficulty variation, ground-truth ablation, and a deterministic programmatic reference.Source: L Judge LLMs Comparison · p. 21

M Generator Models

• Five generators span four capability tiers: small open-source (3B), mid-scale open (8B), large open (32B–70B), and frontier closed (GPT-5.4).All use temperature 0 and their native function-calling prompts, covering a range of output quality.Source: M Generator Models · p. 21

N Rewriting-Preservation Validation Study

• +20.7 pp for hard-task preservation and +27.3 pp for medium hardening are unanimity increases after excluding GPT-5.4, the most conservative meta-judge.Across 198 triplets, meta-judges assessed preservation and hardening without generator predictions: preservation rates ranged from 77.8% to 100%, no preservation question received unanimous rejection, and hard-over-medium hardening was unanimous. Medium-over-easy hardening had under 60% unanimity, suggesting some rewrites are paraphrases and degradation curves partly conflate paraphrase robustness with difficulty robustness.Source: N Rewriting-Preservation Validation… · pp. 23–24, Table 22 · p. 24

O GPT-5.4 Self-Preference Check

• GPT-5.4’s aggregate self-bias of +0.012 is not exceptional, but it over-credits its own sequence accuracy by +0.172, versus at most +0.008 for any other generator under the same judge.Its aggregate bias falls within its other-generator range of −0.024 to +0.034; control judge GPT-OSS-120B likewise shows no unusual treatment of GPT-5.4 outputs (+0.072 versus its cross-generator range of +0.050 to +0.097). Isolating a possible sequence-specific mechanism requires a controlled ablation swapping GPT-5.4 out of one role at a time.Source: O GPT-5.4 Self-Preference Check · p. 24

P LLM Judge Configurations; Judge Temperature: Per-Difficulty Results; Chain-of-Thought Reasoning: Full Per-Generator Grid

• Maximum temperature-induced alignment spreads were ≤0.6 pp for Qwen3-32B on Llama-3.3-70B and 0.25 pp for GPT-OSS-120B on Llama-3.1-8B-Instruct; QwQ-32B thinking-on/off differences were ≤0.3 pp.Qwen3-32B's bound held on every difficulty/condition slice, GPT-OSS-120B's result measured with-GT alignment, and QwQ-32B's pattern was uniform across generator quality tiers. The vLLM-hosted judges QwQ-32B, GPT-OSS-20B, and GPT-OSS-120B used temperature 0.15; frontier judges used recommended provider defaults ≤1.0 because their APIs did not support temperature 0 at evaluation time.Source: Judge Temperature: Per-Difficulty… · p. 25, Chain-of-Thought Reasoning: Full… · p. 25, P LLM Judge Configurations · p. 25

Q Per-Metric Breakdown; S Per-Topology Breakdown

• 5 of 6 DAG topologies favor QwQ-32B with ground truth; metric-level differences between judges largely disappear without ground truth.GPT-OSS-120B leads fan_out at 93.9%; topology alignment spans roughly 93% for fan_out to 83% for fan_in, whose cross-branch dependencies complicate verification. With ground truth, sequence alignment is 64.2% for Gemini-2.5-Pro versus 86.4% for GPT-OSS-20B, tool selection is 70.7% for GPT-OSS-20B versus 88.7% for GPT-OSS-120B, and parameter structure/query coverage reach 86–94%; without ground truth, metric spreads are at most 3.0 pp, while Prometheus-2 particularly struggles with structure and coverage.Source: Table 24 · p. 25, S Per-Topology Breakdown · pp. 25–26, Q Per-Metric Breakdown · p. 25

Binary vs. {0, 0.5, 1} Verdict Scale; T Inter-Judge Confusion Matrices

• ρ = 0.03 between standard and binary judge rankings shows that collapsing partial credit substantially changes comparisons rather than uniformly improving alignment.Reference-directed remapping makes every 0.5 verdict correct, inflating GPT-5.4 by 11.9 pp and GPT-OSS-20B by 13.0 pp and replacing the leading QwQ-32B/GPT-OSS-120B pair with GPT-OSS-20B/GPT-5.4. Partial-credit verdicts account for 94.7% and 96.8% of disagreements in the highest- and lowest-agreement pairs; without ground truth, Claude Sonnet 4.5/GPT-OSS-20B agreement rises from 70.3% to 90.1% as both default toward 1.0.Source: T Inter-Judge Confusion Matrices · p. 26, Table 29 · p. 27, Binary vs. {0, 0.5, 1} Verdict Scale · p. 27

Repeated-run variability; V Related Work Survey

• Direct repeated-run validation remains planned; current stability claims extrapolate from temperature sensitivity rather than measured per-judge verdict-flip rates.The authors infer configuration-mean variation ≪0.1 pp from a ≤0.6 pp temperature spread across 3,771 records. Planned duplicate evaluations on a stratified subset would distinguish stochastic verdict flips from systematic inter-judge disagreement.Source: V Related Work Survey · pp. 27–28, Repeated-run variability · p. 27

Prompt variation sensitivity

• 4.8–6.5 pp alignment gains favor structured prompts on the original pairing, but a second pairing reverses the advantage on hard queries.For QwQ-32B judging SmolLM3-3B, structured-minus-free-form gains are +3.9 pp on easy, +2.4 pp on medium, and −0.8 pp on hard. Prompt format therefore depends on judge, generator, and difficulty; smaller wording changes also shift alignment by 1–5 pp.Source: Prompt variation sensitivity · p. 27, Table 31 · p. 29

W Prompts; W.2 Judge prompt: with ground truth (GT condition).

• The judge rubric evaluates static tool-call structure rather than execution or parameter values, using four metrics and structured JSON output.The metrics cover tool selection, parameter structure, sequence/dependencies, and query coverage; scores default to 1.0 absent clear structural violations, and symbolic values are permitted. The appendix reproduces generator, reference-based, reference-free, and Prometheus-2 prompts; reference-free judging omits the expected-tool-calls block while decoding settings remain fixed.Source: W Prompts · pp. 28–29, §W.2 · pp. 29–30

W.2 Judge prompt: with ground truth (GT condition).

• The ground-truth prompt supplies expected calls alongside available tools and generated calls, directing judges to evaluate structure rather than semantics.It instructs conservative scoring, preferring 1.0 unless a clear structural violation exists.Source: §W.2 · p. 30

W.3 Judge prompt: without ground truth (without-GT condition).

• Without ground truth, judges evaluate single-shot, stateless plans using only the query, tool schema, and generated calls, excluding execution correctness.Available tools are the sole source of truth; judges must not penalize placeholders, derived-value mismatches, or missing runtime-dependent values.Source: §W.3 · p. 30
• 1.0 is the default without-ground-truth score unless clear structural violations affect tool selection, parameter structure, sequence and dependencies, or query coverage.Parameter structure is prioritized: judges check argument dictionaries and schema-compatible names, not values or types; optional parameters are unpenalized, and symbolic dependencies are assumed valid.Source: §W.3 · p. 30

W.4 Judge prompt: Prometheus-2 (judge-specialised baseline).

• Four independent calls per record adapt Prometheus-2 (Kim et al., 2024) to four-metric evaluation, merging its single-metric verdicts into the shared judge-response schema.The implementation follows the official absolute-grading template and system message, changing only the score rubric between calls.Source: §W.4 · p. 30

X Free-Form Judge Prompt (A5)

• The A5 free-form ablation removes explicit metric definitions, scoring rubrics, and anchoring instructions while requesting holistic verdicts for the same four metrics.Its JSON output retains an accuracy between 0 and 1 and a one-sentence justification per metric, plus an overall assessment.Source: X Free-Form Judge Prompt (A5) · p. 31