PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian, Sai Harshitha Aluru

arXiv:2608.26623v1cs.AI

TL;DR

The reliability of LLM judges for structured, dependency-driven agentic tool-calling workflows remains largely unexamined. AgentJudgeBench evaluates this reliability across DAG-based tasks and finds that alignment declines with difficulty, with hard no-ground-truth queries converging to a 77–82% ceiling regardless of judge scale.

  • Problem

    The reliability of LLM judges for structured, dependency-driven agentic tool-calling workflows remains largely unexamined.

  • Method

    AgentJudgeBench evaluates six DAG topologies and three difficulty tiers under paired with-ground-truth and without-ground-truth conditions.

  • Results

    Judge alignment declines monotonically with difficulty, while hard queries without ground truth converge to a 77–82% ceiling regardless of judge capacity.

  • Takeaways & Limitations

    Frontier judge scale offers limited practical advantage without ground truth because judges converge within the shared ceiling.

  • Takeaways & Limitations

    The benchmark uses synthetic records and a deterministic scorer, so domain drift remains open and headline absolute numbers reflect the current scoring rule.

Abstract

from arXiv · show

LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.

1 Introduction

AgentJudgeBench studies LLM-judge reliability for structured, dependency-driven tool-calling workflows, where correctness depends on tool selection, arguments, ordering, and query coverage. Across controlled difficulty levels and ground-truth conditions, it reveals difficulty-driven alignment limits and practical differences among judging strategies and models.

  • Benchmark design: 3,808 records span six DAG topologies, three difficulty tiers, programmatically verified traces, five generators, six judges, and four structural metrics.The paired with-GT/without-GT protocol yields 321,648 evaluations across tool selection, parameter structure, sequence accuracy, and query coverage.
  • Empirical findings: 77–82%: all six judges converge on this alignment ceiling for hard queries without ground truth, regardless of model capacity.Alignment degrades monotonically with difficulty and 1.5× faster without ground truth; task difficulty is the primary driver for capable generators, while ceiling height is somewhat prompt-dependent for weaker ones.
  • Mitigations: Structured per-metric rubrics improve alignment by +4.8–+6.5 pp on one pairing, whereas chain-of-thought and temperature have negligible effects.The rubric effect is smaller and reverses on a second pairing, so it does not generalize uniformly across judge-generator pairs.
  • Practical guidance: With ground truth, QwQ-32B best matches the programmatic reference, while GPT-OSS-120B is most human-aligned; without it, frontier judges lead narrowly.Existing tool-calling benchmarks report aggregate pass-rate agreement without varying the structural dimensions relevant to reliability, leaving judge selection and configuration under-motivated.

2 Related Work

Prior tool-calling benchmarks primarily score agents using execution feedback or deterministic trajectory matching, while LLM-as-judge research has focused on open-ended or verifiable text evaluation. Although recent tool-calling benchmarks use LLM judges, they generally treat judging as an implementation detail rather than the object of study.

  • Tool-calling agent evaluation: Tool-calling evaluation predominantly relies on environment-based execution feedback or deterministic scoring against annotated trajectories, rather than studying judge reliability.τ-bench emphasizes simulated users and pass^k reliability instead of per-turn correctness, illustrating that this paradigm evaluates agents themselves.
  • LLM-as-judge research: The LLM-as-judge paradigm began with MT-Bench, where GPT-4 showed strong human-preference agreement but exhibited positional, verbosity, and self-enhancement biases.Later work extended evaluation to hard, verifiable response pairs or trained dedicated judge models.
  • LLM-as-judge research: Dedicated judges such as JudgeLM and Prometheus 2 target text-quality scoring, addressing biases through fine-tuning on distilled verdicts or rubric-conditioned direct assessment.JudgeLM covers 7B–33B judges and identifies position, knowledge, and format biases that mirror those of prompt-based judges.
  • Tool-calling judge deployment: Recent tool-calling benchmarks deploy LLM judges as implementation details; ToolLLM reports 87.1% aggregate human agreement, while StableToolBench and MCP-AgentBench use similar approaches.These benchmarks update the judge models but do not make judge reliability their central study subject.

3 Methodology

AgentJudgeBench uses a controlled synthetic benchmark spanning six workflow-DAG topologies and three difficulty tiers, with deterministic programmatic references and paired LLM-judge evaluations. Alignment is measured per metric and in aggregate against reproducible reference scores under with- and without-ground-truth prompts.

  • Data construction: The benchmark synthetically generates certified agentic records across six DAG topologies and three difficulty tiers to provide controlled coverage, scale, and ground-truth traces.Records formalize user requests with typed tool schemas and ordered traces containing sequential or parallel calls, inputs, outputs, and step descriptions.
  • Programmatic Judge: A deterministic programmatic judge provides the reproducible reference, scoring tool selection, parameter structure, sequence accuracy, and query coverage independently for every record.The reference avoids introducing another LLM’s reliability into evaluation and scales to the full factorial grid.
  • Record-level aggregate: The four metrics receive equal weights because they target orthogonal correctness dimensions, and the resulting judge ranking remains stable under five alternative weighting schemes with Spearman ρ ≥0.832.Practitioners can instead reweight metrics for application-specific priorities such as strict ordering.
  • LLM Judge: Each LLM judge evaluates the same generator outputs with structured prompts either exposing the ground-truth tool-call sequence or omitting it entirely.Decoding is held constant across generator, judge, difficulty, and ground-truth configurations to isolate the independent variables.
  • Alignment Against the Programmatic Reference: Alignment compares each judge’s per-metric verdict with the programmatic reference and reports both aggregate alignment and per-metric breakdowns.Perfect reproduction of the programmatic vector corresponds to align = 100%, while an independent judge attains 50% in expectation.

4 Experiments

Across 3,808 AgentJudgeBench instances, alignment declines monotonically with difficulty and reaches a 77–82% hard-query ceiling without ground truth, while ground-truth exposure, judge agreement, and mitigation effects vary substantially by judge, generator, and prompt design.

  • Ground-truth effects: GT lift is positive for QwQ-32B and GPT-OSS-120B but negative for GPT-5.4 and Gemini-2.5-Pro, because frontier judges can anchor on GT sequence ordering and penalize functionally equivalent deviations.With GT, sequence accuracy is the weakest per-metric dimension with substantial inter-judge spread; without GT, judges compress into a narrow band under the 1.0-default rubric.
  • Main results: 77–82%: On hard without-GT records, all six judges converge across four of five generators, while all 30 generator–judge pairs degrade monotonically from easy to hard and without-GT degradation is roughly 1.5× larger.The convergence indicates a task-level rather than judge-level ceiling; Table 2 reports the full factorial alignment results.
  • Judge agreement: 92.6%: Judges agree more without GT than with GT, but the increase reflects prompt-driven verdict compression; with GT, mean agreement is 79.1% and the highest pair is QwQ-32B×GPT-OSS-120B at 89.4%.The lowest pair is Claude Sonnet 4.5×GPT-OSS-20B at 70.4%; without-GT κ decreases with judge-tier separation, a relationship absent under GT.
  • Ceiling analysis: 79.5%: The six-judge ensemble matches the best individual judge, GPT-5.4 at 79.8%, within 0.4 pp, consistent with correlated structural failure rather than independent judge noise.The hard without-GT ceiling is explained by the 1.0-default prompt, peak generator error on hard records, and uniform per-metric compression.
  • Inference-time ablations: Temperature and chain-of-thought have negligible effects: Qwen3-32B varies by at most 0.6 pp across temperatures, while QwQ-32B thinking changes the mean GT gap by only +0.11 pp and at most 0.3 pp per cell.A second judge–generator pairing confirms even tighter temperature insensitivity, with a maximum spread of 0.25 pp.
  • Prompt mitigation: +4.8–+6.5 pp: Structured per-metric JSON prompts outperform free-form prompts with GT on the original pairing, but the effect shrinks or reverses across another pairing and difficulty level.On the second pairing, structured format leads by +3.9 pp on easy and +2.4 pp on medium, but trails by −0.8 pp on hard.

5 Conclusion

AgentJudgeBench measures LLM-judge reliability on structured, dependency-driven tool-calling through difficulty-stratified rewrites, DAG-topology annotations, and paired with- and without-ground-truth evaluation. Across 321,648 completed evaluations, alignment degrades monotonically with task difficulty.

  • 5 Conclusion: AgentJudgeBench evaluates LLM-judge reliability on structured, dependency-driven tool-calling, an area with limited existing calibration.The benchmark uses difficulty-stratified rewrites preserving ground-truth traces, per-record DAG-topology annotations across six patterns, and paired with-GT/without-GT evaluation.
  • 5 Conclusion: Across 321,648 completed evaluations, judge alignment degrades monotonically as task difficulty increases.

Supplementary Material

The supplementary materials provide the complete benchmark records and verdicts, the SyGra implementation with prompts, and scripts enabling reproduction of the main results.

  • The Hugging Face dataset contains all 3,808 records, generator outputs, and all seven judges’ with-GT/without-GT verdicts underlying every table and figure.
  • The SyGra code release includes data generation, difficulty rewriting, the programmatic judge, the LLM-judge runner, and full prompt templates.
  • A reproducibility script accompanies the implementation under a permissive open-source license.

Limitations

The benchmark’s conclusions are supported by open-weight replication and human validation, but remain limited by GPT-5.4’s non-reproducibility, synthetic data, scorer disagreements, prompt-dependent effects, and evaluation-time-only analysis. Domain drift and transfer to training uses remain open concerns.

  • Reproducibility: GPT-5.4 is a non-reproducible Azure snapshot whose self-bias cannot be independently re-verified, although primary claims replicate on open-weight generators.Its self-bias remains within range on other generators except sequence accuracy (+0.172 versus ≤+0.008), a post-hoc correlational signal.
  • Reference signal: The deterministic scorer agrees with a 120-record human study at 92.5–98.3% on three metrics but only 82.5% on parameter structure.The parameter-structure gap reflects schema-valid extra keys that annotators accept.
  • External validity: Synthetic records and general-purpose judges limit external validity, despite quality gates and 92.7% human agreement; domain drift remains open.Verified per-record references at enterprise scale were unavailable for the paired ground-truth/no-ground-truth protocol.
  • Robustness: Prompt-format advantages only partially replicate and reverse on hard queries, while difficulty effects shift by up to +5.6 pp for weaker generators.Temperature insensitivity replicates tightly; task difficulty is therefore a primary ceiling driver mainly for capable generators.
  • Scope: The study evaluates judge reliability only at evaluation time, leaving transfer of failure modes to reward modeling or model-selection gates for future work.The authors specifically flag ground-truth over-anchoring, without-ground-truth verdict compression, and correlated cross-judge failure under this framing.

Ethical Considerations · A Practitioner Decision Guide

The paper warns that systematic LLM-judge failures can certify incorrect tool-calling outputs and propagate shared blind spots, especially in safety-critical settings. Its practitioner guide therefore recommends complementary evaluation and scenario-specific judge selection, with judge choice least consequential on hard queries without ground truth.

  • Ethical Considerations: The 77–82% without-GT ceiling and frontier-model over-anchoring make sole reliance on automated judges unsafe, particularly for safety-critical tool-calling evaluation.Practitioners may otherwise certify incorrect outputs as correct; the paper recommends deployment guidance and avoiding reliance on any single judge.
  • Ethical Considerations: All six judges share similar hard-query failure patterns, indicating training-distribution biases that can create an evaluation monoculture across pipelines.The paper recommends complementing LLM judges with programmatic scoring and human review.
  • Ethical Considerations: Releasing prompts, scorer code, and raw outputs supports countermeasures against gaming based on over-anchoring, 1.0-default bias, and prompt-format sensitivity.The detailed failure-mode analysis could otherwise be exploited to manipulate judge-based evaluation.
  • Ethical Considerations: Synthetic records avoid real-user privacy concerns but inherit generation-pipeline biases and may under-represent non-English or marginalised workflows.The full evaluation requires 321,648 paired judge calls, motivating a decision guide that reduces unnecessary computation by recommending judges per scenario.
  • A Practitioner Decision Guide: Without ground truth, Gemini-2.5-Pro and GPT-5.4 lead narrowly by ≤1 pp, while the convergence ceiling makes judge choice ≤2 pp apart on hard queries.Without-GT evaluation is the practical baseline because ground-truth tool-call sequences are usually unavailable at inference time.
  • A Practitioner Decision Guide: With ground truth, QwQ-32B dominates all difficulty levels, while GPT-OSS-120B is the best open-weight alternative without reasoning-model inference overhead.These rankings are derived from the deployment-scenario comparison in Table 6.
  • A Practitioner Decision Guide: For one judge across conditions, QwQ-32B provides 89.1% mean GT alignment and competitive 85.6% without-GT performance.The guide presents this as the recommended single-judge compromise across ground-truth conditions.

B Extended Limitations Discussion · C Dataset Statistics

The extended limitations qualify the benchmark’s reference scorer, synthetic single-shot setting, judge coverage, difficulty calibration, and the prompt dependence of the without-ground-truth ceiling. Dataset statistics show uneven topology frequencies and deliberately over-provisioned tool inventories with varied parameter depth and call-sequence length.

  • B Extended Limitations Discussion: The deterministic programmatic scorer scales to 321,648 evaluations, but human agreement falls to 82.5% on parameter structure versus 92.5–98.3% on three other metrics.The limitation arises because schema-valid but semantically defensible parameter choices can be penalized.
  • B Extended Limitations Discussion: Tool redundancy can penalize conceptually correct substitutions; 9 of 21 human tool-selection disagreements involved equivalent tools or defensible extra calls.Corpus-wide prevalence and schema-aware tool-equivalence correction remain future work.
  • B Extended Limitations Discussion: Synthetic records and single-shot, stateless prompts limit real-trace validity, although quality gates and human validation provide 92.7% agreement.The headline findings should therefore be interpreted as comparative rather than absolute-magnitude results.
  • B Extended Limitations Discussion: Prometheus-2 generalizes poorly relative to the six general-purpose judges, showing 20–30 pp lower alignment and near-chance pairwise agreement across conditions.Its weaknesses concentrate on parameter structure and query coverage, and its ground-truth effect is negative relative to most judges.
  • B Extended Limitations Discussion: GPT-5.4 shows no aggregate self-preference, but sequence accuracy exhibits localized self-credit of +0.172 versus ≤+0.008 elsewhere; training-time judge risks remain untested.This is a post-hoc correlational analysis rather than a controlled role ablation.
  • B Extended Limitations Discussion: Temperature insensitivity replicates more tightly, whereas prompt-format gains replicate only partially and reverse on hard queries, indicating judge–generator dependence.Difficulty rewriting is unanimously validated on 93.9% of medium→hard records but only 58.1% of easy→medium records, making medium a robustness check rather than a fully independent tier.
  • B Extended Limitations Discussion: The without-ground-truth ceiling is prompt-dependent for weaker generators: the alternative rubric changes alignment by ≤+1.0 pp for the three strongest but +4.1 and +5.6 pp for Llama-3.1-8B and SmolLM3-3B.Task difficulty remains the primary driver for capable generators.
  • C Dataset Statistics: Fan-in and optional enrichment dominate the dataset at 27.5% and 21.6%, while loop-like records are rarest at 5.9%.These distributions are computed over 3,808 unique records, collapsing three difficulty variants; Table 7 reports the corresponding counts and Figure 6 visualizes them.

D Evaluation Corpus Statistics · E Bootstrap Confidence Intervals for Table 2 · F Difficulty Degradation and Generator Accuracy

The evaluation corpus contains 3,808 records expanded into three difficulty variants, with attrition primarily from unparseable generator outputs and persistent judge nulls. Table 2 uncertainty is quantified with stratified bootstrap intervals, while difficulty-dependent judge alignment and generator accuracy are summarized separately.

  • D Evaluation Corpus Statistics: 93.8% of the 57,120 possible generator-condition records received valid LLM-judge verdicts after excluding unparseable or empty tool-call sequences.The theoretical maximum is 3,808 × 3 = 11,424 generator inputs per generator, with identical judged records across the six judges within each generator–difficulty cell.
  • D Evaluation Corpus Statistics: Approximately 5.7% of inputs were unparseable, concentrated on SmolLM3-3B and Qwen3-32B and reflecting structured-output capability gaps.Attrition was −15.6% for SmolLM3-3B and −10.4% for Qwen3-32B; SmolLM3-3B’s medium-difficulty success rate was 79.8% versus 86.6% on easy and hard inputs.
  • D Evaluation Corpus Statistics: Approximately 3.3% of main-grid judge calls remained null after two retries, with persistent failures concentrated in the longest GPT-5.4 and Llama-3.3-70B outputs.These outputs approached context limits for some judge endpoints, particularly Claude Sonnet 4.5, producing truncation or malformed JSON; persistent nulls were distributed uniformly across topologies and difficulty levels.
  • E Bootstrap Confidence Intervals for Table 2: Table 2 alignment percentages are means over per-record scores, with 95% confidence intervals computed from 2,000 stratified bootstrap resamples for every evaluation cell.Across all five generators, the bootstrap confidence-interval half-widths were at most 0.3 percentage points.
  • F Difficulty Degradation and Generator Accuracy: Figure 8 summarizes mean judge alignment as query difficulty changes, averaging across all six judges and five generators.The supplied passage identifies the difficulty-degradation visualization but does not provide its numerical values.
  • F Difficulty Degradation and Generator Accuracy: Table 11 reports generator programmatic accuracy on AgentJudgeBench, averaged across four metrics and six DAG topologies.The supplied passage identifies the accuracy measure and aggregation scope but does not provide the numerical values.

G Programmatic Judge Validation

Human validation supports the programmatic scorer as a reliable reference overall, but reveals metric-specific and topology-specific disagreement with annotators. These discrepancies change human-aligned judge rankings, while alternative metric weightings and a generous rescoring correction leave the programmatic ranking unchanged.

  • Human validation: 92.7% agreement across 480 metric-level verdicts validates the programmatic scorer overall, with parameter structure weakest at 82.5% versus 97.5% sequence accuracy and 98.3% query coverage.Optional_enrichment had the lowest per-DAG agreement at 85.0%, reflecting ambiguity in optional branches; parameter disagreements mainly involved schema-valid extra keys.
  • Human-aligned judge validation: GPT-OSS-120B is most human-aligned, with a −2.8 pp human-versus-programmatic gap, while all judges are closer to the programmatic scorer and Gemini-2.5-Pro diverges most at −10.2 pp.Humans were more lenient on hard examples, especially sequence accuracy and parameter structure, changing judge rankings relative to programmatic-reference alignment.
  • Ranking robustness: The full six-judge programmatic ranking is unchanged under all tested metric weightings, and QwQ-32B shifts by at most 0.2 pp when parameter structure is excluded.Replacing the programmatic reference with human verdicts changes QwQ-32B from first to fourth, isolating disagreement with the scorer as the ranking-relevant issue.
  • Scorer correction bound: Every judge moves by at most 0.09 pp under the maximally generous parameter-structure correction, while the ranking remains QwQ-32B > GPT-OSS-120B > Claude Sonnet 4.5 > GPT-5.4 > GPT-OSS-20B > Gemini-2.5-Pro.Because the correction credits all potentially valid extra keys, it bounds the effect of a future schema-aware fix from above.

H Over-Anchoring Case Studies · I Worked Example

The case studies show that ground-truth exposure can cause judges to overlook missing steps, unresolved dependencies, or extra calls by anchoring evaluation to the reference. A worked example contrasts this failure mode with a genuine ambiguity in sequence scoring, where semantic and strict structural interpretations diverge.

  • H Over-Anchoring Case Studies: Across 1,322 hard-difficulty Gemini-2.5-Pro over-anchoring instances, three cases show ground truth causing judges to overlook structural flaws penalized by the programmatic judge.Cases 1–2 involve sequence-accuracy anchoring, while Case 3 involves coverage anchoring; with-GT justifications reference the ground-truth trace, whereas without-GT justifications identify the structural flaw.
  • Case 1: Sequence Anchor, Microbial-Growth Linear DAG: Ground-truth exposure made the judge assess only presented steps against the reference, overlooking a missing shelf-life call despite seq = 0.67.Without ground truth, the judge identified a different redundancy critique and withheld full credit.
  • Case 2: Sequence Anchor, Cloud-Budget Linear DAG: Ground-truth anchoring suppressed dependency checking, causing the judge to treat a cloud-budget plan with seq = 0.33 as independently executable despite an unresolved input.Without ground truth, the judge correctly identified that predicted_trend was unavailable from the query or prior calls.
  • Case 3: Coverage Anchor, Power-Grid Linear DAG: Ground-truth presence triggered credit-by-association, causing the judge to credit extra-call coverage despite cov = 0.67 and overlook the missing synthesis step.Without ground truth, the judge independently found only partial coverage.
  • I Worked Example: The worked example is not a judge failure: QwQ-32B with ground truth scored tool = 1.0, param = 1.0, seq = 1.0, and cov = 1.0.The programmatic judge instead assigned seq = 0.33 because the generator swapped independent sibling calls and emitted the fan-in call before resolution.
  • I Worked Example: The worked example’s disagreement reduced alignment to 83.3%, solely because sequence scores differed while the other three metrics matched.Both judges accepted the same coherent two-then-one dependency structure, despite different ordering of the independent first calls.
  • I Worked Example: The divergence reflects a genuine structural-versus-semantic ambiguity in fan-in topologies, not a scorer bug, because sibling branches may be interchangeable under different sequence definitions.The programmatic metric uses position-by-position matching, whereas the LLM judge evaluates the dependency structure semantically.

J C3 Corrupted-GT Control: Full Results … M Generator Models

The C3 control reveals judge-specific effects of corrupted ground truth, while AgentJudgeBench uniquely covers four evaluation dimensions and uses generators spanning four capability tiers. The paper also provides a notation summary for these analyses.

  • J C3 Corrupted-GT Control: Full Results: Under corrupted GT, Gemini-2.5-Pro stays within 0–1.8 pp of its standard-GT alignment across all difficulties on Llama-3.3-70B.The C3 condition randomly samples ground truth from a different record with the same DAG topology.
  • J C3 Corrupted-GT Control: Full Results: QwQ-32B’s corrupted-GT alignment matches its without-GT alignment on the Llama-3.3-70B generator.This is the other side of the judge-level split reported for the C3 control.
  • K Notation: Table 16 summarizes the notation symbols used throughout the paper in their order of appearance.This notation reference supports interpretation of the merged sections’ analyses and results.
  • L Judge LLMs Comparison: AgentJudgeBench is the first system to combine per-metric decomposition, difficulty variation, ground-truth ablation, and a deterministic programmatic reference for agentic tool-calling evaluation.Table 17 compares it with seven existing systems, none of which addresses more than one dimension.
  • M Generator Models: The five generators span small 3B, mid-scale 8B, large 32B–70B, and frontier GPT-5.4 capability tiers.All generators use temperature-0 decoding with the model’s native function-calling prompt.
  • M Generator Models: The generator set is designed to cover a representative range of tool-calling output quality for judge evaluation.The listed models include both open-source and frontier closed models.

N Rewriting-Preservation Validation Study … X Free-Form Judge Prompt (A5)

Across validation, robustness, ablation, and prompt analyses, the study finds that rewrite preservation is strong, judge agreement is structurally constrained, and evaluation outcomes depend materially on prompt format and verdict encoding. Detailed prompt specifications and supplementary analyses clarify these effects while identifying limits on claims about hardening, stochasticity, and GPT-5.4 self-preference.

  • N Rewriting-Preservation Validation Study: +20.7 pp and +27.3 pp are the largest increases in unanimous agreement after dropping GPT-5.4, indicating that its conservative interpretation lowers the published three-judge rates rather than inflating them.GPT-5.4 is non-reproducible and serves elsewhere as generator and judge; the robustness check retains Claude Sonnet 4.5 and Gemini-2.5-Pro.
  • O GPT-5.4 Self-Preference Check: +0.172 self-bias appears only for GPT-5.4 sequence accuracy, while aggregate self-bias is +0.012 and falls within its cross-generator range, so aggregate self-preference is unsupported but a metric-localized effect remains possible.The study tests GPT-5.4-as-judge on its own generations against four other generators using the existing 321,648-evaluation grid.
  • P LLM Judge Configurations / W Prompts / X Free-Form Judge Prompt (A5): The evaluation uses six judges spanning 20B-120B open models, QwQ-32B with thinking enabled, and frontier closed models, with GT prompts supplying expected calls and without-GT prompts relying only on query, tool schema, and generated calls.The prompts enforce static structural evaluation, four JSON-scored metrics, and strict output schemas; Prometheus-2 instead requires four independent metric calls using its official absolute-grading template (Kim et al., 2024).
  • S Per-Topology Breakdown: Under with-GT evaluation, QwQ-32B leads on five of six DAG topologies, GPT-OSS-120B leads fan_out at 93.9%, and fan_in is about 10 pp harder than fan_out.Gemini-2.5-Pro trails QwQ-32B by 5-8 pp, while fan-out benefits from independently verifiable parallel branches.
  • T Inter-Judge Confusion Matrices: 94.7% and 96.8% of off-diagonal disagreements involve a 0.5 verdict, while without GT Claude Sonnet 4.5×GPT-OSS-20B agreement rises from 70.3% to 90.1% as both judges default to 1.0.The highest-agreement with-GT pair is QwQ-32B×GPT-OSS-120B with κ = 0.606, whereas the lowest is Claude Sonnet 4.5×GPT-OSS-20B with κ = 0.225.
  • Binary vs. {0, 0.5, 1} Verdict Scale: +11.9 pp and +13.0 pp are the binary-remapping gains for GPT-5.4 and GPT-OSS-20B, respectively, reshuffling the ranking from QwQ-32B > GPT-OSS-120B to GPT-OSS-20B > GPT-5.4.Because every 0.5 verdict is mapped to the programmatic direction, judges with higher 0.5 rates receive disproportionately inflated binary alignment.
  • Prompt variation sensitivity: ≤0.6 pp temperature variance contrasts with a 4.8–6.5 pp drop from structured per-metric rubrics to free-form prompts, showing prompt format matters more than measured sampling variation on the original pairing.On QwQ-32B with SmolLM3-3B, the format effect is +3.9 pp on easy, +2.4 pp on medium, and −0.8 pp on hard, so its magnitude and direction depend on judge, generator, and difficulty.
Loading 2608.26623v1…