PaperlayerView exact revision on arXiv

Source-linked AI summary

Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

Linhai Ma, Yan Wang, Lingfei Qian, Yupeng Cao, Victor Gutierrez Basulto

arXiv:2610.18497v1cs.AIcs.CL

TL;DR

Tool-call judges accept correct argument values less often when their input lacks a reference for them. DEREF uses a separate model to derive references from the full history, improving overall argument AUROC across three benchmarks and lifting hidden-source results from chance or below to 98.40–99.59.

  • Problem

    Tool-call judges may reject correct defaults, inferred values, or values whose source was cut because their input lacks text stating what the value should be.

    Source: §1 · p. 1

  • Method

    DEREF has a separate model read the full history without seeing the candidate and write each argument’s expected value and source for the judge.

    Source: §1 · pp. 1–2

  • Results

    Across three benchmarks, DEREF improves overall argument AUROC for every judge tested; on the hidden-source test, it lifts results from chance or below to 98.40–99.59.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Controlled tests show judges tend to follow explicit references in their input, including when a visible reference favors an incorrect value over a correct one.

    Source: §1 · p. 1

    Main results use one deriver model, Qwen3.8-27B; derivers from other model families remain unevaluated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Tool-call judges evaluate argument values proposed by LLM agents. We find that they accept a correct value less often when their input holds no reference for it, that is, no text that says what the value should be. This happens when the value is a default value from the function schema that the user did not mention, when it has to be inferred, or when the text it comes from was cut from the judge input. We call this drop in acceptance the derivation gap. It appears across all seven judges we evaluate, including System One models CLM and Jev. The gap is not only a truncation artifact: correct default values have the largest gap on a benchmark with no truncation. A controlled source-relocation test shows that judges follow the reference in their input: they favor a visible incorrect value over a correct value whose source is hidden. We introduce DeRef (Derived Reference): a separate model reads the full history without seeing the candidate and writes a reference for the judge. Across three benchmarks, DeRef improves overall argument AUROC for all judges tested. On the controlled test, it raises AUROC from chance or below to between 98.40 and 99.59.

1 Introduction

Argument judges check proposed tool-call values, but accept correct values less often when their input does not directly show how those values were derived. The paper measures this derivation gap and introduces DEREF to provide judges a reference from the full history.

1 Introduction

  • Correct values copied, defaulted, reformatted, or inferred can lack a visible source; the resulting lower acceptance rate is the derivation gap.A correct vehicle ID from an earlier tool return illustrates a history-only value that all four judges reject when its source is hidden.Source: §1 · p. 1
  • The gap appears across all seven judges, and correct defaults show the largest gap on FCRB even though nothing is cut.Scaling from 4B to 27B parameters does not remove the gap.Source: §1 · p. 1
  • Controlled tests show judges tend to accept values stated in their input and can favor a visible wrong value over a correct value with a hidden source.They do not reliably reconstruct the correct value on their own, even when given the full history.Source: §1 · p. 1
  • DEREF uses a separate model to read the full history without seeing the candidate, write each argument’s expected value and source, and provide that reference to the judge.In the vehicle-ID example, all four judges accept the value with DEREF; with Plain input, they score it below their acceptance threshold.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Across three benchmarks, DEREF improves overall argument AUROC for every judge tested and lifts hidden-source test results from chance or below to 98.40–99.59.The improvement is achieved without modifying the judges.Source: §1 · p. 2

2 Related Work

Prior work studies tool-call evaluation, reference-guided judging, and unsupported but potentially correct content. The paper distinguishes itself by analyzing judge errors according to how each argument value was derived.

2 Related Work

  • Tool-call benchmarks evaluate calls or individual arguments against gold answers, while learned judges score whole calls or arguments and guards check calls before execution (Patil et al., 2025; Zhong et al., 2025; Agarwal et al., 2025; Li et al., 2026b; Lin et al., 2026; She et al., 2026).Prior judge benchmarks report more “correct” labels and less agreement with references as tasks become harder (Li et al., 2026a; Fan et al., 2026b; Verma et al., 2026).Source: §2 · p. 2
  • Reference-guided judges often write or revise an answer before judging, but a judge-generated reference may miss an argument value in a tool return cut from its input (Zheng et al., 2023; Zeng et al., 2024; Li et al., 2026d; Zhou, 2026; Andrade et al., 2026; Li et al., 2026e; Dhuliawala et al., 2024; Raghavendra et al., 2026; Zhang et al., 2025a).Judges can also credit wrong answers without a reference or reject answers matching a wrong reference (Krumdick et al., 2025; Chalamalasetti and Vajjala, 2026; Lee et al., 2026).Source: §2 · pp. 2–3
  • Correct content without contextual support can still be factual, and tool-call arguments may be correct defaults, reformatted values, or inferences without a matching dialogue span (Cao et al., 2022; Joren et al., 2025).The authors identify derivation path as an unexamined way to divide judge errors; sources can also go unused (Modarressi et al., 2025; Kate et al., 2025).Source: §2 · p. 3

3 The Derivation Gap

The section defines argument-level judging and derivation paths, measures acceptance-rate gaps across benchmarks and judges, and uses controlled tests to examine missing information and explicit references.

3.1 Argument-Level Judging and Derivation Paths

  • A derivation path pairs a source in the history or function schema with a transformation that copies, reformats, or infers the candidate value.The five path types are VISIBLE, HISTORY-ONLY, FORMAT, DEFAULT, and NO-SOURCE; correctness is determined by the benchmark gold call, not the assigned path.Source: §3.1 · p. 3

3.2 Benchmarks and Judges

  • The evaluation uses FCRB, multi-step CFB, and financial-query FinTrace, with seven judges; four are used in controlled tests and main experiments.Judges receive a truncated Plain input that cuts tool returns to 1,000 characters; FCRB has no earlier calls to cut.Source: §3.2 · pp. 3–4

3.3 Measuring the Gap

  • The derivation gap compares acceptance rates at a judge-specific threshold, rather than AUROC within a path, so it captures path-specific false rejections.The threshold is set so the judge accepts 80% of correct arguments overall; ties and alternate 70% and 90% thresholds are also checked.Source: §3.3 · p. 4
  • On CFB and FinTrace, out-of-view gaps are significant for all seven judges; on untruncated FCRB, defaults have the largest gap, which base rates do not explain.For FinTrace, the low 45% correctness rate of no-source values versus 97.5% for in-view values may account for part of that gap.Source: §3.3 · p. 4

3.4 Judges Follow the Reference

  • Burying the correct source drops AUROC from 91.89–99.88 to 39.80–49.03 for all four judges; with only a visible distractor, it is 26.71–39.88.With full history, Jev, Qwen3-8B, and Qwen3.8-27B recover to 98.39–99.86.Source: §3.4 · p. 5
  • Giving judges the full history narrows the gap on history-only values, but does not improve no-source AUROC for any of the three tested judges.It lowers AUROC for Jev by 2.83 points and for Qwen3-8B by 13.83 points on no-source values.Source: §3.4 · p. 5
  • On 142 defaults whose DEREF reference says the default applies, schema facts raise acceptance by at most 6.18 points, while the reference raises it to 73.02–88.06.This is a reference-selected subset of the 185 correct defaults.Source: §3.4 · pp. 5–6
  • When a matching reference supports wrong values, mean scores rise for all four judges; three accept 28–49 of 59 wrong history-only arguments at the 0.5 threshold.The section concludes that judges score a value higher when an input reference matches it, whether or not it is correct.Source: §3.4 · p. 6

4 DEREF: Deriving the Reference

DEREF derives a reference from the full history and schema before the judge evaluates a candidate. The judge remains unchanged and scores the candidate with that reference in its input.

4 DEREF: Deriving the Reference

  • A deriver reads the full history and schema, without seeing the candidate, and writes the expected value and source for each argument.If the history does not fix a value, the deriver says so; all candidates at that state share the same per-argument reference.Source: §4 · p. 6
  • The reference is inserted after the history and before the function description, and the unchanged judge scores each argument as before.In the main experiments, judges keep their usual truncated input and receive the reference in addition.Source: §4 · p. 6
  • The main experiments use Qwen3.8-27B as the deriver for every judge, with thinking enabled.The deriver can name values whose sources were cut from the judge’s input because it reads the full history.Source: §4 · p. 6

5 Experiments

Across three benchmarks, DEREF improves argument AUROC for every tested judge and significantly outperforms the strongest prior method when averaged over judges. Ablations and reference swaps indicate that full-history references, rather than additional judge reasoning alone, account for key gains and can strongly steer judgments.

5.1 Setup

  • The main experiments use four judges across three benchmarks, comparing DEREF with prior evidence methods, self-generated references, and full history.Full history removes the input window and cut; self-generated references use the judge’s truncated input.Source: §5.1 · p. 6

5.2 Main Results

  • DEREF raises argument AUROC over Plain in all 12 judge–benchmark pairs by 1.44–22.14 points, and significantly exceeds RevisEval averaged over judges on every benchmark.RevisEval is the strongest prior method across the 12 pairs.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • On buried FCRB-Relocate, DEREF raises all four judges from chance or below to AUROC 98.40–99.59.On the visible version and FinTrace-HC, where judges were already accurate, DEREF makes no judge significantly worse.Source: §5.2 · p. 7

5.3 Ablation: Where the Gain Comes From

  • The largest ablation drop comes from having the deriver use the judge’s cut input; parsing only the expected value costs 1.39 points when stated and 22.37 when absent.The authors retain the judge because its use of the rest of the reference helps when the parser finds no value.Source: §5.3 · pp. 7–8

5.4 Changing the Reference Changes the Judgment

  • On CFB and FCRB, replacing the correct reference value with a same-state wrong value lowers correct-value acceptance 15.36–57.07 points and raises wrong-value acceptance 20.38–57.79 points.All 24 changes are significant; Figure 3 depicts the reference-value swap.Source: Figure 3 · p. 8, §5.4 · p. 8

5.3 Ablation: Where the Gain Comes From

  • In the four-setting comparison, reference with judge thinking off scores best at 96.01, indicating that the gain comes from the reference rather than more judge reasoning.All four settings give the judge the full history.Source: Table 4 · p. 7, §5.3 · p. 8
  • A deriver that sees the candidate names the wrong value under test up to six times as often, so DEREF keeps the candidate hidden from its deriver.DEREF-v is above DEREF on CFB but below it on FCRB and FinTrace.Source: §5.3 · p. 8

6 Conclusion

Argument judges tend to score candidates against explicit references, accepting correct values less often when their input lacks one. DEREF derives a reference from the full history before seeing the candidate, and improves overall argument AUROC across three benchmarks for all six judges tested.

  • DEREF improves overall argument AUROC for all six tested judges across three benchmarks.Source: §6 · p. 8
  • Judges accept correct values less often when their input lacks an explicit reference, including defaults, inferred values, and values whose source was cut.On the two multi-step benchmarks, the pattern also holds for values matching no span of the history.Source: §6 · p. 8
  • DEREF derives an expected-value reference from the full history before the candidate is seen.Source: §6 · p. 8

Limitations

The evaluation has limits in its labels, deriver coverage, judge coverage, and one path-specific result. These boundaries qualify how broadly its findings can be generalized.

Limitations

  • A single gold value per argument can make judges reject another value that is also correct.For CFB and FinTrace, an audit could not confirm 3% and 4% of the constructed wrong labels, respectively; every DEREF gain over Plain remained significant in the worst-case analysis. FCRB labels follow the official BFCL checker and were not independently audited.Source: Limitations · p. 9
  • On FinTrace, DEREF does not shrink the no-source gap for three of four judges.The low base rate of this path may account for part of the remaining gap.Source: Limitations · p. 9
  • Main results use Qwen3.8-27B as the only deriver, held fixed across judges and input conditions.Broader evaluation of derivers from other model families is left to future work; the Qwen3-8B ablation still improves Qwen3.8-27B's argument AUROC over Plain on CFB.Source: Limitations · p. 9
  • The judges tested are dedicated scoring models and open LLMs up to 27B parameters; general-purpose frontier LLM judges remain unevaluated.Source: Limitations · p. 9

Ethical Considerations

The study uses released benchmarks and public agent-run trajectories, without collecting new data from people or attempting to identify anyone. It also flags risks from reference errors and recommends human or independent checks for high-stakes calls.

Ethical Considerations

  • The study uses three public benchmarks and public ComplexFuncBench agent-run trajectories, with no new human data or personal information added.Two authors checked label and path samples by hand; no crowd workers were used, and Jev received only benchmark text and model outputs through an outside API.Source: Ethical Considerations · p. 9
  • A mistaken deriver or a tool return written to mislead it could make a judge accept a wrong argument.The authors therefore frame DEREF as support for review, not a replacement for it.Source: Ethical Considerations · p. 9
  • High-stakes calls should still go to a person or a check that does not rely on the judge.Keeping each derived reference with its scores lets a reviewer see why a value was accepted.Source: Ethical Considerations · p. 9
  • The benchmarks are task-oriented, and the authors did not screen every item for offensive content.They report seeing none in the items read for label and path checks and used the texts as released.Source: Ethical Considerations · p. 9

Artifact Use and Licensing

The paper identifies licenses for its benchmarks, models, and software, and says code, path labels, and two controlled sets will be released after review.

Artifact Use and Licensing

Use of AI Assistants

The authors used AI assistants for language polishing, brainstorming, and LaTeX editing; they reviewed and verified all technical claims, results, analysis, and final writing decisions.

  • AI assistants supported language polishing, brainstorming, and LaTeX editing, while authors reviewed and verified the technical work and final decisions.Source: Use of AI Assistants · p. 9

A Data and Judge Inputs

The section describes benchmark construction and judge inputs, then defines derivation paths and evaluation procedures. It also reports controlled tests of source visibility and checks of label and threshold sensitivity.

A Data and Judge Inputs

  • CFB uses model-call divergences from gold trajectories, FinTrace constructs wrong calls from gold calls, and FCRB retains checker-rejected argument values.The resulting datasets include 1,639 correct and 999 wrong CFB argument values, 7,428 correct and 2,796 wrong FinTrace values, and 1,719 correct and 769 wrong FCRB values.Source: A Data and Judge Inputs · p. 13
  • Worst-case removal of unconfirmed wrong labels leaves significant DeRef-over-Plain AUROC gains of 11.76–20.19 points on CFB and 0.89–6.42 on FinTrace.The deletion removes the 3% of CFB and 4% of FinTrace wrong values whose removal most reduces the gain.Source: A Data and Judge Inputs · p. 13
  • Judge inputs show recent history with dataset-specific truncation, while FCRB parameter lines also expose schema type, enum, and default information.Full-history inputs show earlier calls and returns without a window, subject to length limits for some FinTrace inputs.Source: A Data and Judge Inputs · pp. 13–14
  • In FCRB-Relocate, hidden correct values scored below visible distractor H, with AUROC from 26.71 to 39.88 across four judges.With full history, AUROC reached 98.39 for Qwen3-8B, 99.81 for Jev, and 99.86 for Qwen3.8-27B; the test isolates source visibility rather than showing DeRef beats full history.Source: A Data and Judge Inputs · p. 15
  • Revealing a hidden source raised correct-value scores by 6.30–14.29 points for all four judges, while wrong-value scores moved by at most 3.49 points.On revealed returns alone, the source significantly raised correct scores for all four judges.Source: A Data and Judge Inputs · p. 15

B Derivation Path Census

  • The path census distinguishes VISIBLE, HISTORY-ONLY, FORMAT, DEFAULT, and NO-SOURCE values, assigning each value to the first matching path.VISIBLE searches the cut Plain state; HISTORY-ONLY searches the uncut history; FORMAT allows specified normalization, while DEFAULT includes schema defaults.Source: B Derivation Path Census · p. 16

C Metrics and Statistics

  • The derivation-gap threshold accepts 80% of correct arguments, with tied scores fractionally counted; the gap remains robust at 70% and 90% acceptance rates.All 20 CFB and FinTrace cells retain confidence intervals above zero at both alternative rates; three of 30 FCRB cells lose significance.Source: C Metrics and Statistics · p. 16

D Reading the Reference

This section explains how the authors parse stated expected values and validate the parser. It compares full DeRef references with blocks retaining only a parsed expected value, including cases where parsing finds no value.

  • The parser extracts values from fixed reference sentence patterns, and a four-round check correctly identified the reference conclusion in all 30 sampled arguments per benchmark.A missed parser value can still be stated in the reference, so measured shares of references naming the correct value may be too low.Source: D Reading the Reference · p. 16
  • Where the parser found a value, full reference text added 1.39 AUROC points on CFB, made no significant difference on FCRB, and cost 1.13 points on FinTrace versus EV only.Source: D Reading the Reference · p. 17, Table 8 · p. 17
  • Where no value was parsed, full reference text added 22.37 points on CFB and 3.67 on FinTrace; it still contained the correct value for 77.01% and 45.03% of correct arguments, respectively.Source: D Reading the Reference · p. 17, Table 8 · p. 17

E Call-Level Results

At call level, DeRef improves call AUROC across the evaluated judges and benchmarks, while taking the minimum argument score alone can fail when correct out-of-view arguments become the lowest-scored arguments. The section also reports implementation and cost details.

E Call-Level Results

  • Taking the minimum raises call AUROC by 8.15–16.21 points on FCRB but adds nothing significant on CFB, where the derivation gap is largest on a single path.The authors note that minimum aggregation assumes the weakest argument is wrong, an assumption often broken by correct values out of view.Source: E Call-Level Results · p. 18
  • In 20 of 21 judge-benchmark pairs, an out-of-view argument is the lowest-scored argument in correct calls more often than chance; on CFB, DeRef lowers this share by 10.28–24.37 points.The share remains above chance after DeRef.Source: E Call-Level Results · p. 18, Table 10 · p. 18
  • DeRef raises call AUROC by 1.21–22.11 points for all four judges on all three benchmarks, with all 12 gains significant.On CFB, DeRef gains 11.99–22.11 points where taking the minimum adds nothing under Plain.Source: E Call-Level Results · p. 19

F Implementation Details and Cost

  • One DeRef prompt writes a reference per argument slot, shared across candidates; average candidates per prompt range from 1.33 on FCRB to 1.83 on CFB.The longest DeRef input is 408 tokens on FCRB and 2,048 on CFB and FinTrace; full-history judging averages 3,317 tokens on CFB and 6,761 on FinTrace.Source: F Implementation Details and Cost · p. 19

G Further Results

DEREF improves argument AUROC across benchmark comparisons, while further tests examine full-history inputs, alternative reference methods, and judges beyond the Qwen family. The related-work discussion distinguishes this work by its focus on derivation paths, and the case studies illustrate how visible references affect judgments.

G Further Results

  • On 106 FinTrace-HC pairs, earlier-seen wrong values score higher than unseen wrong values by 7.79 points for CLM, 5.06 for Jev, 3.79 for Qwen3-8B, and 0.72 for Qwen3.8-27B.The first three differences are significant.Source: G Further Results · p. 19
  • With full history, DEREF reaches 84.97 and 96.01 AUROC on CFB for Qwen3-8B and Qwen3.8-27B, beating full history alone by 18.04 and 10.56 points.On FinTrace, it reaches 91.94 and 92.52, gains of 7.20 and 2.75 points over full history alone.Source: G Further Results · p. 19
  • Averaged over four judges, candidate-visible DEREF-v exceeds DEREF by 2.68 points on CFB but trails by 2.57 on FCRB and 0.84 on FinTrace.DEREF-v names the wrong tested value as expected more often than DEREF on CFB and FCRB, and Qwen3.8-27B accepts more wrong values with DEREF-v on both benchmarks.Source: G Further Results · p. 19
  • DEREF improves AUROC for Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 by 8.78–20.84 points, with all six gains significant; results are also stable across three deriver seeds.Across 18 seed comparisons, none is significant and the largest difference is 0.80 points.Source: G Further Results · pp. 19, 21

H Full Comparison Grid

  • Averaged over four judges, DEREF minus Plain is 7.92 points on FCRB, 17.09 on CFB, and 4.95 on FinTrace; minus RevisEval, it is 2.38, 2.07, and 1.46 points.The benchmark comparisons are reported in Table 2.Source: H Full Comparison Grid · p. 22
  • DEREF minus ProvenanceGuard evidence is 7.48 points on FCRB, 12.31 on CFB, and 4.28 on FinTrace; against full history, the average gains are 11.58 on CFB and 4.35 on FinTrace.Every reported confidence interval for DEREF versus ProvenanceGuard evidence lies above zero.Source: H Full Comparison Grid · p. 22

I Extended Related Work

  • The related-work discussion notes that prior studies do not split judge errors by derivation path, while self-generated references improve on Plain input by at most 3.01 points and remain below DEREF.Writing the DEREF reference from cut input instead of full history costs 9.63 points on CFB.Source: I Extended Related Work · p. 26

J Qualitative Case Studies

Related papers

Loading 2610.18497v1…