Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation

1 Introduction

• Citation-like artifacts can be superficial wrappers or evidence of traceability, support-chain completeness, and evidence use, so their relevance depends on the evaluation target.For content-equivalent candidates, strict debiasing should produce Tie; here Tie suppresses a spurious preference rather than expressing indecision.Source: §1 · p. 1
• On human-validated moderate quality gaps, stricter anti-presentation rubrics can turn a preference for the better answer into Tie, even without clear-correctness collapse.The paper distinguishes correct equivalence Tie, potentially calibrated soft-boundary Tie, and costly resolution-destroying Tie on moderate gaps.Source: §1 · p. 2
• TraceJudgeBench audits citation-like artifacts and debiasing side effects in RAG and agent-workflow judging.The benchmark tests whether protocols preserve decision resolution after suppressing citation-like presentation cues.Source: §1 · p. 2
• The central contribution is to treat debiasing as a measurement intervention and audit whether it preserves quality-sensitive decision resolution.The paper reports the central bias–resolution frontier across commercial and open-weight judges, with the full diagnostic matrix for three commercial judges.Source: §1 · p. 2

2 Related Work

• Earlier judge research documents position, verbosity, self-enhancement, and output-format biases; debiased evaluators also address superficial correlates such as length (Zheng et al., 2023; Dubois et al., 2024).The paper positions its question as distinct: whether protocols retain quality-sensitive resolution after bias mitigation.Source: §2 · p. 2
• RAG evaluation treats evidence use as a quality target, while citation-like artifacts can be confounders or task-relevant signals depending on whether evaluation targets correctness or traceability.The paper adopts controlled comparisons that vary presentation while holding or validating substantive quality.Source: §2 · pp. 2–3
• TraceJudgeBench evaluates Tie against each pair’s target relation rather than treating abstention as uniformly calibrated or uniformly erroneous.This connects benchmark separability with the possibility that abstention is appropriate for ambiguous inputs but costly on validated quality gaps (Yauney et al., 2026; Zhou et al., 2026; Noh et al., 2026).Source: §2 · p. 3

3 Problem Setup and Metrics

• Each ordered pair receives one of three labels—A, B, or Tie—and ordering is varied so position effects can be measured.The focus is on citation-like artifacts that may be presentational or coexist with real differences in support-chain completeness, reasoning specificity, or evidence coverage.Source: §3 · pp. 3–4
• The three target relations are content-equivalent pairs, clear correctness conflicts, and graded quality conflicts; their desired judgments are respectively Tie, the correct candidate, and preserved resolution on moderate quality gaps.Graded pairs share the correct final answer but differ in support-chain completeness or reasoning specificity.Source: §3 · p. 4
• The audit reports cited and plain/base win rates and Tie on equivalence pairs, correctness outcomes on conflicts, and better-plain wins, worse-cited wins, and Tie on moderate graded pairs.It also reports order gaps and protocol cost, interprets Tie by tier, and contrasts one-call judging with two-call TRACE-style judging.Source: §3 · p. 4

4 TraceJudgeBench

• The benchmark isolates measurement questions including presentation sensitivity, naturalistic validity, artifact triggers, correctness retention, quality resolution, order effects, and protocol cost.Its diagnostic components, sizes, target relations, and judge coverage are listed in Table 1.Source: §4 · p. 4, Table 1 · p. 5
• Mechanical HotpotQA and FinQA pairs hold substantive content constant while adding citation-like presentation; 1,000 natural HotpotQA pairs complement these controlled examples.Six ablations separate full citation, empty shell, source-only, evidence-only, sentence-id-only, and length-control variants.Source: §4 · p. 4
• Correctness conflicts make the plain candidate correct and cited candidate wrong, with reverse conflicts swapping correctness; position-only controls isolate generic position bias.The graded tiers compare complete versus weakened support or reasoning while keeping final answers correct and matched.Source: §4 · pp. 4–5
• Human validation supports the moderate-gap tier: both annotators label 91/100 HotpotQA pairs better-plain with 100.0% raw agreement, while only 1/100 soft pairs are labeled better-plain by both.The 50-pair FinQA tier has agreement on final-answer equivalence for 50/50 pairs and on reasoning completeness and preference for 49/50, with one adjudicated disagreement.Source: §4 · p. 5, Table 2 · p. 6
• The prompt ladder runs from plain preference through strict tie-required, treated as a stress-test endpoint; TRACE normalizes away presentation while preserving quality-relevant signals before masked judging.TRACE is a two-call audit baseline, not a deployment-ready judge; the full matrix covers three commercial judges, while open-weight runs focus on central audits.Source: §4 · pp. 5–6

5.1 Citation-Like Display Creates Spurious Preference

• 50.5%: GPT-5.5 selects the worse-cited candidate under plain judging on soft graded pairs; on HotpotQA mechanical-equivalence pairs, cited-candidate selection reaches 85.8%.Full-citation and source-only variants are strongest, suggesting source-like authority or traceability display—not additional tokens alone—drives the effect.Source: §5.1 · p. 6

5.2 Strict Rubrics Suppress Presentation Bias While Retaining Clear-Conflict Accuracy

• 0%: stronger anti-citation prompts reduce worse-cited wins in commercial judges, while clear-conflict accuracy remains at or above 93.0%.Figure 2 shows high cited-candidate win rates moving toward Tie under strong anti-citation prompting.Source: §5.2 · p. 6, Figure 2 · p. 7

5.3 Moderate Quality Gaps Reveal Resolution Loss

• At L3, Claude and DeepSeek reach 28.5% and 21.0% Tie on human-validated moderate gaps, despite human Tie prevalence of 9%.In the soft tier, human any-Tie prevalence is 96% and commercial-judge Tie rates are 96–100%; this supports, but does not prove, calibrated Tie.Source: §5.3 · p. 7

5.4 The Bias–Resolution Frontier Appears Across Commercial and Open-Weight Judges

• Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT reproduce the frontier, but differ in strict-prompt resolution and Tie behavior.Qwen retains more moderate-gap resolution than the most abstention-heavy commercial judges; Gemma has a sharp L2-to-L3 Tie jump, followed by partial L4 recovery.Source: §5.4 · p. 7
• 89.0 and 98.0 points: Claude and GPT-5.5 lose this much better-plain resolution, respectively, from L0 to L4 (Figure 3).For every commercial judge, the HotpotQA L0-to-L4 loss and L4-to-TRACE recovery are significant (exact McNemar, p < 0.001).Source: §5.4 · p. 7, Figure 3 · p. 8

5.5 TRACE-Style Decoupling Recovers Moderate-Gap Resolution

• 97.0–100.0% commercially and 96.5–98.5% for open-weight judges: decoupled_quality_v2 preserves better-plain resolution, with worse-cited wins at 0%.It preserves quality-relevant support-chain and reasoning signals while removing display cues.Source: §5.5 · p. 8
• Across 50 FinQA pairs, L0–L2 better-plain rates are 95–97% for GPT-5.5, 92–98% for Claude, and 88–95% for DeepSeek.At L4, Tie reaches 91%, 100%, and 99%, respectively; TRACE recovers 98/100, 100/100, and 100/100 ordered decisions with no Ties.Source: §5.5 · p. 8

5.6 Controlled Ranking Shows System-Level Risk

• Citation-like display improves a workflow variant’s apparent standing under ordinary judging, while stricter or decoupled protocols reduce that advantage.The controlled ranking probe uses 200 HotpotQA queries and four workflow variants; it tests ranking effects rather than producing a new RAG leaderboard.Source: §5.6 · p. 8

6 Discussion

• A protocol that suppresses citation preference can still fail as a measurement instrument if it turns human-validated moderate differences into Tie.Clear-conflict accuracy alone is insufficient because a judge can recognize obvious correctness while missing moderate support-chain differences.Source: §6 · pp. 8–9
• Tie is success on content-equivalent pairs, may be calibrated on soft boundaries, and is a cost on human-validated moderate gaps.The discussion recommends reporting Tie by tier rather than only aggregate win rates.Source: §6 · p. 9
• Citation-like artifacts should be controlled for final-answer correctness, but traceability, evidence use, or workflow faithfulness must remain represented when those are evaluation targets.The protocol should preserve those quality dimensions while preventing source-like presentation from acting as a shortcut.Source: §6 · p. 9
• TRACE requires approximately 3.7–4.3× the tokens and 5.3–5.9× the dollar cost of one-call judging, and may introduce extraction errors.The authors frame it as an audit protocol rather than a universal deployment solution.Source: §6 · p. 9

• TraceJudgeBench audits how LLM-as-a-judge protocols handle citation-like artifacts in RAG and agent-workflow evaluation.Source: §7 · p. 9
• Tie is correct on equivalence pairs, potentially calibrated on soft-boundary pairs, and costly on validated moderate quality gaps.TRACE-style decoupling recovers much of the lost resolution by removing citation-like presentation while preserving quality-relevant support structure.Source: §7 · p. 9
• Debiasing should be reported as a frontier over presentation suppression, correctness retention, quality resolution, Tie cost, order effects, and protocol cost.Source: §7 · p. 9

• TraceJudgeBench is a diagnostic benchmark, not a universal RAG or agent leaderboard.Its controlled, partly template-built splits do not cover the diversity of naturally occurring quality differences, retrieval pipelines, agent tools, or user-facing domains.Source: Limitations · p. 9
• Surface regularities may make tampered candidates detectable without full task reasoning; ablations and order controls reduce but do not eliminate this concern.Source: Limitations · p. 9
• The full diagnostic matrix covers three commercial judges, while open-weight experiments focus on selected audits; most human validation uses two annotators.The FinQA disagreement receives independent adjudication, but broader multi-annotator validation would further strengthen the benchmark.Source: Limitations · p. 9
• TRACE is costly, may introduce extraction errors, and its high recovery alone does not isolate Stage 2’s contribution.Stage 1 can perform part of the quality assessment before masked comparison; fixed-field normalization explicitly records missing support.Source: Limitations · p. 9

Ethics Statement

• The annotation task concerns model outputs, not personal or sensitive user data; graduate-student annotators familiar with NLP evaluation worked under blind ordering and received locally minimum-wage-consistent hourly compensation.HotpotQA-derived material is distributed under CC BY-SA 4.0 and FinQA under the MIT license.Source: Ethics Statement · pp. 9–10

Appendix; A Benchmark Construction Details

• The appendix expands the main paper’s compact diagnostic matrix so results can be traced to splits, protocols, and model settings.It provides construction details, prompt templates, extended results, and reproducibility notes; its cross-reference maps main-paper floats to supporting appendix material.Source: Appendix · p. 12, A Benchmark Construction Details · p. 12

A.1 Mechanical Equivalence Pairs

• Mechanical pairs hold substantive answer content constant while adding citation-like wrappers, so cited-candidate wins indicate presentation sensitivity rather than better answer quality.The construction includes 200 HotpotQA and 200 FinQA pairs.Source: §A.1 · p. 12

A.2 Natural Citation-Looking Pairs

• The natural HotpotQA split contains 1,000 citation-looking pairs with matching final answers; a 200-pair subset is human-validated for stricter protocol comparisons.The full set supports plain-judge presentation-sensitivity audits, not a full strict-protocol matrix.Source: §A.2 · p. 12

A.3 Citation Ablation Variants

• Six ablation variants separate source-like authority, raw evidence, formatting shell, sentence identifiers, and added length.The variants are full citation, empty shell, source-only, evidence-only, sentence-ID-only, and length-control.Source: §A.3 · p. 12

A.4 Correctness Conflicts

• Clear-conflict probes pair one correct and one incorrect answer across three domains, prompt paraphrases, and candidate orders.The 150-pair split covers open-domain multi-hop QA, financial reasoning, and short code-style tasks; decoupled correctness judging is a targeted probe, not a full paraphrase matrix.Source: §A.4 · p. 12

A.5 Graded Soft and Moderate Quality Conflicts

• Graded pairs share a correct final answer: moderate pairs omit or weaken a necessary support-chain step, while soft pairs differ in specificity near the human decision boundary.HotpotQA pairs are constructed from locally checked traces and fixed templates, then blindly human-validated; the 50-pair FinQA split omits an arithmetic-chain step and is tested at five prompt levels in both orders.Source: §A.5 · pp. 12–13

A.6 Representative Examples

• Examples illustrate that equivalence targets Tie, correctness conflicts target better-plain, and omitted support in moderate gaps makes Tie a resolution loss.Soft examples sit near the human decision boundary, so Tie can reflect calibration rather than automatic failure; the FinQA example omits a denominator or conversion step.Source: §A.6 · pp. 13–14

B.1 Annotation Tasks

• Human validation uses blind pairwise labels for equivalence and graded quality gaps, interpreting Tie according to each pair’s target relation.A moderate-pair Tie counts as resolution loss when human consensus establishes a better-plain direction; validation uses two independent graduate-student annotators familiar with NLP evaluation.Source: §B.1 · p. 14

B.3 Why Cohen’s Kappa Is Not Reported

• Cohen’s kappa is omitted because prevalence-skewed validation sets can produce low kappa despite high raw agreement; the appendix instead reports raw agreement, PABAK, and Gwet’s AC1.The authors state that this does not change the qualitative conclusion of strong annotator consensus on the moderate tier.Source: §B.3 · p. 14

C.1 Single-Call Prompt Ladder

• The prompt ladder offers interpretable operating points rather than an exhaustive search, and L2/L3 resolution loss is more informative than the deliberately strict L4 endpoint.Faithful FinQA prompt paraphrases retain the direction of the tested L2/L3 results; whether reasoning completeness may break a tie marks the critical instruction boundary.Source: §C.1 · pp. 14–15

C.2 Generic Pairwise Judge Output Schema; E.1 Equivalence and Natural Protocol Summary; E.3 Moderate Prompt Frontier (Full Ladder)

• The appendix reports a generic A/B/Tie output schema and conservative parsing that sends ambiguous or unparsable responses for manual inspection rather than silently coercing them.The appendix also describes released protocol-specific row manifests and reports the full ladder as model-specific and not necessarily monotonic.Source: §C.2 · p. 15, §E.1 · p. 17, §E.3 · p. 17

C.3 TRACE-Style Decoupled Audit

• TRACE’s normalization can perform part of the quality assessment, so high recovery alone cannot distinguish sound decomposition from decisions already encoded in Stage 1.The authors leave isolation of these contributions to future ablations.Source: §C.3 · pp. 16–17

D.1 Commercial Judges; D.2 Open-Weight Judges

• The full diagnostic matrix covers three commercial judges, while the open-weight experiments replicate only the central moderate-gap frontier rather than the full benchmark.The open-weight coverage excludes equivalence, ablation, correctness-conflict, position-control, and paraphrase-audit splits.Source: §D.1 · p. 17, §D.2 · p. 17

E.1 Equivalence and Natural Protocol Summary; E.2 Citation Ablation Results; E.3 Moderate Prompt Frontier (Full Ladder)

• Stricter and decoupled protocols suppress citation wins on equivalence-like pairs, mostly by moving decisions to Tie; full-citation and source-only variants show the strongest positive effects.The moderate frontier is driven primarily by better-plain decisions converting to Tie, not by renewed preference for the weaker cited candidate.Source: §E.1 · p. 17, §E.2 · p. 17, §E.3 · p. 17

C.3 TRACE-Style Decoupled Audit; E.4 TRACE Recovery on the Moderate Tier

• TRACE recovery supports protocol-dependent resolution loss: removing citation-like display is insufficient unless normalization preserves evidence-chain quality.Its two-call protocol normalizes candidates before masked judging; the reported accounting is approximately 3.7–4.3× the tokens and 5.3–5.9× the dollar cost of one-call judging.Source: §C.3 · pp. 15, 17, §E.4 · p. 17

E.5 FinQA Moderate Robustness Check

• The FinQA check runs 50 pairs in both candidate orders, and the appendix identifies Table A13 as the source for L0–L2 ranges and strict and TRACE endpoints.The supplied passages do not include those table values.Source: §E.5 · p. 17, Table A13 · p. 19
• 46% better-plain at DeepSeek L3 shows FinQA resolution loss before the L4 endpoint; on HotpotQA, Claude and GPT-5.5 drop 89.0 and 98.0 percentage points, respectively, from L0 to L4.The HotpotQA L0-to-L4 losses and L4-to-TRACE recoveries are significant for every commercial judge; the L3-to-TRACE comparison is significant for Claude and DeepSeek, but not GPT-5.5.Source: §E.5 · pp. 17–18

E.6 Equivalence Suppression by Split

• The appendix identifies strict anti-citation prompting as suppressing presentation preference in equivalence settings.The supplied passages state this qualitative finding without giving the split-specific rates.Source: §E.6 · p. 18, Table A14 · p. 19

E.7 Correctness Retention

• The decoupled correctness-retention probe stays at or above 93.0% correct-base wins across commercial judges, indicating the reported resolution loss is not simply correctness collapse.Table A15 is a targeted decoupled probe, not the L4 strict-prompt endpoint.Source: §E.7 · p. 18, Table A15 · p. 20

E.8 Position Controls

• Position-only controls return all Tie for identical plain candidates, while citation-order gaps arise when citation-like display is introduced.The reported controls therefore identify the order coupling as citation-conditioned rather than generic position bias; GPT-5.5 is not included in this targeted control table.Source: §E.8 · p. 20

E.9 Workflow-Ranking Probe; E.10 Direct Workflow-Pair Comparisons

• The workflow-ranking probe compares four workflow variants across 200 HotpotQA queries, all six unordered system pairs, and both candidate orders.The supplied passage describes the probe design but gives no ranking outcomes.Source: §E.9 · p. 20

F Soft Tier: Calibrated Tie Counterpart

• On soft-boundary pairs, abstention is calibrated when human annotators themselves do not commit to a direction; on moderate-gap pairs, abstention loses resolution when humans establish one.The appendix presents the soft prompt frontier as evidence for the calibrated-Tie side of this distinction.Source: F Soft Tier: Calibrated Tie Counterpart · p. 20, Figure A1 · p. 20

G Rationale Keyword Analysis

• Citation-win rationales describe cited outputs as more supported, traceable, or verifiable, while Tie rationales may still mention citation-like display.The authors interpret this as cue perception potentially differing from cue use: a judge may notice display without converting it into preference.Source: G Rationale Keyword Analysis · pp. 20–21
• The rationale keyword audit uses selected subsets rather than a complete rationale matrix, and some missing cells were too small for stable reporting or outside the released audit.The authors treat rationales as supporting evidence, not causal proof or evidence of hidden mechanisms.Source: G Rationale Keyword Analysis · pp. 20–21

H Reproducibility Checklist

• The reproducibility artifact includes benchmark splits, prompts, raw outputs, metadata, human-validation materials, analysis scripts, and figure-generation sources.The artifact is de-identified and excludes private API keys, account identifiers, and personally identifying annotator information.Source: H Reproducibility Checklist · p. 22
• Some commercial endpoints expose limited model-version information, so exact provider-side snapshots may not be recoverable despite preserving available metadata and call dates.The prompt ladder is also a targeted audit, not an exhaustive prompt search; soft and most HotpotQA moderate analyses use one paraphrase per level.Source: H Reproducibility Checklist · p. 22