PaperlayerView exact revision on arXiv

Source-linked AI summary

Backbone-Adaptive Evidence Routing for Robust Pairwise LLM Judging

Zeyan Li, Jing Peng, Jianfeng Xu

arXiv:2609.30751v1cs.AI

TL;DR

Pairwise judges may need different evidence mechanisms across benchmark–backbone conditions, while candidate order must not affect judgment strength. BAER separates preference from reliability, builds three symmetric evidence heads, and selects a head using development data. It achieves the highest accuracy in all eight conditions, with full test-pair coverage.

  • Problem

    A judging protocol that works well for one benchmark–backbone condition may not be strongest for another, and judges exhibit recurring weaknesses such as position preference and susceptibility to surface features.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    BAER separates signed preference from candidate-invariant reliability and uses stacking, expert routing, or candidate-blind reference verification, with a development-selected head frozen for each condition.

    Source: §1 · p. 1

  • Results

    BAER achieves the highest accuracy in all eight conditions, with margins of +0.87 to +7.32 points over the strongest external baseline and a prediction for every test pair.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The results show that no fixed protocol dominates across conditions, supporting condition-specific evidence-head selection.

    Source: §1 · p. 2, §3.2 · p. 3

    BAER’s symmetry constraint requires swapping the responses to reverse preference without changing its strength.

    Source: §2 · p. 2

Abstract

from arXiv · show

Pairwise language-model judges can gather evidence through direct comparison, reasoning, or reference-based verification, but no single protocol is best across benchmarks and judge backbones. We introduce Backbone-Adaptive Evidence Routing (BAER), which adapts the evidence mechanism while preserving candidate symmetry: swapping the two responses may reverse the preference but cannot change its strength. BAER separates each expert's signed preference from candidate-invariant reliability and builds three symmetric heads: evidence stacking, reliability-based expert routing, and candidate-blind reference verification. Development data select one head for each benchmark--backbone condition, and that choice is frozen before testing. Across four benchmarks and two 8B judge backbones, BAER achieves the highest test accuracy among the compared methods in all eight conditions, with full prediction coverage and gains of 0.87--7.32 points over the strongest external baseline. The results show that adapting how evidence is gathered is more reliable than fixing one judging protocol everywhere.

1. INTRODUCTION

Pairwise judges can require different evidence mechanisms across benchmark–backbone conditions, and those mechanisms must remain independent of candidate order. BAER separates preference from reliability and uses condition-level head selection to adapt evidence while preserving candidate symmetry.

1. INTRODUCTION

  • A protocol that works well for one benchmark–backbone condition may not be strongest for another, while judges also show recurring weaknesses including position preference and susceptibility to surface features.Prior methods that correct one condition may discard useful evidence in another.Source: §1 · p. 1
  • When experts disagree, selecting a reliable expert can help; when they share an incorrect answer, combining their judgments may be insufficient and a candidate-independent reference offers another verification basis.The figure’s examples are schematic, and its head assignments match deployed JudgeBench conditions.Source: §1 · p. 1, Figure 1 · p. 1
  • BAER separates each expert’s signed preference from candidate-invariant reliability, then offers stacking, per-pair expert routing, and reference verification as symmetric evidence heads.Swapping candidates reverses signed preference but leaves reliability unchanged.Source: §1 · p. 1
  • Development data select a head for each benchmark–backbone condition, and the selected head is frozen before testing; only routing chooses an expert separately for each pair.Existing approaches mainly improve a fixed protocol, calibrate a predetermined rule, aggregate judges, or abstain rather than adapting the evidence mechanism across conditions.Source: §1 · pp. 1–2
  • Across eight benchmark–backbone conditions, BAER achieves the highest test accuracy among compared methods, with margins of +0.87 to +7.32 points over the strongest external baseline.Development experiments also show when stacking, routing, and reference verification provide useful evidence.Source: §1 · p. 2

2. BAER

BAER gives its judging heads a shared signed-score interface: swapping candidates reverses the score without changing its strength. It represents expert preference separately from reliability, offers three evidence heads, and freezes one head for each benchmark–backbone condition before testing.

2. BAER

  • Every BAER head returns a signed score whose sign selects the winner and whose magnitude measures confidence; swapping candidates negates the score.The selected condition uses one head for all its pairs, while routing may choose different experts pair by pair.Source: §2 · p. 2, §2.3 · p. 3

2.1. Symmetric evidence representation

  • Each of nine protocols supplies preference evidence and order-instability information; under candidate exchange, signed evidence negates while reliability remains unchanged.The stack also interacts mean evidence with candidate-invariant task categories, allowing weights to vary by task type.Source: §2.1 · p. 2

2.2. Backbone-adaptive evidence heads

  • Evidence stacking fits a logistic model over normalized features, with sign augmentation and no intercept enforcing antisymmetry.Its regularization is selected on calibration data and refit on non-test rows.Source: §2.2 · p. 2
  • Candidate-invariant routing estimates each expert’s reliability for a pair and selects the expert with the highest estimated reliability.The router’s features remain unchanged when candidates are exchanged, so the selected expert is preserved while its signed score negates.Source: §2.2 · p. 2
  • Reference verification adds an independent solution as evidence: the solver sees neither candidate, and each candidate is checked separately.The prompt, solution budget, constrained decoding, and aggregation are frozen before testing.Source: §2.2 · pp. 2–3

2.3. Condition-level selection and deployment

  • Development data assign a single head to each benchmark–backbone condition and freeze that choice for final scoring, without using test information per example.Stacking is the initial choice; routing is selected by cross-validation, while reference verification must clear selection and calibration margins.Source: §2.3 · p. 3

3. EXPERIMENTS

BAER is evaluated across four preference benchmarks and two 8B judge backbones, with development data used to select heads before test evaluation. It leads every benchmark–backbone test column, while the head experiments show that the most useful evidence mechanism varies by condition.

3.1. Setup

  • The evaluation uses four preference benchmarks and Qwen 3-8B and Llama 3.1-8B Instruct, judging each pair in both orders and mapping predictions to a canonical frame.Methods are compared on matched test IDs; accuracy is measured without controlling for inference cost, and BAER uses more calls than a single-pass judge.Source: §3.1 · p. 3

3.2. Overall effectiveness

  • BAER achieves highest accuracy in every Table 1 benchmark–backbone column, exceeding the strongest external baseline by +0.87 to +7.32 points; the strongest external method varies across columns.CalibraEval leads six columns, while chain-of-thought and PriDe lead the two JudgeBench columns.Source: §3.2 · p. 3
  • Six of BAER’s eight gains are significant at the 5% level, and it predicts every test pair; chain-of-thought coverage ranges from 74.8% to 100%.The two non-significant comparisons are the JudgeBench columns.Source: §3.2 · p. 4

3.3. Contributions of the evidence heads

  • Stacking leads 5/8 columns, routing-augmented BAER leads 7/8, and full BAER leads 8/8; reference verification raises JudgeBench/Qwen from 60.2% to 70.7%.Routing raises JudgeBench/Llama from 47.6% to 57.7% and UltraFeedback/Qwen from 97.0% to 98.5%.Source: §3.3 · p. 4
  • Six stacks stay within 1.5 points from calibration to test, but the two JudgeBench stacks drop 9.5 and 9.1 points; correct expert judgments exist, yet reliability-based selection remains difficult.An any-expert oracle reaches 95.4%/96.2% on the two JudgeBench conditions, while the JudgeBench/Qwen router’s +4.8 out-of-fold gain reaches only 60.2% on test.Source: §3.3 · p. 4

3.4. From selection to calibration

  • Advancing heads must retain gains on both a small selection split and a larger calibration split; candidate-blind reference verification is the Qwen head that clears both gates, with +3.7/+6.3 points.Claim-graph verification reverses from +2.2 on selection to -14.3 on calibration, and independent pointwise likelihood reverses from +5.9 to -5.0.Source: §3.4 · p. 4
  • Orbit wins only for Qwen on JudgeBench (+1.6), losing 0.8–38.3 points elsewhere.Figure 3 compares accuracy gaps with the strongest external baseline on matched IDs; only candidate-blind reference verification retains a positive gap through selection and calibration, and the Llama reference head was not evaluated.Source: §3.4 · p. 4, Figure 3 · p. 4

4. CONCLUSION

BAER adapts evidence gathering to each benchmark and judge backbone while preserving candidate symmetry. It leads the compared methods in all eight test conditions, with full prediction coverage and gains of +0.87 to +7.32 points.

  • BAER separates signed preference from candidate-invariant reliability and uses stacking, routing, or candidate-blind reference verification as evidence heads.Development data select the head for each benchmark–backbone condition.Source: §4 · p. 4
  • Across four benchmarks and two 8B judge backbones, BAER has the highest accuracy in all conditions, gains +0.87 to +7.32 points over the strongest external baseline, and predicts every test pair.Source: §4 · p. 4
  • The experiments indicate that stacking transfers well on most conditions, routing helps when invariant reliability identifies specialists, and reference verification adds evidence when pairwise experts share an error.The approach adds inference cost and condition-level deployment assumes the benchmark family and judge backbone are known in advance.Source: §4 · p. 4

Related papers

Loading 2609.30751v1…