PaperlayerView exact revision on arXiv

Source-linked AI summary

Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict

Sourabrata Mukherjee, Sunayana Sitaram

arXiv:2609.32407v1cs.AI

TL;DR

LLM judges can favor surface features over answer quality, raising whether the information needed for a correct judgment is absent or merely hidden from the verdict. The paper probes frozen activations and intervenes during judgment, finding recoverable preference signal across benchmarks and a diagnostic that predicts when read-out helps.

  • Problem

    When an LLM judge gives a wrong verdict, it is unclear whether the information needed to judge correctly is absent or remains in internal representations but is not exposed in the output.

    Source: §1 · p. 1

  • Method

    The study probes frozen internal activations and intervenes during forward passes, while testing a surface-confound diagnostic across evaluators and datasets.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across eight pairwise benchmarks, probes improve on verdicts after surface controls and model-family clustering; on LLMBar, the residualized read-out reaches 0.686 versus 0.507 for shuffled labels.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A surface-confound score indicates when internal read-out helps, and recovered preferences improve evaluation and downstream preference learning at the same human-label budget.

    Source: §1 · p. 2

    The diagnostic covers surface confounding, one common failure mode, and is supported for the benchmarks studied rather than every setting.

    Source: §6 · p. 7

Abstract

from arXiv · show

LLM judges are widely used to evaluate model outputs, but their verdicts can be unreliable: a judge may favor the worse answer for its position, length, or other surface features. When a judge is wrong, is the information needed to judge correctly absent from the model, or present in its internal representations but not reflected in the output? We study this across 64 open-weight evaluators and 14 datasets, including causal interventions on 41 judges (editing activations mid-run to see whether the verdict changes). On LLMBar, built so the superficially better answer is the worse one, the verdicts of 50 judges agree with human labels only 0.456 of the time, even after averaging both answer orders. Yet a small probe on the same judges' activations, with no weight updates, reaches 0.846, and 0.686 once surface features such as length and position are residualized out (0.507 with shuffled labels). The gap holds across eight benchmarks and model families, but is not universal: a score of how well surface features alone predict the human label, computed before any probe is trained, predicts the size of the gain (Spearman rho = 0.90). On rubric tasks that score one answer at a time, leaving no surface cue to exploit, reading the internals gives no advantage. The interventions also show that editing activations mid-network already changes the verdict, before it can be read off directly, and locate the pathways carrying position and length bias. At the same label budget, the recovered signal lets a judge flag cases where it is likely wrong and yields better labels for preference learning. A wrong verdict, then, does not mean the judge lacks the information, and a simple diagnostic shows when it is worth recovering.

1 INTRODUCTION

Because judges can favor position, length, or style over quality, the paper asks whether preference information survives inside a wrong verdict. It studies frozen-activation read-out, causal interventions, and when a surface-confound diagnostic predicts usefulness.

1 INTRODUCTION

  • 0.456 verdict agreement contrasts with 0.846 from a frozen-activation probe, remaining 0.686 after surface residualization versus the 0.507 shuffled-label control.The comparison is on LLMBar, with verdict accuracy averaged over both answer orders.Source: Figure 1 · p. 2, §1 · p. 2
  • Across 41 judges, mid-network activation edits already change the verdict before a lens can decode it, and interventions locate pathways carrying position and length biases.A common attribution method points to different components than those whose removal actually changes the verdict.Source: §1 · p. 2
  • Across eight benchmarks, the surface-confound score predicts internal-read-out gains (Spearman ρ = 0.90), while rubric scoring of one answer at a time shows no advantage.The score measures how well response length predicts human labels; the authors propose reading frozen activations when it indicates likely output unreliability.Source: §1 · p. 2
  • At the same human-label budget, recovered preferences improve evaluation and downstream preference learning, while the diagnostic identifies settings where read-out helps or does not.The paper's contributions also include using recovered preferences as training labels and abstaining on likely errors.Source: §1 · p. 2

2 RELATED WORK

Prior work studies judge reliability, reads evaluators' internal representations, and localizes computation with mechanistic interventions. This paper connects those threads by asking why output can lose preference signal and when recovering it is worthwhile.

2 RELATED WORK

  • Prior reliability work documents position, length, and self-preference biases and repairs judges at the output, but does not ask what they computed internally.The cited approaches include preference optimization, multi-objective heads, swap-averaging, self-consistency, and calibration.Source: §2 · p. 3
  • Recent evaluator-probing studies extract preferences or improve score correlations from internal representations, leaving open why output loses signal and when recovery is worthwhile.The passage discusses Maiya et al. (2025), Lai et al. (2025), Girrbach et al. (2025), and Li et al. (2026).Source: §2 · p. 3
  • Mechanistic methods locate evaluator computation, but attribution surrogates can diverge from causal effects; this work tests whether human-labeled preference survives output and when it can be recovered.The passage contrasts its question with Feldhus et al. (2026), whose account focuses on format-dependent scores from a shared internal evaluator.Source: §2 · p. 3

3 SETUP

The setup distinguishes verdicts, internal read-outs, and interventions, then separates decodability, causal influence, and causal use. It compares readers under matched human-label budgets across a broad model and dataset registry.

3 SETUP

  • Internal read-outs decode per-layer estimates from the residual stream using the model's final LayerNorm and unembedding or a learned affine; verdicts use final-layer logits.Lens decoding depth is the fractional layer where the estimate first matches the final verdict.Source: §3 · p. 4
  • Steering, ablation, and counterfactual activation patching test whether interventions change the emitted verdict; causal decision depth marks the earliest layer where a residual patch reliably flips it.The decision may form gradually across layers even though an intervention first finds causally sufficient information at a particular depth.Source: §3 · p. 4
  • Probes test whether human labels are decodable (L1), interventions test causal influence on verdicts (L2), and the study does not establish that judges causally used the decoded signal (L3).Per-judge causal decision depth does not predict per-judge probe gain (ρ = 0.06, p = 0.73, n = 39).Source: §3 · p. 4
  • Five readers receive the same number of human labels: raw and order-averaged verdicts, internal probes, fine-tunes, and output-level fixes; compute and storage differ and are tracked separately.The label budget counts only human-labeled items.Source: §3 · p. 4
  • The registry contains 64 open-weight models from 21 families and 14 published datasets; the causal suite intervenes on 41 of 50 generative judges plus five reward models.The datasets include eight pairwise-preference benchmarks and six auxiliary sets.Source: §3 · p. 4

4 THE INTERNAL–OUTPUT GAP

Internal probes outperform verdicts across pairwise benchmarks, including after surface controls and model-family clustering. The LLMBar-adversarial results show a large gap, while residualization and clustered analyses test its robustness.

4 THE INTERNAL–OUTPUT GAP

  • On LLMBar-adversarial, the residual-stream probe reaches 0.846, versus 0.456 for order-averaged verdicts; 72% of judges score below chance despite a median 0.89 on the easy split.The probe improves over the verdict for every judge, and the result replicates within ±0.012 in a second seed.Source: §4 · p. 4
  • Across eight benchmarks, probes beat raw verdicts by mean +0.117 across 400 cells, improving 89%; family clustering leaves the pooled gain at +0.108 [0.089, 0.127].Family-clustered confidence intervals exclude zero on seven of eight benchmarks; hh-rlhf is the exception.Source: Figure 2 · p. 5, §4 · p. 5
  • After nine surface features are residualized, the residual-stream probe reaches 0.686 versus 0.507 for shuffled labels, beating that control for 50/50 judges.On a length-balanced subset, the probe scores 0.684 versus 0.482 for the order-averaged verdict.Source: §4 · p. 5

5 THE ANATOMY OF A JUDGE

Interventions show that activations can determine a judge’s verdict before a lens can decode it, and identify pathways carrying position and length bias. Direct logit attribution tracks these causal effects poorly because most of the biased effect is indirect.

5 THE ANATOMY OF A JUDGE

  • 34 of 39 judges had verdicts changed by patching at an earlier layer than a lens could decode them (median depths 0.585 vs. 0.906; p=2×10−7).For rubric scoring of one answer, the decision forms later, at 0.9–1.0 depth; causal onset is estimated on a coarse grid.Source: §5 · p. 6
  • DLA’s component-level ranking correlated only ρ = 0.31 with causal-knockout rankings; 82% of the biased effect was indirect.Separate decompositions found MLP-mediated contributions (0.63) exceeded attention-mediated ones (0.39); those fractions need not sum to one.Source: §5 · p. 6
  • Ablating selected attention heads reduces the position-bias gap by 33–54%, versus ≈3% for random heads.Heads selected on one benchmark reduce the bias by 51% on other benchmarks.Source: §5 · pp. 6–7
  • Projecting out a judge’s residual-stream length direction removes length bias (0.41→−0.11) without changing human-label agreement.Directions from one judge also shift another judge’s score, though each encodes the shared bias geometry in its own coordinates.Source: §5 · p. 7
  • The verdict exposes a low-dimensional projection of richer internal state (effective rank ≈2–4), while a mid-layer reward-model read-out correlates +0.2–0.3 more with human labels than its scalar head.The output projection aligns only weakly with the human axis.Source: §5 · p. 7

6 WHEN READING INTERNALS HELPS

The benefit of reading internals depends on surface confounding: a pre-read-out length-only diagnostic tracks probe gains across benchmarks. The advantage concentrates on confounded pairwise tasks and is absent on clean and single-answer rubric settings.

6 WHEN READING INTERNALS HELPS

  • Across eight benchmarks, pre-read-out cf tracks internal-read-out gains at Spearman ρ=0.90, including an independent-run prediction at ρ = 0.95.cf is the cross-validated accuracy of a length-only classifier; the association remains after accounting for judge accuracy and model family.Source: §6 · p. 7
  • On clean benchmarks, the probe’s non-redundant advantage over order averaging falls to ≈0, with JudgeBench at −0.01 and HH-RLHF at −0.02.The figure caption also characterizes the probe’s edge over output-level fixes as confined to confounded cells, vanishing or reversing on clean ones.Source: §6 · p. 7, Figure 3 · p. 8
  • Internals-only cross-judge transfer improves agreement by +0.08 on LLMBar, but is flat to slightly negative on JudgeBench (−0.01) and RB2 (−0.03).The larger +0.32 combined internal-plus-length result is largely carried by length, so the paper reports the internals-only figure.Source: §6 · p. 7
  • The read-out does no better than emitted scores on single-answer rubric tasks: Samiksha probe ρ = +0.017 and Pariksha probe ρ = −0.053.Held-out best layers beat final layers on 52% and 50% of judges, respectively.Source: §6 · pp. 7–8
  • Internal confidence gains +0.39 risk–coverage AUC over verdict margin on LLMBar, where the margin is below chance (0.431).The reader lifts the weakest judge quartile from mean raw accuracy 0.50 to 0.65, while remaining distribution-specific.Source: §6 · p. 8, Figure 3 · p. 8

7 DOWNSTREAM CONSEQUENCES

At matched human-label budgets, reading internals outperforms output-level fixes on confounded cells and yields better preference labels for downstream training. The probe also outperforms the tested full-parameter fine-tune, though its safety comparison is qualified by the untuned fine-tuning recipe.

7 DOWNSTREAM CONSEQUENCES

  • Across the matched-budget comparisons, probe advantages over output-level fixes concentrate on confounded cells, and the DPO labeling advantage grows as labels get scarce.The paper reports that these advantages decline with cf and vanish on length-neutral sets.Source: §7 · pp. 8–9
  • At 256 human labels over 24 cells, the probe beats output-level fixes by +0.17 to +0.21 on LLMBar and RB2, with ≈0 advantage on clean JudgeBench and Arena.The comparison includes self-consistency and a calibrated prompt instructing judges to ignore length and position.Source: §7 · p. 8, Table 3 · p. 9
  • With the same 3,349 labels and held-out 590 items, the frozen probe reaches 0.753 accuracy, above the fine-tune (0.659) and base judge (0.700).The probe leaves base-model weights unchanged; the tested fine-tune’s PKU-Safe accuracy fell from 0.716 to 0.569, while the probe retained 0.707.Source: §7 · pp. 8–9, Table 3 · p. 9
  • Probe labels win on four of five DPO cells and tie on the fifth; on adversarial LLMBar, they recover ≈90% of the verdict-to-gold gap (0.463 →0.787; gold 0.825).The gain survives a 1.5B→3B policy swap (+0.15), while the less-confounded RB2 gain is small (+0.016).Source: §7 · p. 9, Table 3 · p. 9

8 CONCLUSION

The conclusion presents recoverable internal preference information as a complement to unreliable judge verdicts, while noting that the judge’s use of that information remains unresolved. The paper also documents its AI use, data scope, and reproducibility resources.

8 CONCLUSION

  • Across 64 open-weight evaluators, frozen-activation probes recover preference information that can outperform verdicts, including after controls for surface features and model family.The authors describe causal interventions as locating pathways behind bias; the advantage concentrates where surface features predict human labels and largely disappears on rubric tasks without those confounds.Source: §8 · p. 9
  • Whether judges use the recoverable preference information to form their verdicts remains open; the paper establishes decodability and causal influence, not causal use.Source: §8 · p. 9

AI USE STATEMENT

  • General-purpose AI assistants helped with writing, argument stress-testing, code drafts, and presentation, under author review and revision; they did not generate central experimental data or labels.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses existing open-weight models and previously published datasets, treats human preference labels as noisy targets, and notes that bias directions could in principle be steered to strengthen bias.A restricted human-annotated corpus is not redistributed; its schema, prompts, and computed results are released instead.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors identify released code, model and experiment artifacts, evaluation prompts, a BUDGET grid, and a fine-tuning configuration as reproducibility resources.The statement also points to the algorithm, registry, instrument inventory, configurations, seeds, determinism settings, and cost accounting.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

Table 4 serves as a guide to appendix sections and the tables and figures they contain.

A APPENDIX

A.1 ADDITIONAL DEFINITIONS

The appendix defines two read-out quantities used in §5: dimensional collapse and a concept direction.

A.1 ADDITIONAL DEFINITIONS

  • Dimensional collapse compares effective ranks of internal and emitted score representations, while a concept direction is a probe’s unit weight vector used for steering and ablation.Effective rank is measured by the participation ratio; the probe direction predicts a nuisance or target attribute from hℓ.Source: §A.1 · p. 14

A.2 CLAIM–EVIDENCE MATRIX

Table 5 is a claim–evidence matrix for the main paper.

A.2 CLAIM–EVIDENCE MATRIX

  • Table 5 maps each main-paper claim to its evidence, whether that evidence is causal or correlational, its scale, and claims the authors deliberately do not make.Source: §A.2 · p. 14, Table 5 · p. 15

A.3 GUIDE TO THE MAIN-PAPER FIGURES AND TABLES

The guide maps the main-paper figures and tables to their panels, columns, statistics, and controls. Figure 3’s panels summarize four consequences, from probe comparisons to risk–coverage analysis.

A.3 GUIDE TO THE MAIN-PAPER FIGURES AND TABLES

  • The guide names figures on the gap, where decisions form, end-to-end read-out, and consequences; Table 6 supplies panel and column readings, statistics, and controls.Figure 6 expands the consequences to 10 panels and 24 cells with the same labels.Source: §A.3 · p. 15, Table 6 · p. 15, Figure 4 · p. 15
  • Figure 3 summarizes probe advantages, label-efficiency effects, position-head ablations, and risk–coverage AUC across its four panels.The probe’s paired advantage is +0.17–+0.21 on confounded cells and approximately zero or negative on clean cells; the rb2 DPO gain peaks at +0.029 with 256–512 labels.Source: §A.3 · p. 16
  • Table 2a defines the surface-confound diagnostic as cross-validated accuracy of a length-only probe, computed before any internal read-out probe is fitted.Its column guide also defines raw verdicts, order averages, probe gains, and the eight-benchmark mean; all eight probe-versus-raw gains are significant under FDR-BH.Source: Table 2a · p. 17
  • Table 3a compares output-level fixes on identical items across 24 judge–benchmark cells, using the same 256 human labels for every arm.The read-out is a 41-parameter logit-lens probe; self-consistency uses five samples at temperature 0.7, while the calibrated prompt instructs judges to ignore length and position.Source: Table 3a · p. 17
  • Table 3b compares full fine-tuning with a frozen-judge probe on the same 590-item test set, with results also reported excluding the safety split.On pku-safe, full fine-tuning regresses from 0.716 to 0.569 while the probe scores 0.707; the llmbar arms tie at 0.660, with n=47.Source: Table 3b · p. 17

A.4 ADDITIONAL FIGURES

This section provides supplementary documentation on the reader’s recipe, extended results, costs, mechanisms, and scope, including the distribution-specific nature of the read-out and the distinction between recoverable information and information used in a verdict. It also describes the provenance and redistribution restriction of the datasets.

A.4 ADDITIONAL FIGURES

  • The appendix’s companion figures show decision formation at 0.58 depth and summarize read-out gains across 50 judges and eight benchmarks.Figure 4 reports that untrained read-out is wrong at every depth on adversarial items; Figure 5 reports probes beating verdicts, monotonic accuracy gains from abstention, and a gain predicted by cf (ρ=0.90).Source: Figure 4 · p. 18, Figure 5 · p. 18

A.6 THE DEPLOYABLE READ-OUT RECIPE

  • The deployable recipe uses a surface-confound check to decide whether to read internals, then order-averages frozen-judge activations and fits an L2 logistic probe when needed.The cf threshold of 0.75 is a rule of thumb to recheck on new suites; low-confound sets use the free order-average, and low internal confidence triggers abstention and human review.Source: §A.6 · p. 19

A.5 MODEL REGISTRY AND DATASET SHEET

  • The registry covers 64 models, and the dataset sheet covers 14 existing datasets, including eight pairwise-preference benchmarks and six auxiliary sets.The registry comprises 50 panel judges, five read-out-only large judges, and nine scalar reward models; the auxiliary sets support mechanism, rubric, and multilingual analyses.Source: §A.5 · pp. 17–18, Table 7 · p. 20, Table 9 · p. 22

A.7 EXTENDED EVALUATOR RESULTS

  • The internals-only auditor improves held-out-judge human agreement by +0.077 on LLMBar, beating the judge for 78% of judges, but is flat or slightly negative on JudgeBench and RB2.Across eight benchmarks, internal-confidence risk–coverage AUC lifts the no-selection baseline by +0.02 to +0.09; at 25% coverage, accuracy is 0.91–0.98. The larger combined internal-plus-length gains are not evidence for transferred internal structure because length is an within-benchmark oracle on those sets.Source: §A.7 · p. 20
  • Across benchmarks, probes do not transfer naively, while leave-one-dataset-out probing beats the verdict by +0.03; deployment therefore needs labels matched to the evaluation distribution.The internals-only auditor transfers across judges (+0.08 on LLMBar) but not to clean benchmarks; naïve LLMBar-to-RB2 transfer is −0.29 and leave-one-benchmark-out beats the verdict on 1/8.Source: §A.7 · pp. 20–21

A.10 CONSEQUENTIAL EXPERIMENTS: FULL TABLES

  • Probe-labeled DPO wins on four of five cells, with gains scaling with confound and label scarcity; the RB2·Q7B·3B cell is treated as level, not a win.A separate run of that cell returned 0.000, despite the full-pool sweep reporting +0.011.Source: §A.10 · p. 23, Table 17 · p. 26
  • The stacked reader beats order-averaged and final-only readers on LLMBar and RB2, while remaining neutral on clean benchmarks.LLMBar: 0.738 vs. 0.454 / 0.582; RB2: 0.828 vs. 0.678 / 0.707.Source: §A.10 · p. 23
  • On RB2, residual preference content survives full-surface residualization by +0.086 over shuffle, although surface features alone predict labels at 0.980.Across four aligned benchmarks, full-surface and length residualization differ by at most 0.005; the extra eight surface features remove nothing more.Source: §A.10 · p. 23

A.13 EXTENDED LIMITATIONS AND DATA STATEMENT

  • 13 of 14 datasets are publicly obtainable; Samiksha is not openly redistributable.The authors release its schema, rubric definitions, language coverage, evaluation prompts, and computed results, and direct readers to its authors for access under the collection terms.Source: §A.13 · p. 27

Related papers

Loading 2609.32407v1…