PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM: Calibrating Black-Box Judges with Counterfactual Probes

Kakei Yamamoto, So Kuroki

arXiv:2610.18389v1cs.LGcs.AIcs.CL

TL;DR

LLM judges can disagree systematically with human judgments, motivating methods to correct their scoring biases. PRISM probes changes in overall scores when attribute emphasis shifts and learns a correction from human labels. In the main Jev and Sol comparisons, it improves agreement over uncalibrated scores while remaining competitive with rubric-based methods.

  • Problem

    LLM judges can systematically disagree with human judgments, and the paper asks whether responses to imposed evaluative biases can guide debiasing.

    Source: §1 · pp. 1–2

  • Method

    PRISM holds evaluated content fixed, measures overall-score changes under attribute-emphasis probes, and fits a ridge-regression correction from human labels.

    Source: §1 · p. 2

  • Results

    On SummEval and HANNA, PRISM and α-PRISM significantly improve ranking agreement over Vanilla in the main Jev and Sol comparisons while remaining competitive with rubric-based SAJA and G-Eval.

    Source: §1 · p. 2, §4 · p. 11

  • Takeaways & Limitations

    On a matched Jev–SummEval comparison, rubric-derived quality probes and style probes yield similar agreement, supporting calibration without reproducing rubric criteria.

    Source: §3.3 · p. 10, §4 · p. 11

    Additional probe calls incur inference cost; transfer across tasks, model versions, and rater populations remains untested, and benchmark contamination cannot be ruled out.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Large language models are increasingly used as judges to reduce reliance on costly human evaluation, yet their scores can systematically disagree with human judgments. We ask whether responses to an explicitly imposed evaluative bias can help correct a judge's implicit scoring biases. We introduce PRISM (Probe-based Residual Inference of Scoring Misalignment), which holds the evaluated content fixed and measures score changes when attributes are emphasized or de-emphasized. A local sensitivity model motivates representing these responses as a surrogate Jacobian. Ridge regression on human labels learns a correction to the baseline score. The resulting scoring protocol is rubric-free: it uses overall scores without requesting criterion-level ratings. In the main Jev and Sol comparisons on SummEval summarization and HANNA story evaluation, PRISM and its learned-baseline variant significantly improve agreement with human rankings over uncalibrated scores while remaining competitive with rubric-based methods. A controlled SummEval comparison yields similar agreement for rubric-derived quality probes and style probes, supporting calibration without reproducing rubric criteria. The learned coefficients reveal shared and opposing correction directions across judges.

1 Introduction

The paper asks whether controlled emphasis probes can help correct LLM judges’ implicit scoring biases relative to human judgments. It introduces PRISM, which learns a correction from changes in overall scores, and reports improved human-ranking agreement in its main comparisons.

1 Introduction

  • LLM judges can favor verbosity, answer positions, or their own generations, and can agree with one another while diverging from human judgments [ZCS+23] [WLC+24] [SML+25] [PBF24] [MHBS26].The introduction frames improved agreement with human judgments as relevant to evaluation and learning-system feedback.Source: §1 · p. 1
  • PRISM probes a judge by emphasizing or discounting attributes while holding content fixed, then uses score changes to learn a correction toward human judgments.The probe responses form a surrogate Jacobian motivated by a local sensitivity model; ridge regression fits the correction on human labels.Source: §1 · p. 2
  • For Jev and Sol on SummEval and HANNA, PRISM and α-PRISM significantly improve human-ranking agreement over Vanilla while remaining competitive with rubric-based SAJA and G-Eval.The introduction identifies SummEval summarization and HANNA story evaluation as the main comparison tasks.Source: §1 · p. 2

1.1 Related work

  • Unlike rubric-feature approaches such as SAJA, PRISM uses changes in overall scores under shifted emphasis rather than criterion-level ratings.Related work also distinguishes PRISM from methods that modify answers or instructions to diagnose bias, and from perturbation methods that alter evaluated content [YWH+25, LCA+25, RSG16, CWSGM24].Source: §1.1 · pp. 2–3, §1 · p. 2

2 Problem and method formulation

PRISM represents score responses to attribute-emphasis probes as empirical sensitivity features, then learns a shared residual correction from human-labeled examples. The section also examines whether probes provide repeatable signal, how wording strength and presentation order affect responses, and how those diagnostics relate to calibration.

2 Problem and method formulation

  • The prediction target is the expected human score for a fixed evaluation target and specified rater population, while Vanilla uses the judge’s baseline-prompt score without learned correction.Human and judge scores must be placed on comparable scales before fitting.Source: §2 · p. 4

2.1 Probe responses and residual calibration

  • The surrogate Jacobian keeps increased- and decreased-emphasis responses as separate columns, preserving second-difference information discarded by the paired half-difference.The local sensitivity interpretation treats measured contrasts as features directly because the wording assigns no numerical intervention strength.Source: §2.1 · p. 4, §2.2 · p. 7
  • The probe bank specifies attributes, instructions, and score contrasts; it is constructed on calibration data, with evaluation documents excluded from fitting the correction.The evaluated content, task, output format, and score scale remain fixed while emphasis changes.Source: §2.1 · p. 4
  • PRISM fits shared ridge-regression coefficients to human-labeled examples and subtracts the resulting response-based correction from a baseline score; new inputs require only baseline and probed outputs.The baseline coefficient is fixed at one, so an input with zero response to every probe retains its baseline score; α-PRISM instead fits the baseline coefficient jointly with the probe coefficients.Source: §2.1 · p. 5

2.2 Assessing the probe assumptions

  • For Jev’s 14-property bank, all attributes move beyond observed Vanilla repeat variation; the pooled probe-to-repeat RMS ratio is 2.51 [2.40, 2.63], with mean across-run response correlation 0.76.Every attribute’s 95% lower bound exceeds one; extractiveness has the largest reported ratio, 4.14, and hedging the smallest, 1.24.Source: §2.2 · p. 6, Figure 2 · p. 6
  • In Jev’s batched pointwise diagnostic, averaging eight candidate orders raises probe-response correlation to 0.78 and uncalibrated expert Spearman to 0.647, from 0.32 and 0.587 across orders and for one order, respectively.Averaging common orders reduces presentation sensitivity but does not establish permutation invariance.Source: §2.2 · p. 6, Figure 2 · p. 6
  • Across ten Jev runs, projected responses generally rise with emphasis for length, concrete detail, and formality, while extractiveness is non-monotone and hedging varies little across less-weight instructions.Pooled over six properties, shared-component RMS is 0.91, 0.89, and 0.61 times directional RMS at slight, standard, and strong wording, respectively; the wording categories are not numerical steps.Source: §2.2 · p. 7, Figure 3 · p. 7

3 Calibration performance and interpretation

Across SummEval and HANNA, PRISM variants improve agreement with human rankings over Vanilla and remain competitive with rubric-based methods. The fitted corrections vary across judges, while ablations show similar performance from style and rubric-derived probes.

3.1 Agreement with reference judgments

  • Across four judge-task settings, PRISM and α-PRISM significantly improve on Vanilla and achieve competitive agreement with rubric-based methods.Table 2 reports within-document or within-prompt Spearman correlation; the strongest method varies by judge and task.Source: Table 2 · p. 8, §3.1 · pp. 8–9
  • On four additional Jev benchmarks, Vanilla exceeds both SAJA variants, while PRISM has small further gains whose paired 95% intervals include zero.The benchmarks cover dialogue quality, helpfulness, conversational preferences, and correctness.Source: §3.1 · p. 9, Table 3 · p. 9

3.2 Correction profiles across judges

  • Jev and Sol both add the fluency response, but their inferred corrections for accuracy and grounding point in opposite directions.Positive coefficient components subtract the corresponding response; negative components add it.Source: §3.2 · p. 9

3.3 Ablations of the correction

  • Paired PRISM retains shared score changes, including second-difference information, discarded by the half-difference; its point estimate improves for Jev and is nearly unchanged for Sol.Both paired 95% gain intervals include zero, and adding the explicit Taylor constraint contributes little.Source: §3.3 · p. 9, Table 4 · p. 10
  • For Jev on SummEval, rubric-derived quality probes and style probes yield similar agreement, with little benefit from combining the banks.The compared banks each contain eight non-overlapping attributes.Source: §3.3 · p. 10, Table 4 · p. 10
  • Median gains typically rise and then level off as attributes are added, and mean gains persist after every single-attribute omission.Gains are evaluated across every probe subset of each size, with regularization selected on training data.Source: §3.3 · pp. 10–11, Figure 4 · p. 10
  • Jev’s agreement remains stable across moderate relative penalties, but approaches Vanilla as very large penalties shrink the correction toward zero.Main predictors select regularization on training data.Source: Figure 4 · p. 10, §3.3 · p. 11

4 Conclusion

PRISM uses attribute-emphasis score changes as a surrogate Jacobian and learns a ridge correction from human labels. Across SummEval and HANNA, it improves on Vanilla while remaining competitive with rubric-based methods; matched Jev–SummEval results also show similar agreement from style and rubric-derived quality probes.

4 Conclusion

  • PRISM represents changes in overall scores under attribute-emphasis instructions as a surrogate Jacobian and learns a ridge correction from human labels.Inference uses baseline and probed overall scores without requesting criterion-level judge ratings.Source: §4 · p. 11
  • On SummEval and HANNA, PRISM and α-PRISM significantly improve on Vanilla in the main Jev and Sol comparisons while remaining competitive with rubric-based SAJA and G-Eval.In the matched Jev–SummEval comparison, style probes yield similar agreement to rubric-derived quality probes.Source: §4 · p. 11
  • Learned coefficients expose shared and opposing correction directions across judges, making probe-response contributions interpretable.Source: §4 · p. 11

Limitations

The evidence is limited to fixed-input overall-score calibration, with transfer across tasks, model versions, and rater populations still untested. Probe calls add inference cost, and benchmark contamination cannot be ruled out; independent evaluation on newly collected data would strengthen generalization evidence.

Limitations

  • Transfer across tasks, model versions, and rater populations remains untested, and benchmark contamination cannot be ruled out.Evaluation on newly collected data would strengthen the evidence for generalization.Source: Limitations · p. 11
  • Additional probe calls incur inference cost, motivating comparisons under matched budgets and adaptive probe selection.Source: Limitations · p. 11
  • The experiments calibrate overall scores for fixed inputs, leaving biases from joint candidate presentation in batched pointwise scoring or listwise ranking for future study.The proposed next step is instruction probes that preserve candidate content.Source: Limitations · p. 11

A.3 Calibration and uncertainty

  • SummEval was used during method and probe development, so an independent benchmark is needed to assess generalization after that development process.The reported intervals are conditional on recorded outputs and fold assignment and are unadjusted for multiple comparisons.Source: §A.3 · p. 18

A.5 Additional benchmark protocols

The additional benchmark protocols specify the datasets, targets, splits, and probe-bank fitting used in the expanded evaluation. Comparisons use held-out contexts, but the benchmarks also inform method development, and MT-Bench splits can share questions across portions.

A.5 Additional benchmark protocols

  • The expanded study adds four benchmarks and a separate HANNA confirmation experiment with a development/confirmation split and expanded probe pool.The confirmation experiment is distinct from the full-dataset comparison in table 2.Source: §A.5 · p. 18
  • Across the five benchmarks, evaluation uses dataset-specific human-rating, win-rate, or binary-correctness targets; JudgeBench correctness is distinct from agreement with human ratings.HANNA targets average three raters across six criteria; USR targets mean Overall ratings, HelpSteer2 targets helpfulness, and MT-Bench training targets human win rate.Source: §A.5 · pp. 18–19
  • Probe screening, fitting, and hyperparameter selection use exploration data, while paired uncertainty intervals resample eligible confirmation contexts and condition on the fitted predictors.The probe bank and development protocol are not included in those intervals.Source: §A.5 · pp. 19–20
  • MT-Bench splits by question–turn, leaving 43 questions represented in both portions, so these results do not establish generalization to unseen questions or account for dependence between turns.The benchmarks also inform method development, so the comparisons assess the resulting designs rather than validation on a new sample.Source: §A.5 · p. 19
  • The expanded probe bank includes 52 properties, 20 reading contexts or situations, eight evaluator roles, five focus instructions, and four stakes instructions; probe units are screened against neutral clauses.Property probes contribute two oriented response columns, while one-sided instructions contribute one; relative ridge penalties are selected by five-fold context-grouped exploration validation.Source: §A.5 · p. 20

A.6 Data provenance and use

The appendix documents data provenance and examples, then tests whether probe responses are reproducible beyond score variation from repeated Vanilla calls or neutral instructions. Probe displacements exceed neutral-instruction displacements, though their magnitude varies across attributes and judge settings.

A.6 Data provenance and use

  • The study reuses English-language benchmark annotations for automatic-evaluation research and makes no claim of multilingual or demographic representativeness.Distribution notices do not establish separate rights to all underlying third-party material; the USR annotation download states no separate data license.Source: §A.6 · p. 20

B Additional probe-response diagnostics

  • The appendix presents further examples and diagnostics supporting the sensitivity measurements and local interpretation in section 2.2.The recorded prompt examples hold the article, summary, Vanilla score, and instruction template fixed while changing the attribute clause.Source: B Additional probe-response diagnostics · p. 21, §B.1 · p. 21

B.1 Recorded prompt examples

  • In the recorded coverage example, emphasizing coverage lowers the score, while de-emphasizing it raises the score with the article and summary held fixed.The example is one recorded response, not an average effect across attributes.Source: §B.1 · p. 21

B.2 Repeated scores and neutral instructions

  • For Jev on 100 SummEval documents, probe displacement is 2.54 times neutral-clause displacement, with property ratios from 1.25 for hedging to 4.19 for extractiveness.The neutral-clause RMS is 0.988 times the repeat reference; the probe-to-neutral ratio includes all 14 properties.Source: §B.2 · p. 21, Figure 5 · p. 22
  • Probe responses are reproducible across runs and exceed neutral-clause reproducibility, while their magnitude depends on judge setting: the probe-to-neutral RMS ratio is 1.96 for Jev and 1.15 for Sol.The vanilla score repeat correlation of 0.997 measures score stability, not displacement repeatability.Source: §B.2 · p. 21, Figure 6 · p. 22

B.3 Baselines and a local weight interpretation

The local-weight interpretation depends on how probe responses are referenced: Vanilla-baseline contrasts include effects of adding a control instruction. Reproducible response profiles are estimated across runs, and joint instructions show both alignment with single probes and non-additive interactions.

B.3 Baselines and a local weight interpretation

  • Vanilla-baseline contrasts combine the response to adding the control with changes relative to it, and the ablation retaining or dropping that component also changes model capacity.The shared component has 2.25, 2.26, and 1.64 times the reproducible RMS of the oriented response at slight, standard, and strong wording, respectively.Source: §B.3 · p. 23
  • The local sensitivity interpretation assumes smooth mean scores and small opposite weight changes; it is conditional because attribute-specific controls need not share a latent weight vector or reproduce Vanilla.Comparisons of wordings do not identify intervention size or establish that joint instructions compose the same weight changes as separate instructions.Source: §B.3 · p. 23

B.4 Estimating reproducible response profiles

  • Reproducible RMS and response correlation are estimated from products across distinct runs, with pooled property second moments and document-level bootstrap resampling.For the ten-run wording analysis, 45 unordered run pairs are averaged rather than treated as independent observations.Source: §B.4 · pp. 23–24

B.5 Joint instructions

  • Joint-instruction main effects align with single-probe half-differences at pooled projections of 0.81 for slight and 0.63 for standard wording, while interaction RMS is 0.30 and 0.34 of main-effect RMS.These measurements show the observed joint instructions are not additive and do not distinguish prompt-to-weight-map changes from curvature in the weight-to-score function.Source: §B.5 · p. 24

B.6 Presentation order and evaluation format

Batched candidate scoring is sensitive to presentation order, and averaging orders improves probe-response stability and uncalibrated expert agreement. Pairwise scoring remains below pointwise scoring in this experiment, even when both pair orders are averaged.

B.6 Presentation order and evaluation format

  • Reversing candidate order yields Jev baseline Kendall correlation of 0.57 versus 0.93 between repeats, while 93% of order-induced variance is an order-by-summary interaction.The estimated first-position shift is −0.16 on the 0–4 scale; additive position effects account for 2.7% of order-induced variance.Source: §B.6 · p. 24
  • Averaging eight candidate orders raises probe-response correlation across order sets to 0.78 and uncalibrated expert Spearman to 0.647, matching the pointwise estimate.Single-order batched pointwise scoring has mean expert Spearman of 0.587; from four orders onward, the paired interval for the difference from pointwise includes zero, which does not establish equivalence.Source: §B.6 · pp. 24–25
  • Before residual calibration, pairwise judgments score 0.077 lower in within-document Spearman than pointwise scoring with one order per pair, and 0.052 lower when both orders are averaged.Reversing each pair reduces but does not close the deficit in this experiment.Source: §B.6 · p. 25

B.7 Probe diagnostics across tasks

Across five benchmarks, probe responses show reproducible signal beyond neutral controls, while reversed presentation order affects scores more than repeating the same order. The full-judge comparison also notes uneven score-collection reliability.

B.7 Probe diagnostics across tasks

  • All five benchmarks have probe/control RMS ratios above two, and at the mildest wording level the reproducible sign-independent component is 0.50–0.99 times the directional component.Probes have higher repeat correlation than neutral clauses; these diagnostics support retaining both response components but do not predict calibration gains.Source: §B.7 · p. 25
  • Figure 8 compares Jev-1.13 pairwise judgments with pointwise scoring before calibration, using within-document Spearman differences and paired document-bootstrap intervals.The figure contrasts one presentation order per pair with averaging both orders; the caption supplies no outcome values.Source: Figure 8 · p. 26

C.1 Full judge comparisons

  • The four-judge comparison focuses on Jev and Sol amid uneven score collection; Gemma’s SummEval G-Eval needs substantial imputation, and Sol also needs SummEval fallbacks.Sol’s HANNA scores are complete under the response-format instructions; acquisition and missing-score treatment are specified in section A.4.Source: §C.1 · p. 26

C.2 Additional benchmark results

The additional results report paired comparisons and ablations of response representations and regularization. Taylor calibration is a constrained quadratic correction of the recorded responses, while learning the baseline coefficient yields small gains in the reported SummEval settings.

C.2 Additional benchmark results

  • Paired-accuracy intervals condition on the fitted predictor and selected bank; splits and acquisition details are specified in section A.5.The section also reports a separate HANNA confirmation study.Source: §C.2 · p. 26

D.1 Response representations and diagonal Taylor calibration

  • The Taylor predictor matches recorded scores at u = 0 and u = ±e_k, but the prompt wordings do not identify numerical steps or internal derivatives.It is an explicit quadratic model of observed contrasts without cross-attribute Hessian terms.Source: §D.1 · p. 28
  • Taylor-minus-paired 95% intervals span zero for Jev, Sonnet, Sol, and Gemma; paired-minus-half-difference intervals also span zero for Jev and Sol.All three predictors reuse the same saved responses.Source: §D.1 · p. 29

D.2 Regularization and the baseline coefficient

  • Learning α increases mean Spearman by 0.005–0.008 across four SummEval settings; Sol’s paired gain is 0.0065 with 95% interval [0.0010, 0.0127].The other three intervals include zero.Source: §D.2 · p. 29
  • Jev selects γ = 0.1 in all five folds; the other judges select different penalties across folds, with α left unpenalized in α-PRISM.The penalty is parameterized relative to mean squared training response; fixed-grid results are a sensitivity analysis, not the main-result selection.Source: §D.2 · p. 29

D.3 Robustness to probe attributes

In the matched Jev–SummEval comparison, style probes perform similarly to rubric-derived quality probes. The correction profiles show mostly stable coefficient directions, with interpretation constrained by the recorded probe bank and elicitation protocol.

D.3 Robustness to probe attributes

  • The style bank supplies correction signals without explicitly asking about rubric criteria, and its agreement is similar to the quality bank in this matched setting.The paired style-minus-quality gain is 0.005 with 95% interval [−0.008, 0.018]; the union adds little, and these differences are not a formal equivalence test.Source: §D.3 · p. 30, Table 18 · p. 31

D.4 Full correction profiles

  • The normalized correction entries describe component size, not additive fractions of explained variance, and remain dependent on correlated columns, the probe bank, and ridge penalty.The table reports five-fold means and ranges, not confidence intervals.Source: §D.4 · p. 32, Table 20 · p. 33
  • Across four judges, 24 of 30 directional coefficients retain their sign across all five folds; these approximate bias coordinates depend on the recorded bank and elicitation protocol.Figure 9’s positive coefficients subtract the response and negative coefficients add it; its bank and elicitation differ from Jev/Sol’s figure 4(a).Source: §D.4 · p. 32, Figure 9 · p. 32

D.5 Expanding and selecting the probe pool

Expanding the probe pool does not consistently improve calibration, even when added probes pass a response-signal screen. On HANNA, finer score resolution coincides with fewer ties and zero probe responses and higher PRISM agreement.

D.5 Expanding and selecting the probe pool

  • More probes do not consistently improve calibration, even after screening for reliable score movement.The comparison uses exploration labels and grouped cross-validation for penalty selection; the primary bank retains all signal-screened units.Source: §D.5 · p. 33, Table 21 · p. 34
  • For the expanded primary HANNA fit, within-context Spearman changes from 0.414 to 0.568.All 260 tested property units pass the neutral-control screen, but only 100 individually improve exploration nested-CV accuracy by more than 0.002; that count is not a held-out test of each unit.Source: §D.5 · pp. 33–34

D.6 Score resolution

  • For Sonnet on HANNA, changing from a 1–10 to 0–100 scale reduces tied pairs from 55.7% to 19.4% and zero probe responses from 80.6% to 55.7%.Spearman changes from 0.445 to 0.432 for Vanilla, 0.430 to 0.483 for PRISM, and 0.439 to 0.484 for α-PRISM; changed anchors and missing-response rates also belong to this comparison.Source: §D.6 · p. 34
Loading 2610.18389v1…