PaperlayerView exact revision on arXiv

Source-linked AI summary

What Pairwise Evaluation Misses: Reward Scores across Prompts

Mateusz Modrzejewski, Tomasz Radzikowski

arXiv:2610.31090v1cs.CLcs.AI

TL;DR

Pairwise evaluation leaves score components shared across responses to a prompt untested, although pooled cross-prompt decisions can depend on them. The paper audits between-prompt variation against human ratings and probes prompt mismatch across released scorers and matched training conditions. Released scorers show greater between-prompt variation than observed human ratings, while some of that variation is useful for global selection.

  • Problem

    Pairwise evaluation compares responses to the same prompt, so a score component shared by those responses cancels and remains untested despite the use of pooled scores across prompts.

    Source: §1 · p. 1

  • Method

    The paper audits between-prompt score variation against human ratings and uses sibling-rating regression and controlled prompt substitutions to assess agreement and wrong-prompt sensitivity.

    Source: §1 · p. 1, §3.2 · p. 3, §3.3 · p. 3

  • Results

    On HelpSteer2, six core reward models have between-prompt shares of 0.34 to 0.64, above the human reference; fixed generic prompts preserve clustering while weakening agreement with prompt-level human ratings on HelpSteer.

    Source: §1 · p. 2, §6 · p. 5

  • Takeaways & Limitations

    Reward evaluation should assess between-prompt variation, wrong-prompt sensitivity, and usefulness for selection separately, because clustering alone does not establish agreement with human judgments.

    Source: §11 · p. 8

    The audit uses English rating sets with only two to four responses per prompt, limiting the precision of between-prompt share estimates.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reward models score responses for preference optimization, filtering and ranking, and pooling scores across prompts needs them to be comparable. Yet pairwise evaluation compares two responses to the same prompt, so any score component shared by a prompt's responses cancels and goes untested. We study this component by comparing between-prompt variation in model scores with the same variation in human ratings of the same responses. We propose a human-referenced audit, built from a between-prompt variance share, a sibling-rating regression and prompt substitutions, and apply it to eighteen released scorers and six matched training conditions. We find that scores cluster by prompt more than the ratings do: on HelpSteer2, six reward models have between-prompt shares of 0.34 to 0.64 against a human share near zero. Replacing the original prompt with a fixed instruction keeps substantial clustering but weakens agreement with prompt-level human ratings, so clustering alone does not show agreement with humans. Part of that variation is useful: subtracting prompt means lowers the helpfulness of globally selected responses, and scores stay sensitive to a wrong prompt even when training shuffles prompt means. We propose that reward evaluation check between-prompt variation, sensitivity to a wrong prompt and usefulness for selection separately.

1 Introduction

Pairwise evaluation cannot test score components shared across responses to a prompt, even though those components can affect decisions that pool scores across prompts. The paper introduces a human-referenced audit and applies it to released scorers and matched training conditions.

1 Introduction

  • Pairwise comparisons leave shared per-prompt score components untested, although those components can change decisions that pool scores across prompts.Within-prompt uses such as best-of-n selection remain unchanged, while global reward thresholds can be affected.Source: §1 · p. 1
  • The audit compares between-prompt score variation with human ratings, tests agreement with sibling-rating regression, and probes wrong-prompt sensitivity through controlled substitutions.It is applied to eighteen released scorers and six training conditions at two sizes of one base-model family.Source: §1 · p. 1

3.1 The between-prompt share

  • A score’s between-prompt variance share is its intraclass correlation, compared with the same share in human ratings of the same responses.The one-way random-effects model separates prompt effects from response residuals; the share can also rise when within-prompt score differences are compressed.Source: §3.1 · p. 2, §2 · p. 2

3.2 Prompt substitutions

  • The generic-prompt condition scores each response under one fixed instruction while retaining its original prompt grouping; transplant conditions score it under wrong prompts.The transplant match margin compares reward under the response’s own prompt with mean reward under wrong prompts, in original-condition standard deviations.Source: §3.2 · p. 3

3.3 The sibling-rating regression

  • The sibling-rating regression predicts standardized reward from ratings of the shown response and a sibling response to the same prompt.A positive sibling coefficient indicates a larger association with prompt-mean ratings than with within-prompt deviations, but rating noise and sequential annotation can also produce it.Source: §3.3 · p. 3

3.4 Global selection and matched training

  • Prompt-centered pooling subtracts the mean score of sibling responses before cross-prompt comparisons, while matched training contrasts rating and preference objectives with controls that isolate supervision signals.The selection evaluation also centers using an independent pool of twelve generated responses per prompt.Source: §3.4 · pp. 3–4

4 Setup

The setup evaluates released scorers on three rating sets and uses a matched training comparison within the Qwen3 family. It also checks whether reproduced RewardBench scores and numerical precision agree with reported or original scores.

4 Setup

  • The analysis uses HelpSteer2, HelpSteer, and Reddit validation sets, with HelpSteer2 results also reported on its 356 single-turn prompts.HelpSteer2 includes 519 prompts and 1,038 responses; the HelpSteer split has 378 prompts with four responses each, and Reddit has 1,000 posts.Source: §4 · p. 4
  • The core analysis covers six 7B–8B reward models, with additional size ladders, generative judges, and larger released scorers for breadth.ArmoRM and DPA have documented training exposure to HelpSteer ratings.Source: §4 · p. 4
  • Seven models’ reproduced RewardBench or RewardBench 2 scores are within 1.0 point of published scores across all twelve comparisons, with a median gap of 0.35.ArmoRM’s combined score is reproduced within 1.2 points on RewardBench and 4.8 points below on RewardBench 2 when scoring its helpfulness head; fp32 and bf16 rewards correlate at 0.9996 or higher.Source: §4 · p. 4

5 Human references differ across rating sets

The human reference is the between-prompt share of ratings, reported separately for each rating set. The two sets differ in response source and rating protocol, and their influences cannot be separated.

5 Human references differ across rating sets

  • HelpSteer’s human share is 0.26 [0.20, 0.33], while HelpSteer2’s is −0.01 [−0.11, 0.10] on validation prompts.HelpSteer2 annotators rated two responses to each prompt in sequence; HelpSteer used independent ratings.Source: §5 · p. 4
  • The rating sets differ in response source and rating protocol, so the study reports their human references separately rather than separating those influences.HelpSteer’s reliability-adjusted estimate is treated as a sensitivity range, not a ceiling, because it relies on an assumed noise model and borrowed reliability.Source: §5 · p. 4
  • Reddit’s between-post vote share is context only because votes carry exposure.Source: §5 · p. 5

6 Between-prompt score variation

Across rating sets, released reward models generally have more score variation between prompts than the human ratings do. Prompt substitutions and controls qualify what that clustering means and how it relates to human judgments.

6 Between-prompt score variation

  • 0.34 to 0.64 is the between-prompt share for six core models on HelpSteer2 validation, above the same-prompt human reference in every interval.On single-turn prompts, the six core models range from 0.28 to 0.45 against an observed independent human reference of 0.26.Source: §6 · p. 5, Table 1 · p. 6
  • A fixed generic prompt leaves substantial clustering: five of six core models stay within 0.10 of their original share, while URM rises by 0.26 to 0.59.Generic-prompt and original-prompt means correlate 0.13 to 0.81 on HelpSteer2 and at most 0.54 on HelpSteer.Source: §6 · p. 5
  • Log-length adjustment changes the share by at most 0.05, despite removing 0.27 to 0.33 of between-prompt variance for Tulu-3, ArmoRM and DPA.The adjustment also removes substantial within-prompt variance.Source: §6 · p. 5
  • 0.45 to 0.65 is the original-prompt share for all eighteen HelpSteer scorers, above the observed human share of 0.26 with separated intervals.HelpSteer’s reliability-adjusted reference depends on borrowed reliability, and two of six core model shares fall within its sensitivity range.Source: §6 · p. 5
  • A high score share alone does not establish spurious variation: calibrated scores retain between-prompt variance above the human values, but explain only 14 to 40% of rating variance.On HelpSteer2, calibrated-score shares are 0.13 to 0.19 against a human value near zero; on HelpSteer they are 0.09 to 0.16 against 0.22.Source: §6 · p. 5

7 Prompt sensitivity and human agreement

  • On HelpSteer, every model’s score follows prompt-level human helpfulness more steeply than within-prompt rating deviations, with paired slope differences of 0.20 to 0.30.Under a generic prompt, the between-prompt slope falls to within 0.10 of zero for three of six models; wrong prompts weaken the shown-response rating association for every model.Source: §7 · pp. 5–6
  • Figure 2 compares model shares across original and generic prompts and after log-length adjustment, with rating-set-specific human references and the fraction of variance removed by adjustments.Source: Figure 2 · p. 6

8 Centering and global selection

Centering on prompt means can improve agreement with centered ratings while lowering agreement with raw ratings and reducing helpfulness in global selection. In this HelpSteer task, raw scores outperform full centering, and tuned partial centering shows no demonstrated improvement.

8 Centering and global selection

  • 0.14 to 0.21 points lower is the mean helpfulness of the 30% kept by sibling-centered scores versus raw scores for every core model.Centering lowers every model’s HelpSteer correlation with raw ratings by 0.08 to 0.15, while raising agreement against the centered rating.Source: §8 · p. 6
  • Figure 3 compares raw and sibling-centered score correlations against raw and centered ratings, then reports helpfulness changes for the top 30% selected responses.Panel (c) also compares sibling centering with centering on twelve independently generated reference responses.Source: §8 · pp. 6–7, Figure 3 · p. 7
  • 0.05 to 0.33 points lower is the kept-set helpfulness after centering on twelve independent reference responses, for every model.Held-out selection favors no or minimal centering; raw scores outperform full centering, and tuned partial centering shows no demonstrated improvement.Source: §8 · p. 7
  • A centering rule should be evaluated against the intended selection target, using reference responses independent of candidates when estimating prompt means.Source: §8 · p. 7

9 Effects of training objectives and targets

Matched training arms show that objectives and target information relate differently to wrong-prompt sensitivity, between-prompt shares, and sibling-rating coefficients. The paper situates these probes as complements to prior pairwise mismatch tests.

9 Effects of training objectives and targets

  • The regression arm has larger standardized random-wrong-prompt penalties than Bradley–Terry at both model sizes, while Bradley–Terry has lower pooled sibling coefficients under independent ratings.At 0.6B, penalties are 2.3 to 2.7 versus 1.6 to 2.1 standard deviations; at 1.7B, they are 2.5 to 2.9 versus 1.7 to 2.3. The regression arm’s between-prompt share is lower at both sizes.Source: §9 · p. 7, Figure 4 · p. 8
  • Shuffling prompt means across prompts leaves a 2.91-standard-deviation mismatch penalty at 1.7B while reducing the share to 0.20, versus 0.33 with true means and 0.46 without them.The findings distinguish mismatch discrimination from the between-prompt share: the former can remain large with wrong prompt means, while the share follows target information about prompt level.Source: §9 · pp. 7–8

10 Related work

  • Related work’s pairwise contrasts detect response-specific mismatch but cancel shifts shared by both responses; this paper adds fixed-response cross-prompt contrasts and human-referenced variation.Prior work also examines template offsets and prompt-level variance without human-rating references (Hasanaliyev et al., 2026).Source: §10 · p. 8

11 Conclusion

Pairwise accuracy leaves cross-prompt score comparisons untested, and the conclusion calls for evaluating between-prompt variation, mismatch sensitivity, and usefulness for selection separately. The paper also cautions that its estimates and downstream implications have specific scope limits.

11 Conclusion

  • Some prompt-level variation is useful: full centering lowers helpfulness in global selection, including with independent reference responses.Source: §11 · p. 8

Limitations

  • The estimates use English rating sets with only two to four responses per prompt, limiting the precision of between-prompt shares.Source: Limitations · p. 9
  • The matched training result covers one base family at two sizes and does not establish a general ordering across model families.Source: Limitations · p. 9
  • The downstream filtering comparison establishes neither a preference difference nor equivalence under its two judges.Source: Limitations · p. 9

Ethical considerations

  • Global filtering can change which prompt groups are represented, so applications should assess those changes against the intended quality criterion.Source: Ethical considerations · p. 9

A Prompt-shift invariance of pairwise objectives

Adding any prompt-constant component to a reward leaves pairwise comparisons and within-prompt rankings unchanged. The appendix explains why pairwise objectives therefore do not identify cross-prompt score levels, while rating regression penalizes shifts from rated prompt means.

B Experimental details

The experimental details describe the rating sets, scorer inventory, substitution and human-reference measures, and matched training and downstream evaluation setups. They also specify the scoring, fitting, and resampling procedures used in these analyses.

  • HelpSteer validation uses 378 prompts with exactly four rated responses, while HelpSteer2 uses 511 distinct first turns and 10,160 two-response training prompts.Source: B Experimental details · p. 11
  • Scorer inputs use dataset-specific chat formatting and truncation, and transplant prompts are sampled randomly or selected by similarity and length.Source: B Experimental details · p. 11
  • The scorer inventory records model parameters, training signal, documented dataset use, weights revision, and licence.Source: Table 2 · p. 12
  • The substitution protocol compares each prompt condition with scores under the original prompt; Table 4 reports observed and reliability-adjusted human helpfulness shares.Source: Table 3 · p. 12, Table 4 · p. 12
  • Table 5 reports judge between-prompt shares on expected digits, top-digit logits, and ranks; Table 6 describes paraphrase stability and a sensitivity calculation.Source: Table 5 · p. 13, Table 6 · p. 13
  • The matched training setup uses 6,667 HelpSteer2 preference pairs, five seeds per arm, and regression, Bradley–Terry, rating-margin, and binarized objectives.Source: B Experimental details · pp. 13–14
  • Probe analyses use grouped cross-fitting and prompt-resampled intervals, while partial centering is tuned across prompt folds and uses an independent response pool.Source: B Experimental details · p. 14
  • Leaderboard and filtering experiments score five policies on 356 prompts, with prompt mixes and judge-based comparisons specified in the protocol.Source: B Experimental details · p. 14

C Supporting results

Supporting analyses examine what prompt-level score variation captures, how it affects selection and ranking, and how robust it is across prompts, models, and training conditions. Results show both useful and potentially misleading variation, so prompt clustering alone does not establish agreement with human ratings.

Feature probes and variance decomposition

  • Across response probes, prompt probes, and crossed fits, substantial reward variation remained unexplained, while matching a response to its own prompt was the largest crossed-fit term.The own-prompt match term was 4 to 18 times the response effect; response-feature residual shares remained 0.26 to 0.55.Source: C Supporting results · pp. 14–15

Leaderboard, filtering and history checks

  • Raw and centered filters selected substantially different training sets (Jaccard 0.30), but downstream evaluation established neither a policy difference nor equivalence.The policies won 0.50 to 0.51 of judged comparisons against the base, and neither judge preferred one policy over the other.Source: C Supporting results · p. 15

Human references and rating reliability

  • HelpSteer2’s three-rater mean had reliability 0.73, while the approximate noise-corrected HelpSteer human share was 0.55.The authors report no corrected variance because, under independent additive noise, the between-prompt variance component itself is not attenuated.Source: C Supporting results · p. 15

Correlation with prompt-level human ratings

  • On HelpSteer2, whole-prompt score–human-mean correlations were 0.36 to 0.55, but disjoint-half correlations were within 0.08 of zero.Under a generic prompt, HelpSteer correlations fell to −0.03 to 0.15.Source: C Supporting results · p. 15

Sensitivity to prompt-set composition

  • On single-turn HelpSteer2 prompts, longer-prompt-half mean rewards were lower by 0.02 to 0.39 standard deviations for every released model, while human helpfulness differed by −0.05 [−0.19, 0.08].On all prompts the reward shift was 0.21 to 0.73; two balanced topic splits moved the mean by under 0.15.Source: C Supporting results · p. 15

Reference-pool size and shrinkage

  • On HelpSteer, centering each score on siblings reduced the top-30% kept set’s mean helpfulness, with the cost shrinking as the baseline grew.Centering on one, two, or three siblings cost 0.19 to 0.30, 0.14 to 0.23, and 0.14 to 0.21 points; an independent-pool shrinkage baseline cost 0.04 to 0.31 points.Source: C Supporting results · pp. 15–16

Scoring robustness

  • Removing the end-of-sequence marker cut random-prompt margins by 1.4 to 3.0 standard deviations for four core models, while removing the start marker changed margins by at most 0.12.The 24B judge reproduced the 70B judge’s five-policy ranking (Kendall’s tau 1.00; pairwise agreement 0.885 over 1,424 pairs).Source: C Supporting results · p. 16

Additional training conditions

  • At 1.7B, binarized-rating regression had pair accuracy 0.753 against 0.745 and 0.748, yet its mismatch margin was 1.95 and its share 0.13, the lowest among 1.7B arms.The mismatch margin was 0.81 below the regression arm, with every seed below every regression seed.Source: C Supporting results · p. 16

D Complete results and model comparison

The complete results compare released scorers and training conditions across between-prompt variation, human-rating relationships, prompt mismatch, selection, and robustness checks. The reported tables and figure define the comparisons and their measurement conventions, while several supporting checks caution against treating clustering as a standalone agreement measure.

Benchmark comparisons

  • Across scalar reward models, the share–RewardBench 2 Ties correlation was −0.61 over fourteen models, with interval [−0.96, −0.02]; the authors caution that these descriptive correlations involve dependent models.Correlations with RewardBench 1 and 2 were −0.46 and −0.47 over fifteen models, with intervals including or touching zero.Source: D Complete results and model comparison · p. 16

Model comparison

  • Figure 5 places eighteen released scorers and five seeds of each main matched arm by HelpSteer2 validation between-prompt share and mismatch penalty under four random wrong prompts.It shows 95% prompt-bootstrap intervals and HelpSteer2 and HelpSteer references; the HelpSteer reference uses a different response population.Source: D Complete results and model comparison · p. 16, Figure 5 · p. 17

Between-prompt shares and adjustments

  • The tables report human and model between-prompt shares from raw scores and pooled ranks on HelpSteer2 and HelpSteer, plus rank-minus-raw differences.Table 12 compares calibrated model between-prompt variance with observed human variance; Table 13 reports variance removed by adjustments and shares before and after.Source: Table 10 · p. 19, Table 11 · p. 20, Table 12 · p. 20

Prompt substitutions

  • The prompt-substitution tables compare original and generic-prompt means, between-prompt shares, and their concordance, including split-half estimates from disjoint responses.Table 14 also reports length adjustment and paired share differences; on HelpSteer2 the six core models’ disjoint-half prompt-level correlations were within 0.08 of zero.Source: Table 14 · p. 22

Selection and centering

  • Tables 15–17 compare raw and sibling-centered agreement and top-30% selection, and report helpfulness changes under several centering baselines.Table 17 includes leave-one-out, independent-pool, partial-centering, and shrinkage methods with paired prompt-bootstrap intervals.Source: Table 15 · p. 22, Table 16 · p. 22, Table 17 · p. 23

Regression, mismatch, and leaderboard measures

  • Tables 18–21 cover sibling-rating regression, mismatch penalties and match accuracy, scoring coverage, and unequal-coverage leaderboard comparisons.The regression tables report shown and sibling coefficients and between- and within-prompt slopes; Table 20 compares random and other prompt variants.Source: Table 18 · p. 24, Table 19 · p. 25, Table 20 · p. 26

History checks

Training and additional checks

  • Tables 25–28 report per-seed and five-seed matched-training results, standardized prompt-subset reward contrasts, transplant-condition mismatch margins and match accuracy, and mean absolute reward shifts under three paraphrases.Table 27 compares longer versus shorter prompt halves on all prompts, single-turn prompts, and single-turn prompts under the generic prompt; its topic contrast uses two balanced splits and its last row gives the human rating difference. Table 28 expresses paraphrase shifts in original-condition standard deviations.Source: Table 25 · p. 28, Table 26 · p. 29, Table 27 · p. 29

Related papers

Loading 2610.31090v1…