PaperlayerView exact revision on arXiv

Source-linked AI summary

HelpSteer2-Preference: Complementing Ratings with Preferences

Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, Yi Dong

arXiv:2410.01257v2cs.LGcs.AIcs.CL

TL;DR

Researchers lack evidence on whether Bradley-Terry or Regression reward models perform better when trained on appropriately matched data. The paper adds preference annotations and justifications to HelpSteer2, compares the approaches, and proposes a combined model; the matched variants perform similarly, while the combination reports strong benchmark results.

  • Problem

    Bradley-Terry and Regression reward models require different data formats, and there is a lack of empirical comparisons using appropriately matched data.

    Source: §1 · p. 2

  • Method

    The authors add preference annotations and human-written justifications alongside HelpSteer2 ratings, compare Bradley-Terry and Regression models, and propose combining the approaches.

    Source: §1 · p. 2

  • Results

    Helpfulness-Only and Scaled BT scored 92.7 vs 93.0 RewardBench Overall, while the combined approach scored 94.1 on RewardBench.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    The combined reward model can be used with REINFORCE to align an Instruct model, which reaches 85.0 on Arena Hard.

    Source: §1 · p. 2

    The collected preferences show a 5.1% Response 2 preference difference, which may reflect response sampling or annotator position bias.

    Source: §3 · p. 4

Abstract

from arXiv · show

Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. This reward model can then be used with REINFORCE algorithm (RLHF) to align an Instruct model to reach 85.0 on Arena Hard, which is No. 1 as of 1 Oct 2024. We open-source this dataset (CC-BY-4.0 license) at https://huggingface.co/datasets/nvidia/HelpSteer2#preferences-new -- 1-oct-2024 and openly release the trained Reward and Instruct models at https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward and https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct

1 INTRODUCTION

Reward models guide language models toward helpful and safe responses, but researchers lack evidence on whether Bradley-Terry or Regression training is better when compared with appropriately matched data. The paper releases preference annotations alongside HelpSteer2 ratings, compares the two approaches, and proposes a combined reward-modeling approach.

1 INTRODUCTION

  • The comparison gap reflects incompatible collection formats, while meaningful evaluation also requires identical prompts and responses and high-quality annotations.The authors note that post-hoc conversions from individual ratings may miss preferences between responses with equal Likert-5 scores, and that noisy labels can obscure differences between approaches.Source: §1 · p. 2
  • The paper releases a preference dataset with directions, strengths, and human-written justifications, described as the first open-source general-domain dataset with such justifications.The preference annotations were collected alongside HelpSteer2 Likert-5 ratings.Source: §1 · p. 2
  • The authors use the released data to compare Bradley-Terry and Regression reward models head-to-head, including models that use preference justifications.Source: §1 · p. 2
  • The proposed combination of Bradley-Terry and Regression reward modeling scores 94.1 on RewardBench and reaches 85.0 on Arena Hard when used with REINFORCE for alignment.The authors report the RewardBench score as the best-performing model as of 1 Oct 2024 and the Arena Hard score as No. 1 as of that date.Source: §1 · p. 2

2 DATASET

The dataset collection process pairs ratings of two responses with preference choices and justifications. Annotators rate responses on several dimensions before choosing among preference options.

2 DATASET

  • Annotators first rate each response on a Likert-5 scale for helpfulness, correctness and coherence, complexity, and verbosity.Source: §2 · p. 2
  • They then choose among seven preference options, each paired with a preference score and a justification.Source: §2 · p. 2
  • The listed options distinguish three degrees of preference for Response 1 over Response 2: slightly better (-1), better (-2), and much better (-3).Source: §2 · p. 3

3. Response 2 is much better than Response 1

The section describes how preference annotations were collected and filtered, and reports their agreement and relationship to HelpSteer2 helpfulness ratings. The analysis also notes a modest preference for Response 2 and finer-grained distinctions among responses with similar ratings.

3. Response 2 is much better than Response 1

  • Three to five annotators independently label each task; they must choose a preference unless both responses are invalid, and both-invalid responses are excluded.Source: §3 · p. 3
  • The dataset retains 7,118 preference pairs after selecting the three most similar annotations, filtering tasks with spread above two, and excluding zero averages.The retained pairs comprise 6,766 training and 352 validation examples.Source: §3 · p. 3
  • 0.878 Cohen’s κ after excluding zero preferences, up from 0.843 after outlier filtering and 0.489 raw, exceeds HelpSteer2 helpfulness agreement at 0.791.The final preference agreement is described as strong.Source: §3 · p. 3
  • 41.6% of tasks preferred Response 2 versus 36.5% preferring Response 1, with the preference distribution concentrated near zero.The analysis includes zero-preference samples, although they are not used for subsequent training.Source: §3 · p. 3
  • The 5.1% Response 2 preference difference may reflect response sampling or annotator order, but is smaller than the reported 25%–75% position bias in LLM-as-a-judge settings.Source: §3 · p. 4
  • 0.9024 Pearson’s R links preference distribution with helpfulness-score differences; 45.5% of equal-score pairs still drew a preference, and 58.0% of slight preferences accompanied a one-point score gap.Source: §3 · p. 4

3 REWARD MODEL

The section evaluates reward models on RewardBench and compares regression, Bradley–Terry, and justification-based training. Its methods include preference-strength-aware Bradley–Terry losses and regression initialization.

3.1 EVALUATION

  • RewardBench covers 2,985 tasks across Chat, Chat-Hard, Safety, and Reasoning, with overall accuracy averaging the four category scores.Reasoning accuracy weights math samples so math and coding contribute equally.Source: §3.1 · pp. 4–5

3.2 TRAINING

  • SteerLM Regression predicts five HelpSteer2 attribute scores from the end-of-response representation and minimizes mean squared error.The authors also train a model on helpfulness alone.Source: §3.2 · p. 5
  • Regular Bradley–Terry models learn a scalar reward and maximize the chosen-versus-rejected reward gap.Models project the end-of-response representation to a scalar reward.Source: §3.2 · p. 5
  • Margin BT incorporates preference strength, from 1 for slightly better to 3 for much better, as a margin in its loss.The preference-strength margin is placed inside the log-sigmoid function.Source: §3.2 · p. 5
  • Scaled BT places preference magnitude outside the log-sigmoid, increasing the update for pairs known to differ more.Unlike Margin BT, it does not require the reward gap to reach the margin.Source: §3.2 · p. 5
  • Bradley–Terry models initialized from helpfulness-only regression are also tested, because the regression model may provide a better starting point than the base model.Source: §3.2 · p. 5
  • Pairwise Justifier models generate a preference justification and statement from the prompt and two responses, then train with cross-entropy.The authors also test multiple justifications per task and response-order flipping with corresponding label updates.Source: §3.2 · pp. 5–6

4 REWARD MODEL RESULTS

The section compares reward-model performance and application fit across regression, Bradley–Terry, and justification-based approaches. The combined Scaled BT and ExPO model leads overall on RewardBench, while performance varies by category and intended use.

SteerLM Regression

  • 93.0 vs. 92.4 overall RewardBench: helpfulness-only SteerLM Regression slightly outperforms training on all five HelpSteer attributes.A single helpfulness score avoids conflicting objectives and the need to combine five predicted scores.Source: §4 · p. 6

Bradley-Terry (from scratch)

  • 92.7 vs. 91.5 overall RewardBench: Scaled BT performs better than Regular BT and Margin BT when trained from scratch.The authors attribute this likely to its use of preference-magnitude information.Source: §4 · p. 6

Complementing SteerLM Regression with Bradley-Terry

  • 92.7 vs. 93.0 overall RewardBench: optimized Scaled BT and helpfulness-only regression perform about equally, while their combination improves performance.The initialized Scaled BT model reaches 93.7; ExPO uses the regression model as the weak model and the initialized Scaled BT model as the strong model, with a final extrapolation factor of 1.52.Source: §4 · p. 7

Pairwise Justifier

  • Pairwise Justifier models score 90.0 or lower overall on RewardBench, while regression and Bradley–Terry models decompose comparison into simpler scoring objectives.The authors suspect the pairwise comparison formulation is harder to optimize.Source: §4 · p. 7
  • Training on multiple justifications raises overall score from 86.1 to 88.1%, mostly through Chat-Hard gains from 71.7 to 82.2%; flipping response order adds 1.3% overall.Preference-label-only training outperforms training on the statement alone or the full justification; results use a 6.6k-task, general-domain free-form dataset, so other data settings may yield different conclusions.Source: §4 · p. 7

Comparison with External Baselines

  • 94.1 vs. 93.8 overall RewardBench: Scaled BT + ExPO slightly exceeds Skywork-Reward-Gemma-2-27B, but trails it on Chat-Hard, 85.7 vs. 91.4.The combined model reaches 93.7 before ExPO; Table 1 reports RewardBench model performance.Source: Table 1 · p. 6, §4 · p. 7
  • On Chat-Hard, Scaled BT + ExPO is within 2.2% of Skywork on human-annotated subsets but trails by 10.8–19.2% on GPT-4-judged subsets.Chat-Hard includes LLMBar and MT-Bench data, with GPT-4 judgments used for some subsets and for MT-Bench ground truth.Source: §4 · p. 8, Table 2 · p. 8

When to use each type of reward models?

  • Regression is suited to SFT-data filtering, Bradley–Terry to RLHF, and Pairwise Justifier to human-in-the-loop evaluation; their strengths differ in calibration, interpretability, and inference cost.Regression supports absolute or relative score filtering, whereas Bradley–Terry supports relative filtering; justifiers can produce explanations but may require hundreds of generated tokens.Source: §4 · pp. 8–9, Table 3 · p. 8

5 ALIGNED MODELS

The section evaluates DPO, PPO, and REINFORCE alignment of Llama-3.1-70B-Instruct using HelpSteer2-Preference and a trained reward model. Most attempted algorithms improve over the base model, with REINFORCE outperforming PPO and the best aligned model competitive with frontier models on the reported benchmarks.

5.1 EVALUATION

  • Evaluation uses GPT-4-Turbo MT Bench, AlpacaEval 2.0 Length Controlled, and Arena Hard; mean MT Bench response characters indicate response length.MT Bench also serves as a validation metric for checkpoint selection.Source: §5.1 · p. 9

5.2 TRAINING

  • DPO uses HelpSteer2-Preference; PPO and REINFORCE use HelpSteer2-Preference prompts, with the Scaled BT + ExPO reward model for PPO and REINFORCE.PPO initializes its value model with the reward model; REINFORCE uses a KL-regularized reward and samples four responses per prompt for its leave-one-out baseline.Source: §5.2 · p. 9

5.3 RESULTS

  • Most attempted alignment algorithms improve over Llama-3.1-70B-Instruct, demonstrating the strength of the HelpSteer2-Preference dataset and trained reward model.Table 4 reports aligned-model performance across metrics, with higher better except for response length.Source: §5.3 · p. 9, Table 4 · p. 10
  • DPO variants consistently improve over the base model on MT-Bench and AlpacaEval 2.0 LC, but none beats PPO or REINFORCE across the three alignment metrics.Scaled DPO performs best among the three DPO variants.Source: §5.3 · p. 9
  • REINFORCE shows a marked advantage over PPO, which the authors associate with its leave-one-out baseline rather than PPO’s learned critic.They hypothesize that the baseline difference contributes to REINFORCE’s greater reward maximization and more stable training.Source: §5.3 · p. 10
  • The best REINFORCE model is competitive with GPT-4o and Claude 3.5 Sonnet across MT Bench, AlpacaEval 2.0 LC, and Arena Hard.Its longer responses still receive a high length-controlled AlpacaEval score, but the automated evaluations use GPT-4-Turbo as judge and may not support holistic comparisons across diverse use cases.Source: §5.3 · p. 10
  • Only REINFORCE among the models shown in Table 5 correctly answers the strawberry-count prompt.Source: §5.3 · p. 10, Table 5 · p. 10

ETHICS STATEMENT

This material describes how annotators compare responses, how preference justifications are prepared, and the ethical safeguards reported for data collection. The guidelines prioritize instruction following and correctness, while the examples illustrate preference strengths and invalid responses.

ETHICS STATEMENT

  • The collection excluded unsafe content and personally identifiable information; Scale AI conducted an ethical review and described worker pay and support safeguards.Worker support included channels for questions and concerns, while pay disputes were reviewed by trained specialists.Source: ETHICS STATEMENT · p. 11

A RELATED WORKS

  • Ranking studies find ratings and forced-choice rankings can capture complementary preference information; related datasets include general-domain and task-specific examples.The cited overview contrasts preference-pair datasets with datasets built around Likert ratings.Source: A RELATED WORKS · pp. 16–17

B.1 RANKING PRIORITIZATION

  • Annotators prioritize following the prompt’s instructions, then correctness; formatting and clarity guide preferences when higher-priority criteria do not decide the comparison.Requested formats count as instruction following, while unrequested formatting is a lower-priority consideration.Source: §B.1 · p. 17, §1 · p. 17, §2 · p. 17

B.2 RANKING STRENGTH

  • The preference scheme distinguishes slight, clear, and significant advantages, and includes a separate option for when neither response is valid.The strongest preference level is for major differences such as correctness or instruction-following failures.Source: §B.2 · p. 18, §1 · p. 18, §2 · p. 18

B.3 RANKING EXAMPLES

  • Examples show slight advantages for extra detail, stronger preferences for more substantive information, and invalid responses when both answers are wrong.One strong-preference example cites both failure to follow requested lowercase item markers and factual errors; the invalid-response example explains that the correct time difference depends on daylight-saving time.Source: §B.3.1 · p. 19, §B.3.2 · p. 19, §B.3.3 · p. 20

C JUSTIFICATION PRE-PROCESSING

  • Because free-text justifications vary in format, preprocessing splits them into sentences and separates preference statements from elaborations for more consistent use.Duplicate trailing preference statements and introductory phrases are removed during standardization.Source: C JUSTIFICATION PRE-PROCESSING · pp. 20–21
  • Justifications without an explicit preference statement were excluded rather than reconstructed, avoiding possible hallucinations or contradictory information.The authors judged that retaining the additional 7.6% of justifications did not warrant that risk.Source: C JUSTIFICATION PRE-PROCESSING · p. 21
  • The Overall Preference Justification set contains 6,618 samples, while the All Justifications dataset contains 24,698 justifications.The Overall set has 6,287 training and 331 validation samples; the All Justifications set has 23,487 training and 1,211 validation justifications.Source: C JUSTIFICATION PRE-PROCESSING · p. 21, Table 6 · p. 21

D PREFERENCE JUSTIFICATION ANALYSIS

The justification analysis finds that annotators commonly compare both responses and emphasize helpfulness and correctness. It also describes the training setups and evaluation procedures used for reward models and aligned models.

D PREFERENCE JUSTIFICATION ANALYSIS

  • 83% of justifications mention both responses, and preference statements favor Response 2 by 6–7%.Justifications average four sentences or about 500 characters; nearly all full justifications mention both responses.Source: D PREFERENCE JUSTIFICATION ANALYSIS · p. 21
  • 12.9% of justifications contain correctness-related keywords and 11.0% helpfulness-related keywords; the analysis explains less than 32.0% through its selected keywords.The word-level analysis draws on the 500 most frequent words, which the authors note may miss relevant expressions.Source: D PREFERENCE JUSTIFICATION ANALYSIS · p. 22
  • LLM classification preserves the attribute ordering: helpfulness and correctness lead, followed by coherence and verbosity, with complexity least mentioned.The LLM-based analysis finds a higher proportion of justifications mentioning each attribute than the low-recall word-level analysis.Source: D PREFERENCE JUSTIFICATION ANALYSIS · p. 22
  • 90.1% of justifications mention HelpSteer-adjacent factors, compared with 5.9% for stylistic factors and 1.4% for sycophantic factors.The authors suggest that this pattern reflects guidance to focus on response substance.Source: D PREFERENCE JUSTIFICATION ANALYSIS · p. 22

E TRAINING HYPER-PARAMETERS

  • Regression models used two epochs, while Bradley–Terry models used one because additional training overfit and worsened validation loss and RewardBench performance.Pairwise Justification models also used one epoch; initial two-epoch experiments changed results minimally.Source: E TRAINING HYPER-PARAMETERS · p. 23
  • PPO used two rounds and checkpoint selection amid training instability; REINFORCE used four samples per prompt and a leave-one-out baseline.The paper plots reward curves for both algorithms.Source: E TRAINING HYPER-PARAMETERS · p. 24, I REWARD CURVES · p. 25

F COMPUTE REQUIREMENTS

  • Compute requirements are reported in H100-equivalent node-hours, with three A100 node-hours converted to one H100 node-hour.Experiments used nodes with eight A100/H100-80GB SXM GPUs.Source: Table 9 · p. 24

H ALIGNED MODEL EVALUATION DETAILS

  • Aligned-model evaluation covers MT Bench, AlpacaEval 2.0 Length Controlled, and Arena Hard, using judge-model scoring or comparisons with GPT-4 baselines.MT Bench uses GPT-4-Turbo scoring; AlpacaEval applies a length correction; Arena Hard reports win rate against GPT-4-0314.Source: H ALIGNED MODEL EVALUATION DETAILS · pp. 24–25

J FURTHER ANALYSIS FOR SCALED BRADLEY TERRY

Scaled Bradley–Terry loss weights preference strength in addition to whether the model ranks the chosen response above the rejected response. Its reported loss comparisons emphasize stronger penalties for major errors while keeping losses low for correct predictions.

J FURTHER ANALYSIS FOR SCALED BRADLEY TERRY

  • Scaled BT keeps loss below 1 on the representative correct predictions, while Margin BT reaches 2.1269 for a correct prediction with ∆r=1 and m=3.The authors warn that a high loss on a correct prediction could lead optimization away from that prediction.Source: J FURTHER ANALYSIS FOR SCALED BRADLEY… · pp. 25–26, Table 10 · p. 26
  • For the drastic mistake ∆r=-3 and m=3, Scaled BT’s loss is 9.1458 versus 6.0025 for Margin BT, although their eight-scenario average losses are similar.The reported averages are 2.3619 for Scaled BT and 2.4767 for Margin BT.Source: J FURTHER ANALYSIS FOR SCALED BRADLEY… · p. 26, Table 10 · p. 26
  • The authors argue Scaled BT may be more robust to annotation errors because it assigns higher losses to strong-preference samples, where annotator errors are less common.They contrast these with slight-preference samples, where they say errors are more common.Source: J FURTHER ANALYSIS FOR SCALED BRADLEY… · p. 26

K REWARD MODEL ABLATION STUDIES

Reward-model ablations favor filtered, higher-quality data and preference annotations collected for pairwise training; in aligned-model tests, HelpSteer2-Preference performs better than HelpSteer2 across the reported benchmarks.

K REWARD MODEL ABLATION STUDIES

  • No setup in the reward-model ablations exceeded the Helpfulness Only SteerLM Regression baseline.The comparison used similar training settings and hyperparameter search across ablations.Source: K REWARD MODEL ABLATION STUDIES · p. 26, Table 11 · p. 27
  • 93.7 RewardBench fell to 92.7 without filtering wide-spread samples, then to 92.3 when using all annotations instead of the three most similar.The authors interpret these results as support for training on a smaller, higher-quality dataset with noisy samples removed.Source: K REWARD MODEL ABLATION STUDIES · p. 27
  • 42 prompts matched RewardBench under a high-recall, low-precision 7-gram definition, often because of shared templates; this was 0.42% of HelpSteer2-Preference and 1.4% of RewardBench.The authors repeated training and obtained 93.3 RewardBench, compared with 93.7 for the original model, and judged the overlap impact minimal.Source: K REWARD MODEL ABLATION STUDIES · p. 27
  • HelpSteer2-Preference reached 93.7 RewardBench, versus 92.9 for pairwise-converted HelpSteer2 and 93.0 for the original Helpfulness-only model.The authors conclude the two datasets contain complementary label information useful in a two-stage reward-model training process.Source: K REWARD MODEL ABLATION STUDIES · p. 27
  • 92.2 RewardBench for helpfulness-only Regression initialized with Scaled Bradley-Terry was below 93.0 for training Regression alone.The authors suggest the Bradley-Terry reward range of -35 to 5 can produce initially large losses against helpfulness targets from 0 to 4.Source: K REWARD MODEL ABLATION STUDIES · pp. 27–28

L ALIGNED MODEL ABLATION STUDIES

  • Across DPO and PPO, HelpSteer2-Preference aligned models better than HelpSteer2 on MT Bench, AlpacaEval 2, and Arena Hard.The PPO model using the HelpSteer2-trained reward model did not benefit from two-stage PPO training, unlike the Scaled BT + ExPO model.Source: L ALIGNED MODEL ABLATION STUDIES · p. 28

M FURTHER REINFORCE MODEL METRICS

REINFORCE training leaves performance on MMLU, GSM8K, and HumanEval little changed relative to the starting Llama 3.1 70B Instruct model.

M FURTHER REINFORCE MODEL METRICS

  • Performance on MMLU, GSM8K, and HumanEval changed little after REINFORCE training compared with Llama 3.1 70B Instruct.The authors interpret this as suggesting REINFORCE does not cause catastrophic forgetting of capabilities gained from prior training.Source: Table 13 · p. 28, M FURTHER REINFORCE MODEL METRICS · p. 28

Related papers

Loading 2410.01257v2…