PaperlayerView exact revision on arXiv

Source-linked AI summary

The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models

Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, Xiaoyu Shen

arXiv:2410.06554v2cs.CLcs.AI

TL;DR

The paper asks whether higher reward-model accuracy necessarily improves language models, a key question for aligning their outputs. It tests reward models across QA-FEEDBACK tasks and finds that moderate-accuracy models yield the best language-model performance, while noting limits in dataset and model scope.

  • Problem

    The paper examines whether higher reward-model accuracy improves language-model performance, a question relevant to RLHF alignment.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The study tests Longformer-based reward models on QA-FEEDBACK relevance, factuality, and completeness tasks, then evaluates trained language models with independent high-accuracy reward models.

    Source: §1 · p. 1, §2 · p. 2, §3.1 · p. 3

  • Results

    Language models trained with moderately accurate reward models achieve the best performance, with the reported pattern extending across T5-small, T5-base, and T5-large.

    Source: §1 · p. 1, §3.2 · p. 3, §4 · p. 5

  • Takeaways & Limitations

    Within the studied setting, moderate accuracy can provide more task-aligned feedback and support balanced, stable training and better generalization.

    Source: §4 · p. 5

    The findings rely on QA-FEEDBACK’s long-form factual inquiries, so generalizability to conversational and other question-answering datasets remains to be validated.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback significantly enhances Natural Language Processing by aligning language models with human expectations. A critical factor in this alignment is the strength of reward models used during training. This study explores whether stronger reward models invariably lead to better language models. In this paper, through experiments on relevance, factuality, and completeness tasks using the QA-FEEDBACK dataset and reward models based on Longformer, we uncover a surprising paradox: language models trained with moderately accurate reward models outperform those guided by highly accurate ones. This challenges the widely held belief that stronger reward models always lead to better language models, and opens up new avenues for future research into the key factors driving model performance and how to choose the most suitable reward models. Code and additional details are available at https://github.com/EIT-NLP/AccuracyParadox-RLHF.

1 Introduction

The introduction challenges the assumption that more accurate reward models necessarily improve language-model performance. Experiments suggest that moderate accuracy, task-appropriate rewards, and balanced training can yield better outcomes.

  • RLHF incorporates human feedback to address fine-tuning’s exposure bias and limited sequence-level reward optimization.Traditional fine-tuning trains on ground-truth data rather than models’ own predictions.Source: §1 · p. 1
  • The study challenges the common assumption that higher reward-model accuracy improves language-model performance through more precise training feedback.Source: §1 · p. 1
  • Language models perform best with moderately accurate reward models, challenging the assumption that higher reward-model accuracy directly improves outcomes.The experiments cover relevance, factuality, and completeness using QA-FEEDBACK and Longformer-based reward models.Source: §1 · p. 1
  • The authors identify moderate accuracy and balanced training as beneficial, contrary to the view that higher accuracy is invariably better.Source: §1 · p. 1
  • Moderately accurate reward models provide more task-appropriate rewards than the most accurate models.Source: §1 · pp. 1–2
  • Moderately accurate reward models support balanced, stable training and better generalization, according to the study’s analysis of KL divergence trends.Source: §1 · p. 2

2 Motivation and Problem Settings

The study asks how reward-model strength relates to language-model performance and whether an optimal accuracy range exists. It investigates this relationship across factuality, relevance, and completeness tasks, using test-set classification accuracy to characterize reward-model strength.

  • The study seeks an optimal range of reward-model accuracy that maximizes language-model performance across NLP tasks.Source: §2 · p. 2
  • Reward-model strength is defined as binary classification accuracy on test sets, while language-model performance is measured with independent high-accuracy reward models.The evaluation focuses on factuality, relevance, and completeness.Source: §2 · p. 2
  • The study analyzes how reward-model classification accuracy and the number of training steps relate to language-model performance.Source: §2 · p. 2

3 Experiment and Results

Across relevance, factuality, and completeness tasks, the experiments found that reward models of moderate accuracy often yielded better language-model performance than the most accurate models. The analysis compares reward behavior and KL-divergence patterns to examine how these models differ during training.

3.1 Basic Experimental Setup

  • Experiments used SFT T5-small, T5-base, and T5-large models, Longformer-based reward models, PPO training, and independent high-accuracy reward models for evaluation.The tasks were relevance, factuality, and completeness, using QA-FEEDBACK; a KL threshold interrupted training when policy divergence exceeded it.Source: §3.1 · pp. 2–3

3.2 Are High-Accuracy and Deeply Trained Reward Models Always the Best?

  • Moderate-accuracy reward models with appropriate training steps produced optimal performance across relevance, factuality, and completeness, with similar trends across T5 model sizes.The T5-small figures show the task results; the text reports consistent trends for T5-base and T5-large.Source: §3.2 · p. 3, Figure 1 · p. 3, Figure 2 · p. 3

3.3 How Do Best and Most Accurate Reward Models Differ?

  • Best-performing models gave higher, more variable rewards for relevance; higher-mean, less-variable rewards for factuality; and lower-mean, more-variable rewards for completeness.The primary analysis used T5-small; the text reports similar trends for T5-base and T5-large.Source: §3.3 · p. 4, Figure 4 · p. 4, Figure 5 · p. 4
  • For relevance and factuality, reward variability was described as encouraging broader response exploration and improving generated-text quality.For completeness, the analysis says greater variability helps ensure thorough text evaluation.Source: §3.3 · p. 4

3.4 How Do Best and Most Accurate Rewards Impact Models?

  • The best reward model showed lower KL divergence and variance for relevance, higher mean but lower variance for factuality, and higher mean and variance for completeness.The plotted KL trends are for T5-small; similar trends were reported for T5-base and T5-large.Source: §3.4 · p. 5, Figure 7 · p. 5, Figure 8 · p. 5
  • The analysis characterizes typically moderate-accuracy best-performing reward models as balancing training stability and adaptability across the three tasks.The reported T5-base and T5-large trends were consistent with T5-small and were interpreted as balancing overfitting and underfitting.Source: §3.4 · p. 5

4 Conclusion and Future Work

The study concludes that moderately accurate reward models yield optimal language-model performance, with task-aligned feedback and balanced, stable training associated with better generalization. It calls for further investigation of reward-model overfitting and out-of-distribution robustness.

4 Conclusion and Future Work

  • Moderately accurate reward models yield optimal language-model performance, challenging the belief that higher accuracy is always more beneficial.Source: §4 · p. 5
  • Task-aligned feedback and balanced, stable training from moderately accurate reward models promote better generalization.Source: §4 · p. 5
  • Future work should examine reward-model overfitting on out-of-distribution tasks and investigate regularization, data augmentation, and explicit OOD evaluation.Source: §4 · p. 5

Limitations

The study’s conclusions are bounded by its QA-FEEDBACK data, T5 model family, and single reward-model architecture. The authors call for validation across additional datasets, model architectures, and reward-model designs.

Limitations

  • QA-FEEDBACK focuses on long-form responses to factual inquiries, so results need validation on conversational and other question-answering datasets.Source: Limitations · p. 5
  • Evaluation used T5 models of different scales; broader architectures such as Llama2 remain to be examined for robustness.Source: Limitations · pp. 5–6
  • Reward models used a single architecture, leaving the effects of reward-model size, capacity, and design unexplored.Source: Limitations · p. 6

A Supplementary 3D Surface Plots

Supplementary surface plots show that moderate reward-model accuracy or strength yields the best performance across relevance, factuality, and completeness for T5-base and T5-large.

A Supplementary 3D Surface Plots

B Supplementary Reward Analysis

Supplementary reward-analysis figures plot training steps against rewards and show mean and variance for relevance, factuality, and completeness in T5-base and T5-large.

B Supplementary Reward Analysis

C Supplementary KL Divergence Analysis

The supplementary KL-divergence analysis presents training-step plots and mean-and-variance summaries for relevance, factuality, and completeness, separately for T5-base and T5-large.

C Supplementary KL Divergence Analysis

  • For relevance, Figures 22 and 25 plot KL divergence against training steps and show mean and variance for T5-base and T5-large, respectively.Source: Figure 22 · p. 9, Figure 25 · p. 9
  • For factuality, Figures 23 and 26 plot KL divergence against training steps and show mean and variance for T5-base and T5-large, respectively.Source: Figure 23 · p. 9, Figure 26 · p. 9
  • For completeness, Figures 24 and 27 plot KL divergence against training steps and show mean and variance for T5-base and T5-large, respectively.Source: Figure 24 · p. 9, Figure 27 · p. 9

D Hyperparameter Settings

The hyperparameter settings are presented in five tables covering model, reward-model, environment, PPO-training, and training-procedure configurations.

D Hyperparameter Settings

Related papers

Loading 2410.06554v2…