PaperlayerView exact revision on arXiv

Source-linked AI summary

The Trickle-down Impact of Reward (In-)consistency on RLHF

Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, Dong Yu

arXiv:2309.16155v1cs.CLcs.LG

TL;DR

Reward models may fail to adapt their rewards when instruction meaning changes, and the paper examines how that inconsistency affects RLHF. It introduces a benchmark and two consistency-improvement techniques; standard models perform near chance against humans, while more consistent models yield more useful RLHF responses.

  • Problem

    The paper studies whether reward models recognize semantic changes between instructions and adapt their rewards, and how inconsistency affects downstream RLHF.

    Source: §1 · p. 1

  • Method

    The paper introduces Contrast Instructions to test matching between similar instructions and different ground-truth responses, plus ConvexDA and RewardFusion for training- and inference-stage consistency improvements.

    Source: §1 · p. 2

  • Results

    More consistent reward models yield more useful RLHF responses, while standard ranking-trained models perform near chance on Contrast Instructions compared with humans’ approximately 80% correct rankings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Reward consistency offers an intrinsic evaluation perspective for preference-based reward models, while the question of which reward modeling best benefits RLHF remains open.

    Source: §1 · p. 2, §9 · p. 9

    Contrast Instructions does not necessarily assess all capabilities required of a reward model.

    Source: §7.4 · p. 9

Abstract

from arXiv · show

Standard practice within Reinforcement Learning from Human Feedback (RLHF) involves optimizing against a Reward Model (RM), which itself is trained to reflect human preferences for desirable generations. A notable subject that is understudied is the (in-)consistency of RMs -- whether they can recognize the semantic changes to different prompts and appropriately adapt their reward assignments -- and their impact on the downstream RLHF model. In this paper, we visit a series of research questions relevant to RM inconsistency: (1) How can we measure the consistency of reward models? (2) How consistent are the existing RMs and how can we improve them? (3) In what ways does reward inconsistency influence the chatbots resulting from the RLHF model training? We propose Contrast Instructions -- a benchmarking strategy for the consistency of RM. Each example in Contrast Instructions features a pair of lexically similar instructions with different ground truth responses. A consistent RM is expected to rank the corresponding instruction and response higher than other combinations. We observe that current RMs trained with the standard ranking objective fail miserably on Contrast Instructions compared to average humans. To show that RM consistency can be improved efficiently without using extra training budget, we propose two techniques ConvexDA and RewardFusion, which enhance reward consistency through extrapolation during the RM training and inference stage, respectively. We show that RLHF models trained with a more consistent RM yield more useful responses, suggesting that reward inconsistency exhibits a trickle-down effect on the downstream RLHF process.

1 INTRODUCTION

The paper studies whether reward models consistently match responses to semantically appropriate instructions and how this affects downstream RLHF. It introduces a consistency benchmark, reports that standard reward models perform near chance while humans reach about 80%, and presents methods to improve consistency without extra training cost.

1 INTRODUCTION

  • Contrast Instructions measures reward consistency using lexically similar instructions with different ground-truth responses and tests whether models score corresponding instruction-response pairs above mismatched combinations.The benchmark is proposed as an intuitive, scalable evaluation strategy.Source: §1 · p. 2
  • Standard ranking-trained reward models perform near chance on Contrast Instructions, while humans rank responses correctly in approximately 80% of cases.The benchmark is constructed from four open-source human-preference or instruction-tuning datasets.Source: §1 · p. 2
  • ConvexDA and RewardFusion improve reward consistency during reward-model training and inference, respectively, without extra training cost.The reported improvements still leave a large gap to human performance.Source: §1 · p. 2
  • RLHF models trained with more consistent reward models generate more useful responses.The paper describes reward inconsistency as a trickle-down effect on downstream chatbot accuracy and usefulness.Source: §1 · pp. 1–2

2 PRELIMINARIES: REWARD MODELING FOR RLHF

In RLHF, a reward model assigns scalar rewards to generated outputs and is trained on human preference pairs to score preferred responses higher in their instruction context. The reinforcement-learning stage optimizes the policy for reward while applying a per-token KL penalty relative to the supervised fine-tuned model.

2 PRELIMINARIES: REWARD MODELING FOR RLHF

  • A reward model is a scalar function used to train the generative model during RLHF, supervised by a dataset of human preferences.Each preference instance includes an instruction and a pair of responses, with one preferred by humans.Source: §2 · p. 3
  • The reward model is trained by minimizing a ranking loss that encourages higher scores for human-preferred responses in the instruction context.The compared responses are paired with the same instruction.Source: §2 · p. 3
  • PPO-based RL training maximizes an objective that rewards higher-scoring responses and includes a per-token KL penalty from the supervised fine-tuning model.The value function is initialized from the reward model, and an optional term regularizes toward responses likely under the pretraining dataset.Source: §2 · p. 3

3 CONTRAST INSTRUCTIONS: MEASURING REWARD (IN-)CONSISTENCY

Contrast Instructions evaluates whether reward models preserve instruction-response correspondence when similar-looking instructions have different meanings and preferred answers. The benchmark measures both response and instruction consistency and can be constructed automatically from preference datasets.

3 CONTRAST INSTRUCTIONS: MEASURING REWARD (IN-)CONSISTENCY

  • Reward models can overfit to their training distribution, producing inference-time inconsistencies between their predictions and human preferences.The standard objective trains models to distinguish response preferences within an instruction context.Source: §3 · p. 3
  • Contrast Instructions pairs lexically similar instructions with different semantics and their respective human-preferred responses; a consistent model scores matching pairs above mismatches.The benchmark compares a matching pair against alternatives formed by swapping instructions or responses.Source: §3 · p. 3
  • Response consistency measures whether a model scores the preferred response above its distractor for a given instruction.This metric resembles the response-ranking objective used to train standard reward models.Source: §3 · pp. 3–4
  • Instruction consistency measures whether a model scores the matching instruction above a distractor when given a response.Unlike response consistency, this capability is not explicitly trained by the standard ranking objective.Source: §3 · pp. 3–4

3.1 CONSTRUCTING CONTRAST INSTRUCTIONS AUTOMATICALLY

  • Contrast Instructions can be constructed automatically by pairing lexically similar instructions whose ground-truth responses differ.The construction aims to make models recognize nuanced semantic differences and match each instruction to its answer.Source: §3.1 · p. 4
  • The benchmark uses StackExchange, WMT, RealSumm, and Twitter datasets, selecting instruction pairs with SimCSE cosine similarity from 0.75 to 0.9.The four datasets cover question answering, machine translation, summarization, and paraphrase generation, respectively.Source: §3.1 · p. 4
  • Table 1 presents statistics and examples for the four resulting Contrast Instructions benchmark datasets.The caption identifies the table's scope as dataset statistics and examples.Source: Table 1 · p. 4

4 INCONSISTENCY OF EXISTING REWARD MODELS

The evaluation tests standard ranking-objective reward models on Contrast Instructions, where they must distinguish reward assignments across similar instructions. These models perform near chance on both consistency measures, far below human performance.

4 INCONSISTENCY OF EXISTING REWARD MODELS

  • The evaluation finetunes LLaMa-7B reward models on four human-preference datasets, including a multi-task mixture, and compares them with majority votes from three annotators.Human judgments use 100 randomly selected data points.Source: §4 · p. 5
  • Standard 7B reward models perform close to random guessing on both response consistency (Cres) and instruction consistency (Cins), far below estimated human performance.The results are averaged across four Contrast Instructions benchmarks.Source: §4 · p. 5, Table 2 · p. 5
  • Multi-task training does not seem to help, possibly because the tasks have limited commonalities; models perform slightly better on Cres than Cins.The authors suggest the ranking objective more closely resembles Cres than Cins.Source: §4 · p. 5

5 ENHANCING REWARD MODEL CONSISTENCY

The paper proposes training- and inference-stage techniques to improve reward-model consistency without increasing training cost. Both improve consistency, their combination works best, but substantial distance from human performance remains.

5.1 CONSISTENCY-INDUCING TRAINING VIA CONVEXDA

  • ConvexDA augments preference examples with response-word substitutions and replaces each original input with a representative augmented example, without changing the objective or increasing training cost.The selection uses a vertex of the augmented examples’ convex hull in embedding space.Source: §5.1 · pp. 5–6

5.2 CONSISTENCY-INDUCING INFERENCE VIA REWARDFUSION

  • RewardFusion estimates a response’s reward by taking a weighted average of its reward and rewards for similar retrieved training examples.At inference, retrieval uses similarity between instruction-response pairs.Source: §5.2 · p. 6

5 ENHANCING REWARD MODEL CONSISTENCY

  • The experiments apply ConvexDA and RewardFusion to single-task reward models trained on each human-preference dataset and evaluate consistency on four benchmarks.The methods are designed to be agnostic to Contrast Instructions’ format and setup.Source: §5 · p. 5, §5.3 · p. 6

5.3 EXPERIMENTS AND RESULTS

  • Combining ConvexDA and RewardFusion improves reward-model consistency most, but performance on Contrast Instructions remains far below human performance.The methods target training and inference, respectively, and do not add training cost.Source: §5 · p. 5, §5.3 · p. 6, Table 3 · p. 6

6 TRICKLE-DOWN EFFECT OF REWARD INCONSISTENTENCY ON RLHF

The paper compares downstream RLHF models trained with standard and more consistent reward models. Human and automatic evaluations favor the more consistent model, with the clearest fine-grained human improvement in usefulness.

6 TRICKLE-DOWN EFFECT OF REWARD INCONSISTENCY ON RLHF

  • The human evaluation compares two RLHF models trained with standard versus Contrast Ft. reward models on 500 StackExchange test questions.Contrast Ft. denotes a reward model finetuned on Contrast Instructions-format examples sampled from StackExchange training data.Source: §6 · pp. 6–7
  • Human evaluation finds higher generation quality for the RLHF model trained with the more consistent reward model.The evaluation assesses individual response acceptability and pairwise preference between models.Source: §6 · p. 7, Figure 2 · p. 7
  • On MT-Bench, more consistent reward models yield RLHF responses rated more preferable under both one-turn and multi-turn settings.GPT-4 grades responses on a scale from 1 to 10.Source: §6 · p. 7, Table 4 · p. 7

6 TRICKLE-DOWN EFFECT OF REWARD INCONSISTENTENCY ON RLHF

  • Human raters find a statistically significant improvement in usefulness (p < 0.01), but minimal impact on relevance or factuality.Usefulness tends to decrease further into standard-RM responses, an issue mitigated with a more consistent RM.Source: §6 · p. 7, Table 5 · p. 7

7 DISCUSSION

Standard RM evaluation does not reflect improvements in reward consistency, while closer analysis finds greater reward-score variance for responses with similar human scores. The discussion also notes that Contrast Instructions covers only part of RM consistency and robustness.

7.1 DOES THE STANDARD RM EVALUATION CAPTURE REWARD (IN-)CONSISTENCY?

  • Improvements in Cres and Cins from CONVEXDA + REWARDFUSION are not reflected in RMEVAL, including for contrast-finetuned RMs.The reported pattern suggests standard evaluation may miss changes in reward consistency.Source: Table 7 · p. 8, §7.1 · p. 8

7.2 A CLOSER LOOK TO WHY (OR WHERE) REWARD MODELS ARE INCONSISTENT

  • Reward scores show higher variance for response pairs closer in human score, making evaluation on easy good-versus-bad pairs liable to overestimate RM performance.The figure compares standard RMs with RMs enhanced by CONVEXDA and REWARDFUSION on TWITTER and REALSUMM.Source: Figure 3 · p. 8, §7.2 · p. 9

7.3 CONSISTENCY CHECK BEYOND CONTRAST INSTRUCTIONS

  • CONTRAST INSTRUCTIONS does not cover every RM consistency or robustness phenomenon; preliminary tests also find high attack success and vulnerability to adversarial and backdoor attacks.The authors suggest these findings may have implications for RM and subsequent RLHF safety.Source: §7.3 · p. 9

7.4 BROADER IMPACT AND LIMITATIONS

  • The authors identify limited understanding of what reward modeling best benefits RLHF and note that human-annotated extrinsic evaluations can be costly and unreliable.They present CONTRAST INSTRUCTIONS as a scalable intrinsic metric, while acknowledging it does not assess every RM capability.Source: §7.4 · p. 9

8 RELATED WORK

Related work covers consistency research in NLP and RLHF approaches that learn rewards from human annotations and optimize language models with reinforcement learning.

Consistency in NLP

  • Prior NLP work commonly defines consistency as invariant behavior under meaning-preserving alternations, while this paper studies consistency relative to human preference.The cited examples include Ribeiro et al. (2020), Elazar et al. (2021), Goel et al. (2021), and Wang et al. (2022b).Source: §8 · p. 9

Reinforcement Learning from Human Feedback

  • RLHF learns a reward function from human annotations and uses reinforcement learning methods such as PPO to optimize language models toward human preferences.The passage frames helpful, honest, and harmless alignment as a key implication of RLHF research.Source: §8 · p. 9

9 CONCLUSION

The paper studies reward inconsistency through Contrast Instructions and presents it as one direction for improving reward modeling for RLHF. Which type of reward modeling would most benefit RLHF remains an open question.

9 CONCLUSION

  • The paper uses CONTRAST INSTRUCTIONS to uncover and study reward inconsistency in reward modeling for RLHF.It presents this as one perspective and direction for improving reward models for RLHF.Source: §9 · p. 9
  • Which type of reward modeling would most benefit RLHF remains an open question.The authors hope the findings facilitate future research and evaluation on this problem.Source: §9 · p. 9
  • The authors position the findings as a resource for future research and evaluation on reward modeling for RLHF.Source: §9 · p. 9

SUPPLEMENTARY MATERIAL

The supplementary material provides Contrast Instructions details, full benchmark results, RL training settings, and background and evaluation material for adversarial consistency analyses.

Appendix Contents

A DETAILS OF CONTRAST INSTRUCTIONS

  • Contrast Instructions are built from test sets, retrieving instructions within a selected similarity range, such as [0.8, 0.9], and excluding the original instruction.The dataset is split into training, development, and testing sets for RM training and evaluation.Source: A DETAILS OF CONTRAST INSTRUCTIONS · p. 16

B FULL RESULTS ON CONTRAST INSTRUCTIONS

  • Standard-recipe RMs only slightly outperform random guessing on CONTRAST INSTRUCTIONS and fall significantly behind human performance.Table 8 covers instruction and response perturbations; it also identifies contrast-finetuned and multi-task settings.Source: Table 8 · p. 16

C CONFIGURATIONS OF RL TRAINING

  • The RL training configuration uses the same StackExchange-finetuned SFT model across the compared groups and trains the reward model with LoRA in INT8.The passage also specifies the learning rate, optimizer, KL penalty, scheduler, and pretraining gradient coefficient.Source: C CONFIGURATIONS OF RL TRAINING · p. 17

D.1 BACKGROUND

  • The adversarial setup perturbs a human-preferred response to induce an incorrect RM prediction while keeping its semantics closely aligned; the analysis uses character- and word-level attacks, including a simple augmentation-based Vanilla Attack.The cited attack methods include VIPER, DeepWordBug, PWWS, Genetic Attack, and TextFooler.Source: §D.1 · p. 17

D.2 EVALUATION

  • Adversarial accuracy measures RM accuracy on adversarial data as perturbations accumulate, with ϵ_i denoting the perturbation generated at iteration i.The passage says the metrics target consistency at both reward-score and performance tiers.Source: §D.2 · p. 17

D.3 RESULTS

The results examine reward-model vulnerabilities and defenses, including backdoor attacks and adversarial text perturbations. They also report analyses of ConvexDA, RewardFusion, and response examples from downstream RLHF models.

D.3 RESULTS

  • Performance degrades as text-attack perturbations accumulate across evaluated tasks; every tested model was vulnerable, while consistency ranked LLaMa-7B > GPT-J-6B > GPT2-XL-1.5B > GPT2-0.1B.Source: Figure 4 · p. 18, §D.3 · p. 18, Figure 5 · p. 19

E.1 BACKGROUND

  • Backdoor attacks poison RM training samples with trigger-embedded inputs and target labels, then test whether triggered inputs receive the target label at inference.Source: §E.1 · p. 20

E.2 EVALUATION

  • ASR measures accuracy on poisoned test samples, while CACC measures accuracy on clean test samples and whether benign-input behavior remains normal.Source: §E.2 · p. 21

E.3 RESULTS

  • Both word- and sentence-level backdoor attacks achieve exceedingly high ASR, but require RM training-data manipulation, making real-world feasibility questionable.Source: §E.3 · p. 21, Table 9 · p. 21

F.1 ANALYSES OF CONVEXDA

  • ConvexDA is compared with standard augmentation on WMT19 for Contrast Instructions performance, original-test performance, and efficiency, using N=3 or 5 added samples per sentence.Source: §F.1 · p. 21

F.2 ANALYSES OF REWARDFUSION

  • δ=0.95 is reported as RewardFusion’s best retrieval-similarity threshold, while ConvexDA shows a better performance-efficiency trade-off than vanilla DA.Source: §F.2 · p. 21, Figure 8 · p. 22

G CASE COMPARISONS BETWEEN RLHF MODELS GUIDED BY A CONSISTENT AND INCONSISTENT RM

Related papers

Loading 2309.16155v1…