PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward Difference Optimization For Sample Reweighting In Offline RLHF

Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan, Cam Tu Nguyen

arXiv:2408.09385v2cs.CLcs.AI

TL;DR

Offline RLHF captures which response is preferred but not how strongly it is preferred. Reward Difference Optimization reweights response pairs using coefficients predicted by a model that considers both responses, and 7B experiments on HH and TL;DR report effectiveness in automatic and human evaluations.

  • Problem

    Offline RLHF ranking methods capture which response is preferred but not the degree of preference, even though pairs can differ substantially in quality gap.

    Source: §1 · p. 2

  • Method

    Reward Difference Optimization reweights response pairs with reward-difference coefficients, predicted by a model that uses interactions between the responses.

    Source: §1 · p. 2

  • Results

    Experiments with Alpaca-7B on HH and TL;DR report effectiveness in reward-model- and GPT-4-based automatic metrics and human evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The conclusion reports that coefficients consistently help DPO and RRHF, while LLM-based and human evaluations validate the difference model’s advantages over the reward model.

    Source: §6 · p. 8

    Experiments are limited to 7B language models, so whether the conclusions hold for larger models is unknown.

    Source: §7 · p. 9

Abstract

from arXiv · show

With the rapid advances in Large Language Models (LLMs), aligning LLMs with human preferences become increasingly important. Although Reinforcement Learning with Human Feedback (RLHF) proves effective, it is complicated and highly resource-intensive. As such, offline RLHF has been introduced as an alternative solution, which directly optimizes LLMs with ranking losses on a fixed preference dataset. Current offline RLHF only captures the "ordinal relationship" between responses, overlooking the crucial aspect of how much one is preferred over the others. To address this issue, we propose a simple yet effective solution called Reward Difference Optimization, shorted as RDO. Specifically, we introduce reward difference coefficients to reweigh sample pairs in offline RLHF. We then develop a difference model which captures rich interactions between a pair of responses for predicting these difference coefficients. Experiments with 7B LLMs on the HH and TL;DR datasets substantiate the effectiveness of our method in both automatic metrics and human evaluation, thereby highlighting its potential for aligning LLMs with human intent and values

1 Introduction

RLHF aligns language models with human preferences but is complex and resource-intensive, motivating offline methods that train on fixed preference data. This paper addresses offline methods’ loss of preference-strength information with Reward Difference Optimization and a pairwise difference model.

1 Introduction

  • RLHF uses a reward model and PPO with four large models, while the policy and critic are updated online, making the approach resource-demanding and complicated.Source: §1 · p. 1
  • Offline RLHF uses ranking losses on fixed preference data, but captures which response is preferred without how much better it is.Treating pairs with different quality gaps equivalently may result in suboptimal offline RLHF performance.Source: §1 · pp. 1–2
  • Reward Difference Optimization (RDO) introduces coefficients that reweight response pairs as additional supervision in offline RLHF methods.The coefficients are proposed for methods including RRHF, DPO, and KTO.Source: §1 · p. 2
  • The difference model predicts preference differences directly by using attention-based interactions between response pairs, rather than assigning each response an independent reward score.The paper also describes a specifically designed training strategy for the model.Source: §1 · p. 2
  • Experiments with Alpaca-7B on HH and TL;DR report effectiveness in automatic metrics based on reward models and GPT-4, as well as human evaluation.The experiments use two datasets for alignment with human preferences.Source: §1 · p. 2

2 Reward Difference Coefficients

Reward difference coefficients weight offline preference-optimization losses according to the reward gap between preferred and dispreferred responses. The paper describes incorporating these coefficients into existing methods, including RRHF and DPO.

2 Reward Difference Coefficients

  • A reward model’s scores for the preferred and dispreferred responses define a reward difference coefficient that represents how much better the preferred response is.Source: §2 · p. 3
  • The coefficient scales each response pair’s alignment loss, and α controls its effect, with α = 0 disabling the coefficient.Source: §2 · p. 3
  • The coefficient is designed to steer more gradients toward pairs with larger differences, assuming the reward model can distinguish essential differences from minor mistakes.Source: §2 · p. 3
  • The coefficients are incorporated into RRHF and DPO losses; for DPO+rc, pair weighting uses both the independent reward model and DPO’s implicit reward.RRHF+rc also includes an SFT cross-entropy regularization term to keep the language model near its supervised-finetuning version.Source: §2 · p. 3

3 Reward Difference Prediction

The paper replaces independent response scoring with a model that predicts reward differences from a query and a pair of responses. Its training combines preference supervision with regularizers for identical and reversed response pairs.

3.1 Reward Difference Model

  • The difference model takes a query and two responses as input and predicts their reward difference; a negative score means the first response is worse, with magnitude indicating how much worse.Source: §3.1 · pp. 3–4
  • Unlike a pointwise reward model, the difference model’s last-token embedding can attend to both responses, providing a more informative representation for predicting their difference.Figure 2 contrasts the traditional and RDO pipelines: the latter uses predicted difference scores to supervise alignment.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Difference Model Training

  • Difference-model training uses cross-entropy supervision based on whether the dataset labels the first response better than the second.Source: §3.2 · p. 4
  • Two regularizers target scores of zero for duplicate responses and opposite scores when the order of a response pair is reversed.The paper motivates these terms as addressing same-response scoring issues and supporting more efficient training.Source: §3.2 · p. 4

4 Experiments

The experiments evaluate reward difference coefficients and a difference model across offline RLHF methods, datasets, and evaluation approaches. Results report improvements from adding reward coefficients and advantages for the difference model over pointwise reward models.

4.1 Common Settings

  • The experiments use Alpaca-7B with RRHF, DPO, and KTO on HH and TL;DR preference datasets.The HH and TL;DR datasets have different response-generation and training setups for RRHF and DPO.Source: §4.1 · p. 5

4.2.1 Experimental Design

  • HH experiments train RRHF for three epochs and DPO for one, while TL;DR experiments train both for one epoch with α set to 0.5.The HH runs test α values of 0.5 and 1.0; the TL;DR RRHF starting model is Llama2-7B.Source: §4.2.1 · p. 6

4.2.2 Experimental Results

  • Reward difference coefficients consistently enhance offline RLHF performance across the reported experimental results.The experiments report reward-model evaluations on HH and TL;DR and LLM-as-judge evaluations; the corresponding tables report those results.Source: §4.2.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • RRHF+rc and DPO+rc outperform their vanilla counterparts across α values, sampling strategies, and reward models on HH.The comparisons include greedy and beam search, with gains reported according to different reward models.Source: §4.2.2 · p. 6
  • On TL;DR, reward difference coefficients improve performance across sampling strategies and base offline RLHF methods.Source: §4.2.2 · p. 6
  • Majority voting by GPT-4, GPT-3.5, and moonshot-v1 favors reward coefficients across offline RLHF methods and datasets.Table 3 reports the LLM-as-judge evaluation.Source: §4.2.2 · p. 6, Table 3 · p. 6

4.3.2 Experimental Results

  • On the HH test set, the difference model exceeds the two reward-model baselines, including a three-epoch Alpaca-7B reward model, though that comparison uses unequal training epochs.Regularization raises accuracy from 0.708 to 0.715, with further improvement possible through β0 and β1 tuning.Source: §4.3.2 · p. 7, Table 4 · p. 7

4.4.2 Experimental Results

  • For alignment comparisons, the difference model predicts coefficients for RRHF+rc and DPO+rc, and is favored over the reward model by LLM judges and human evaluators.The human evaluation compares RRHF+rc variants on 300 HH test samples.Source: §4.4.2 · p. 7, Table 5 · p. 7, Table 6 · p. 7

5 Related Work

Offline RLHF methods and reward models address alignment from different angles, but the methods discussed still miss the degree of preference difference. The paper positions reward-difference modeling as a way to enhance offline RLHF.

5.1 LLMs Alignment with Offline RLHF

  • Offline RLHF methods focus on ordinal relationships between responses, neglecting how much one response is preferred over another.The section surveys methods including SLiC-HF, DPO, RSO, RRHF, PRO, ReST, KTO, SimPO, and IPO.Source: §5.1 · pp. 7–8

5.2 Reward Modeling for Alignment

  • Reward models can guide alignment and preference collection, but conventional Bradley-Terry models often fall short of capturing true human preferences.Fine-grained and uncertainty-aware approaches retain pointwise scoring, while SLiC-HF predicts a categorical choice rather than a quantitative preference difference.Source: §5.2 · p. 8
  • Offline RLHF usually omits reward modeling, while recent studies report its importance for LLM alignment.The paper presents its approach as a simple way to enhance multiple offline RLHF methods.Source: §5.2 · p. 8

6 Conclusion

The paper weights offline RLHF samples by reward difference and introduces a difference model to predict those coefficients. Experiments report benefits across methods and evaluations, while larger-model scaling and alignment generalization remain future directions.

6 Conclusion

  • Reward-difference coefficients weight comparisons by how much one response is preferred, and the difference model predicts these coefficients from response-pair interactions.The coefficients are used during training to focus on more certain comparisons.Source: §6 · p. 8
  • The coefficients are effective with DPO and RRHF, and LLM-based and human evaluations favor the difference model over the reward model.Source: §6 · p. 8
  • The authors identify larger-model evaluation and preserving generalization after alignment as future research directions.The reported effectiveness is with 7B models.Source: §6 · p. 8

7 Limitation

The reported limitations concern model quality, training cost under particular sequence-length conditions, the difference model's suitability for direct reward scoring, and the experimental scale.

7 Limitation

  • The effectiveness of the methods can depend on the quality of the reward model or difference model.Source: §7 · p. 9
  • Difference-model training uses half as many queries but longer sequences, and may require more time and resources when queries are short relative to responses.Source: §7 · p. 9
  • The difference model may not directly provide the response reward score required by PPO, though a baseline response for each query can support a single-response score.Source: §7 · p. 9
  • Experiments and analyses are limited to 7B models, leaving it unknown whether the conclusions hold at larger scales.Source: §7 · p. 9

8 Ethical Concerns

The paper flags sensitive and offensive material in HH and HH+ as research-only data. Its appendix also describes reward-coefficient use with KTO and the associated experimental settings.

8 Ethical Concerns

  • HH and HH+ contain sensitive and offensive content and are intended solely for research purposes.The authors state that views expressed in the data do not reflect their beliefs.Source: §8 · p. 9

A.1 Reward difference coefficient on KTO

  • For KTO, the reward-difference score serves as the coefficient for paired desirable and undesirable responses, despite KTO's pointwise samples.Source: §A.1 · p. 11

A.1.1 Experiment settings

  • KTO experiments compare training with and without reward coefficients on Alpaca-7B using HH and TL;DR, with one epoch and a 5e-7 learning rate.The settings use LoRA rank 8; reward coefficients come from an Alpaca-7B HH reward model and an open-source DeBERTa model for TL;DR.Source: §A.1.1 · p. 11

A.1.2 Experimental results

The experiments report that reward-difference coefficients help KTO, and that the difference model compares favorably with reward-model baselines in additional settings. The accompanying analysis argues that coefficients change sample optimization speed, not DPO’s analytical optimum.

A.1.2 Experimental results

A.2 Additional experiments for difference model

  • The proposed additional experiments are intended to test the robustness of the difference model’s superiority, including RRHF comparisons against another reward model and DPO at β = 0.5.The authors describe these as further evidence for the robustness of their conclusion.Source: §A.2 · p. 11, §A.2.1 · p. 11, §A.2.2 · pp. 11–12

A.2.1 RRHF with Difference model V.S. Alpaca reward model

  • The difference model outperforms the Alpaca-7B reward model for RRHF under the reported experimental and evaluation settings.The comparison is reported in Table 9.Source: §A.2.1 · p. 11, Table 9 · p. 11

A.2.2 DPO with bigger β β

  • With DPO β = 0.5, the difference model still outperforms the baseline reward model; ties increase as both aligned models remain closer to the initial SFT model.The comparison is reported in Table 10.Source: §A.2.2 · pp. 11–12, Table 10 · p. 12

B Why Reward Coefficients?

  • The analysis assumes hard-to-tell samples are noisier and explains that one-epoch training may underfit them, yielding smaller coefficients than for easier samples.The authors use this as a rationale for the reward coefficients’ behavior on noisy versus clean comparisons.Source: B Why Reward Coefficients? · p. 12
  • For DPO, the reward coefficient leaves the analytical optimum unchanged but alters each sample’s optimization speed, relatively reducing noisy-sample gradients and the likelihood of overfitting them at the same training pace and epochs.The analysis presents this as a possible explanation for the benefit of reward coefficients, using DPO as an example.Source: B Why Reward Coefficients? · p. 12

C Training and Inference Consumption

The paper reports that the added model stage is a small part of DPO training, while inference cost depends on the number of responses per query. It also provides a qualitative response example and finds that higher difference-model confidence tends to accompany higher prediction accuracy.

C.1 Analyses of the additional cost of introducing reward/difference model

  • Training and inference for the difference model take a relatively small part of complete DPO training, and their one-time cost is compared with potentially repeated DPO epochs.The method first trains and runs inference with a difference or reward model to obtain coefficients, then performs offline RLHF.Source: §C.1 · pp. 12–13, Table 11 · p. 13

C.2 Comparison between reward model and difference model

  • With six responses per query, difference-model inference is longer than reward-model inference, but RRHF training remains the dominant time; with two responses, difference-model inference can be faster.The two-response comparison is described for HH, where queries average about 164 tokens and responses about 145 tokens; the difference model uses twice the reward model’s maximum sequence length.Source: §C.2 · p. 13

D Case Study

  • In the granola example, RRHF+rc with the difference model gives clear, organized instructions covering the steps from combining ingredients through baking.Vanilla RRHF’s response is unfocused, while RRHF+rc with the reward model remains vague and lacks concrete instructions.Source: D Case Study · pp. 13–14

E.1 Correlation between difference model accuracy and confidence

  • A larger absolute reward-difference score tends to coincide with higher difference-model prediction accuracy, supporting its use to distinguish hard-to-tell from easy-to-tell samples.The correlation is reported in Table 15.Source: §E.1 · p. 14

E.2 LLM-as-judge evaluation details

The evaluation details describe LLM-as-judge scoring and human comparison procedures. The paper also identifies tables of per-model LLM judgments and example responses from RRHF variants.

E.2 LLM-as-judge evaluation details

  • Response quality was evaluated by gpt-3.5-turbo-0105, gpt-4-0125preview, and moonshot-v1, which scored paired responses and supplied explanations.Figure 3 presents the evaluation prompt.Source: §E.2 · p. 14, Figure 3 · p. 14
  • The paper reports majority-voting results from three LLMs in the main text and also presents detailed results given by each powerful LLM in Tables 12 and 13.Source: §E.2 · p. 14

E.3 Human evaluation details

  • Three graduate-student volunteers compared responses for helpfulness and general quality after order shuffling and source blinding, with majority decisions used as final results.Table 14 presents responses from different RRHF variants.Source: §E.3 · p. 14, Table 14 · p. 15

F Future Works

The authors identify limits in model scale, evaluation domain, and hyperparameter tuning. They report an observed cross-task drop after HH alignment and leave its mitigation as future work.

F Future Works

  • All experiments use 7B language models, leaving the scaling behavior of reward coefficients and the comparison model unstudied.The authors identify performance at larger model sizes as an open question.Source: F Future Works · p. 15
  • Evaluation is in-domain, and the authors observe that a model aligned on HH may lose summarization ability on TL;DR.They identify generalization to out-of-domain tasks and datasets as a future direction.Source: F Future Works · p. 15
  • The authors say resource limits may have left tuning of α and β0, β1 insufficient and propose further experiments.Source: F Future Works · p. 15
Loading 2408.09385v2…