PaperlayerView exact revision on arXiv

Source-linked AI summary

SLiC-HF: Sequence Likelihood Calibration with Human Feedback

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J. Liu

arXiv:2305.10425v1cs.CLcs.AI

TL;DR

The paper addresses the complexity of using RLHF-PPO to learn from human preferences and whether feedback collected for another model can be reused. It applies SLiC-HF to calibrate sequence likelihood with preference data, and finds improved TL;DR summarization over SFT baselines while competitively matching prior RLHF-PPO with simpler, more efficient training.

  • Problem

    Reference-based supervised learning and metrics are limited for improving beyond reference summaries, while RLHF-PPO is complex and costly to train.

  • Method

    SLiC-HF calibrates an SFT model’s sequence likelihood using human-preference pairs, obtained directly from feedback data or through sampled candidates and ranking models.

  • Results

    SLiC-HF significantly improves SFT baselines on Reddit TL;DR summarization and performs competitively with prior RLHF-PPO, including when using feedback collected for different models.

  • Takeaways & Limitations

    SLiC-HF is a simpler, easier-to-tune, and computationally efficient alternative to PPO-based RLHF for the evaluated summarization setting.

  • Takeaways & Limitations

    Direct calibration can produce increasing, unstable sequence lengths, likely because off-policy feedback decodes are out of distribution.

Abstract

from arXiv · show

Learning from human feedback has been shown to be effective at aligning language models with human preferences. Past work has often relied on Reinforcement Learning from Human Feedback (RLHF), which optimizes the language model using reward scores assigned from a reward model trained on human preference data. In this work we show how the recently introduced Sequence Likelihood Calibration (SLiC), can also be used to effectively learn from human preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human feedback data collected for a different model, similar to off-policy, offline RL data. Automatic and human evaluation experiments on the TL;DR summarization task show that SLiC-HF significantly improves supervised fine-tuning baselines. Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF implementation used in past work while being much simpler to implement, easier to tune and more computationally efficient in practice.

1 Introduction

SLiC-HF applies sequence likelihood calibration to human preference data as a simpler alternative to RLHF. On Reddit TL;DR summarization, it improves supervised fine-tuning and competes with prior PPO-based RLHF, including with off-policy feedback data.

  • Motivation: Supervised learning is limited by reference-summary quality, while ROUGE cannot measure improvements beyond those references.Human preferences can distinguish higher-quality summaries even when they differ from web-mined references.
  • Prior approach: RLHF uses a reward model trained on pairwise human preferences, then applies PPO to refine an SFT model toward higher predicted reward.A KL penalty typically keeps the RLHF policy near the original supervised policy.
  • Prior approach: RLHF-PPO adds value and reward networks, roll-outs, slower optimization, and more complex hyper-parameter tuning.These requirements increase memory use and reduce the maximum trainable model size under a fixed memory budget.
  • Contribution: SLiC-HF replaces reference-based ranking with human preference ranking, using either off-policy preference data or a learned ranking model.This extends SLiC from reference similarity criteria to arbitrary, potentially reference-less ranking functions.
  • Results: SLiC-HF improves TL;DR summarization, matches or exceeds the prior 6B RLHF-PPO model with a 770M T5-Large model, and further improves T5-XXL.The feedback data was collected for different models, making the setting similar to off-policy offline RL.

2 Method

SLiC-HF fine-tunes an SFT model by calibrating sequence likelihood on human-preference pairs while regularizing toward target sequences. It can obtain those pairs from sampled SFT candidates ranked by learned models or directly from off-policy feedback data.

  • 2.1 Sequence Likelihood Calibration: SLiC-HF first fine-tunes Pθft on supervised reference data, then aligns the SFT model’s sequence likelihood using SLiC.Human-preference data is added to the standard supervised fine-tuning data.
  • 2.1 Sequence Likelihood Calibration: The loss contrasts positive and negative sequence likelihoods with margin δ, while cross-entropy regularization toward yref keeps the model near the SFT model.The regularizer serves a role similar to KL regularization without requiring an extra copy of SFT weights.
  • 2.2 SLiC-HF with Sample and Rank: SLiC-HF-sample-rank samples m candidates from the SFT model and determines positive-negative pairs using trained reward or pairwise ranking models.The pointwise reward model scores candidates using the probability of “Good,” while the ranking model compares candidate pairs.
  • 2.3 SLiC-HF Directly On Human Feedback: SLiC-HF-direct calibrates directly on positive and negative sequences from human feedback without training or using a ranking or reward model.Its advantage is greater simplicity and efficiency, but its off-policy data distribution may differ from the SFT model’s decode distribution.
  • 2.4 Regularization Target: Regularization targets can be either supervised references or the best-ranked sampled candidate selected by a ranking or reward model.The target choice controls which sequence receives the cross-entropy regularization.

3 Experimental Results

Experiments evaluate SLiC-HF on Reddit TL;DR using T5 generation models, human-feedback data, and ranking-based automatic evaluation.

  • Experiments use Reddit TL;DR data with 117k/6k/6k train, validation, and test examples, plus 64k human preferences.
  • The ablation uses T5-large (770M) for generation and T5-XXL (11B) for ranking and reward modeling.
  • Generation models are fine-tuned on the supervised training split, while ranking and reward models use human-feedback training data and validation-selected checkpoints.
  • SLiC-HF-sample-rank calibrates on 8 sampled decodes using temperature 0.7 and top-k 40.
  • Automatic evaluation uses beam search with beam size 4 and measures model-summary win rate against human references using the T5-XXL ranking model.

3.3 Reward Model and Ranking Model Accuracy

The pairwise ranking model is more accurate than the reward model on human-feedback validation data.

  • 73.23% ranking-model accuracy exceeds the reward model’s 71.34% accuracy on DHF validation.The ranking model is about 2% more accurate.

3.4 SLiC Ablation

The ablation compares continued fine-tuning and SLiC-HF variants for leveraging human feedback, finding gains from ranked decodes and practical differences between direct and sampled calibration.

  • SLiC-HF vs Continue Fine-tuning on Filtered Data: 51.65% win rate against references improves over the SFT baseline’s 44.96% after continued fine-tuning on positive human-feedback sequences.
  • SLiC-HF vs Continue Fine-tuning on Filtered Data: Selecting the best 1 of 8 decoded summaries raises win rate against references to above 60%.
  • SLiC-HF vs Continue Fine-tuning on Filtered Data: Pairwise ranking-model filtering performs slightly better than pointwise reward-model filtering.
  • Apply SLiC-HF Directly On Human Feedback Data: SLiC-HF-direct’s calibration loss decreases while sequence length keeps increasing instead of converging, whereas SLiC-HF-sample-rank converges robustly.
  • Apply SLiC-HF Directly On Human Feedback Data: 82.92% win rate against references is achieved by SLiC-HF-direct when the ranking model selects the best checkpoint.
  • SLiC-HF vs Continue Fine-tuning on Filtered Data: SLiC-HF-sample-rank with a ranking model gains about 3% win rate over its reward-model version, consistent with higher ranking-model agreement with human preferences.
  • SLiC-HF vs Continue Fine-tuning on Filtered Data: Using SFT targets or best-ranked decodes as regularization shows little difference, and sample-rank remains applicable without ground-truth references.

3.5 Human Evaluation

Four-way human evaluation compares reference summaries, SFT variants, and SLiC-HF; SLiC-HF is preferred and achieves higher quality and factuality. Length-controlled analysis also favors SLiC-HF, while comparisons with RLHF-PPO show competitive performance.

  • SLiC-HF Ablation Study: 73% of the time, raters chose SLiC-HF as the best model in the four-way human evaluation.The evaluation used 100 validation examples and compared reference, SFT, continued fine-tuning, and SLiC-HF summaries.
  • SLiC-HF Ablation Study: SLiC-HF had significantly higher average quality and was the most factual model in the four-way evaluation.Average quality aligned well with the ranker win-rate reported in Table 1.
  • SLiC-HF Ablation Study: Length-controlled quality analysis clearly preferred SLiC-HF over SFT and continued fine-tuning across relative-length buckets.Mean scores were calculated among examples bucketed by their relative length to the reference.
  • SLiC-HF vs RLHF-PPO: The T5-Large SFT baseline had slightly higher quality and win rate than the 6B decoder-only SFT baseline, but the difference was not statistically significant.The comparison was conducted through a two-way side-by-side human evaluation.
  • SLiC-HF vs RLHF-PPO: SLiC-HF-sample-rank with a ranking model performed better than RLHF-PPO, while its reward-model variant had similar performance.SLiC-HF summaries were slightly longer, but their length-controlled win rate was similar to RLHF-PPO.

3.6 Scaling Up SLiC

Scaling SLiC-HF-sample-rank was evaluated by increasing generation-model parameters and the number of decoded candidates. Larger models improved both SFT and SLiC-HF, whereas more candidates provided little benefit.

  • Scaling model parameters: Increasing the generation model from 770M to 11B significantly improved both the SFT and SLiC-HF models.This comparison is reported in Table 4 for SLiC-HF-sample-rank.
  • Scaling decoded candidates: Increasing the number of decoded candidates m from 8 to 64 did not help much.The study varied candidate count separately from generation-model size.

4 Further discussion on SLiC-HF vs. RLHF-PPO

SLiC-HF reduces the memory, decoding, and optimization burdens associated with RLHF-PPO while using pairwise preference information directly. This design also avoids value-function estimation and may support more stable optimization.

  • Optimization design: RLHF-PPO requires separate value and reward networks, while SLiC-HF uses the relative rank of two summaries without relying on a value-function sub-model.The paper conjectures that using the cleaner preference signal may produce more stable optimization.
  • Compute and memory efficiency: SLiC-HF uses 1/4 the memory for model weights during training because rewards are computed offline and in parallel.The saved memory could instead support training larger models.
  • Compute and memory efficiency: SLiC-HF decoding can be significantly faster because all decoded samples use the same policy, enabling completely parallel decoding.PPO decoding is blocked by policy updates and occurs inside the training loop, whereas SLiC-HF step times are similar to fine-tuning.
  • Compute and memory efficiency: SLiC-HF can cache input sequence encoded states across the m decodes because they are sampled from the same SFT policy.This may matter for summarization and other long-context tasks, where inputs tend to be longer than outputs.
  • Compute and memory efficiency: SLiC-HF computes rewards outside the training loop, providing parallelism advantages over RLHF for reward computation per episode.This complements its offline treatment of decoded samples.

5 Related work

Related work has aligned language-model likelihoods or policies with reference-based rewards, human preferences, and AI feedback. SLiC-HF instead applies sequence-likelihood calibration to preference rankings predicted from paired summaries.

  • Reward alignment: RL optimization of BLEU or ROUGE can improve those metrics while human quality judgments suffer from metric misalignment.
  • Reward alignment: RLHF trains reward models to predict carefully collected human judgments and commonly uses KL regularization to limit departure from the supervised model.
  • Sequence-level ranking: BRIO rank-orders model-generated decodes using reference similarity and a list-wise loss, whereas SLiC-HF aligns likelihood with predicted human preference between summaries.
  • Feedback sources: SLiC-HF can use AI feedback exactly as it uses human feedback and is indifferent to whether the feedback originates from AI or humans.

6 Conclusion

The paper proposes SLiC-HF for calibrating sequence likelihood on human feedback data. On Reddit TL;DR summarization, it improves SFT baselines and offers a simpler, easier-to-tune, computationally efficient alternative to RLHF-PPO.

  • Conclusion: SLiC-HF calibrates sequence likelihood using human feedback data.
  • Conclusion: On Reddit TL;DR summarization, SLiC-HF significantly improves supervised fine-tuning baselines.
  • Conclusion: SLiC-HF presents a competitive alternative to past RLHF-PPO implementations while being simpler to implement, easier to tune, and computationally efficient.The stated evidence concerns the Reddit TL;DR summarization experiments.

A Human Evaluation

The human evaluation task presents four anonymized summaries in a random order for each example. This setup prevents fixed position or model identity from determining the presentation.

  • Human evaluation: Four summaries are randomly shuffled for each human-evaluation example, and the models are anonymized.
  • Human evaluation: Random shuffling changes summary positions across examples during the human evaluation task.
  • Human evaluation: Model anonymization removes model identities from the displayed summaries during evaluation.
Loading 2305.10425v1…