PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Effects of RLHF on LLM Generalisation and Diversity

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, Roberta Raileanu

arXiv:2310.06452v3cs.LGcs.AIcs.CL

TL;DR

The paper asks how RLHF pipeline stages affect OOD generalisation and output diversity, properties important for deploying LLMs across varied settings and applications. It evaluates SFT, reward modelling, RLHF, and BoN across summarisation and instruction following, using distribution-shifted tests and multiple diversity metrics. RLHF generally improves performance relative to SFT but lowers output diversity, creating a tradeoff whose scope and origins remain open for further study.

  • Problem

    The paper addresses limited understanding of how RLHF pipeline stages affect OOD generalisation and output diversity, both important for varied real-world use cases.

  • Method

    The study evaluates SFT, reward modelling, RLHF, and BoN across summarisation and instruction following using realistic OOD shifts and syntactic, semantic, and logical diversity metrics.

  • Results

    RLHF produces more performant models than SFT in-distribution and out-of-distribution, but with lower output diversity both per-input and across-input.

  • Takeaways & Limitations

    RLHF suits settings with large distribution shifts, whereas SFT may be preferable when applications require a wide variety of outputs.

  • Takeaways & Limitations

    The paper provides empirical findings without a theoretical explanation and evaluates only selected combinations of base models, tasks, and methods.

Abstract

from arXiv · show

Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date, such as OpenAI's ChatGPT or Anthropic's Claude. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF is still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: out-of-distribution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current LLM use cases. We find that RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which fine-tuning method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.

1 INTRODUCTION

The paper examines how stages of LLM fine-tuning affect out-of-distribution generalisation and output diversity. Across its evaluation, RLHF improves generalisation relative to SFT but reduces diversity, revealing a tradeoff that motivates further methods research.

  • 1 INTRODUCTION: OOD generalisation matters because LLMs must remain performant and reliable beyond their training-data distribution.The paper notes that the pipeline stage responsible for observed generality is not established.
  • 1 INTRODUCTION: Output diversity is crucial for creative, open-ended, and red-teaming applications requiring varied high-quality outputs.Prior work reported diversity decreases from RLHF, but the paper identifies a lack of rigorous cross-task analysis.
  • 1 INTRODUCTION: The study evaluates SFT, reward modelling, RLHF, and BoN for in-distribution performance, OOD performance, and output diversity.It uses realistic distribution shifts and syntactic, semantic, and logical diversity metrics across summarisation and instruction following.
  • 1 INTRODUCTION: RLHF improves in-distribution and OOD performance compared with SFT.This is reported as a principal finding of the study.
  • 1 INTRODUCTION: RLHF substantially decreases diversity for outputs sampled for a given input and produces less diverse text on some across-input metrics.The latter pattern implies greater similarity in generated text regardless of the input.
  • 1 INTRODUCTION: The findings expose a tension between generalisation and diversity and motivate methods that improve both without sacrificing either.The paper also asks whether this tension is fundamental to fine-tuning or reflects current-method deficiencies.

2 BACKGROUND AND RELATED WORK

The background frames RLHF as an alternative to demonstration-based SFT when human evaluation or ranking is easier than collecting full demonstrations. Prior work leaves OOD generalisation and broad diversity effects insufficiently characterised, which this paper addresses.

  • 2 BACKGROUND AND RELATED WORK: SFT trains pretrained language models on demonstrations, whereas RLHF is useful when ranking outputs is easier than gathering accurate demonstrations.This motivates preference-based fine-tuning for complex downstream tasks.
  • 2 BACKGROUND AND RELATED WORK: The study focuses on the popular RLHF pipeline because understanding it is especially relevant to widely deployed large-scale systems.Other recent fine-tuning approaches are discussed but not used in the main analysis.
  • 2 BACKGROUND AND RELATED WORK: Most prior RLHF studies evaluate on the same input distribution used for fine-tuning, leaving generalisation properties poorly understood.A notable exception tested Reddit summarisation models on news-article summarisation, but the broader question remained limited.
  • 2 BACKGROUND AND RELATED WORK: Existing studies found lower RLHF output diversity in specific settings, primarily using self-BLEU.This paper extends that work with externally validated metrics covering multiple forms of diversity.

3 MODEL TRAINING

The paper trains and compares SFT, reward-model-based, RLHF, and BoN systems using a LLaMA 7B base model, with OPT experiments providing trends across model scales. RLHF uses PPO initialized from the corresponding SFT model and constrained by a KL penalty.

  • 3 MODEL TRAINING: The main experiments use the LLaMA pretrained 7 billion parameter decoder-only causal language model.Appendix experiments use OPT models at five sizes to examine trends across model scales.
  • 3 MODEL TRAINING: Reward models are initialized from the base model, augmented with a scalar head, and trained to predict preferences between paired outputs.Each pair contains one preferred output for a given input.
  • 3 MODEL TRAINING: SFT models learn from reference input-output pairs using cross-entropy loss on outputs conditioned on inputs.
  • 3 MODEL TRAINING: RLHF models use PPO, initialize from SFT, and optimize a policy using the reward model and a learned value function.The policy and value function share the model backbone.
  • 3 MODEL TRAINING: The RLHF reward combines reward-model output with a KL penalty relative to the SFT policy, using β_KL = 0.05.The coefficient controls the penalty weight and was selected to balance performance and overoptimisation.
  • 3 MODEL TRAINING: BoN samples N = 16 summaries from SFT and uses the reward model to select the best one.Sampling uses temperature 0.7 to balance improved performance and computational cost.

4 DATASETS AND TASKS

The experiments cover Reddit-post summarisation and instruction following, using task-specific datasets and models. The instruction-following models come from AlpacaFarm, while summarisation reward training uses human preference comparisons.

  • 4 DATASETS AND TASKS: The study investigates summarisation and instruction following as two fine-tuning settings.
  • 4 DATASETS AND TASKS: Summarisation models produce summaries of Reddit posts using approximately 120,000 posts from a filtered TL;DR dataset.
  • 4 DATASETS AND TASKS: Reward-model training for summarisation uses approximately 64,000 human-annotated summary comparisons.Each comparison pairs summaries for a Reddit post and labels one as preferred according to quality criteria.
  • 4 DATASETS AND TASKS: Instruction following is included because it is a major use case for the investigated LLM fine-tuning techniques.
  • 4 DATASETS AND TASKS: Instruction-following experiments use LLaMA 7B SFT, RLHF, and reward models released by AlpacaFarm.These models take text instructions as input and generate preferred answers.

5 MODEL EVALUATION

The paper evaluates fine-tuning techniques using GPT-4 preference judgments, distribution-shifted test sets, and multiple measures of output diversity. It distinguishes generalisation across inputs from diversity within and across inputs.

  • Performance evaluation: GPT-4 compares model outputs with human-annotated references and can also perform head-to-head policy comparisons.The reference comparison yields preference vs reference (PvR), a percentage win rate against the reference output.
  • Generalisation evaluation: OOD evaluation compares performance on in-distribution and differently distributed test sets, with lower generalisation gaps indicating better generalisation.The test sets include corresponding reference outputs for evaluation.
  • Generalisation evaluation: Summarisation evaluation uses TL;DR as the in-distribution test set and CNN/DailyMail as an out-of-distribution test set from a different news domain.This tests whether the model learned a more general summarisation skill.
  • Generalisation evaluation: Instruction-following evaluation uses regenerated AlpacaFarm-style inputs in distribution, plus AlpacaEval and Sequential Instructions out of distribution.Sequential Instructions contain multiple steps that may build on one another and require completing all steps satisfactorily.
  • Diversity evaluation: Output diversity is measured with EAD, Sentence-BERT similarity, and NLI diversity, covering syntactic, semantic, and logical variation.The measures are reported as supported by prior work and aligned with human diversity evaluations.
  • Diversity evaluation: For each policy, the study samples K = 16 outputs for each of N = 500 inputs and evaluates diversity both per input and across inputs.Per-input diversity concerns π(y|x), whereas across-input diversity concerns π(y).

6 EXPERIMENTAL RESULTS

Across summarisation and instruction following, RLHF generally improves absolute ID and OOD performance over SFT, especially under larger distribution shifts, but substantially reduces output diversity. BoN can outperform RLHF on summarisation, while RLHF outperforms BoN on instruction following.

  • Generalisation: RLHF performs better than SFT in absolute terms on both in-distribution and out-of-distribution evaluations.This pattern is reported across the paper’s generalisation experiments and is also supported by results on OPT models across model scales.
  • Generalisation: Bo16 outperforms RLHF, which outperforms SFT, on both ID and OOD summarisation performance.The summarisation comparison uses TL;DR as ID and CNN/DailyMail as OOD.
  • Generalisation: RLHF outperforms Bo16 and SFT across ID and OOD instruction-following evaluations, including AlpacaEval and Sequential Instructions.On the harder Sequential Instructions shift, RLHF generalises much better; on AlpacaEval, the models generalise similarly under GPT-4 PvR.
  • Generalisation: RLHF and Bo16 improve their head-to-head winrates against SFT by approximately 3.5% from ID to AlpacaEval OOD.This metric reveals a generalisation difference that GPT-4 PvR does not show on the easier AlpacaEval task.
  • Diversity: RLHF substantially lowers per-input output diversity compared with SFT, while its across-input diversity is also lower but by a much smaller amount.The per-input scores measure diversity among samples for one input; across-input scores use single outputs from a range of inputs.
  • Diversity: Across-input diversity results provide evidence of RLHF-induced mode collapse, although NLI finds no meaningful diversity difference between models.The mode-collapse interpretation concerns RLHF producing similar styles across different inputs; NLI indicates comparable logical diversity.

7 DISCUSSION AND CONCLUSION

The paper finds a tradeoff: RLHF improves absolute in-distribution and out-of-distribution performance over SFT, but reduces output diversity. Which method is preferable depends on distribution shift, diversity needs, and reward-model generalisation.

  • RLHF produces more performant models than SFT both in-distribution and out-of-distribution, but with lower output diversity per input and across inputs.
  • Generalisation metrics controlling for in-distribution performance give mixed results: RLHF is better under the hardest instruction-following shift but similar or slightly worse under easier shifts.
  • Implications for Practical Applications: RLHF is useful when large distribution shifts are likely, whereas SFT may be preferable when applications require varied outputs.
  • Future Research Directions: Future work should explain RLHF’s diversity reduction and investigate methods that recover diversity without sacrificing performance.

A LIMITATIONS AND FUTURE WORK

The authors identify limits in theoretical explanation, experimental breadth, evaluated methods, and human assessment. They propose testing broader settings and methods, explaining the observed effects, and evaluating human judgments.

  • The study empirically measures RLHF’s effects but does not provide a theoretical explanation for its generalisation and diversity results.
  • The experiments cover multiple base models and tasks, but additional model–task combinations and methods remain untested.
  • Future research should test whether the effects generalise, explain why they arise, and examine their impact on human judgments of outputs.
  • Only SFT, RLHF, and BoN are evaluated, leaving other human-feedback fine-tuning methods for future comparison.
  • The evaluation uses automatic metrics without human evaluation, although the metrics were validated against human preferences.

D.1 VALIDATING GPT-4 EVALUATION

The authors validate GPT-4 as a summarisation evaluator by comparing it with human preferences on validation datasets and model outputs. They report good aggregate and per-example agreement.

  • Summarisation: 71.7% accuracy on TL;DR and 65.5% on CNN/DailyMail align with reported inter-annotator agreement for those datasets.
  • Summarisation: GPT-4 and human labellers are compared on 100 model-output datapoints, producing 200 human annotations across datasets and model types.
  • Table 1 reports GPT-4 agreement with human raters at both aggregate and individual levels across summarisation datasets and SFT and RLHF models.
  • Instruction Following: For instruction following, the study uses an evaluator previously shown to agree well with human labellers and does not independently validate it.

E.2 POLICY TRAINING

The policy-training setup treats language generation as reinforcement learning, contrasting SFT, PPO-based RLHF, and BoN sampling. Training uses shared or frozen components and validation-based hyperparameter selection.

  • The language model is treated as an RL policy whose actions are tokens, states are current inputs, and episodes end at a token limit or end-of-sequence token.
  • SFT is behavioural cloning trained with cross-entropy on output tokens, while RLHF uses PPO initialized from the corresponding SFT model.
  • RLHF’s reward combines reward-model output with a KL penalty relative to the SFT policy, weighted by βKL = 0.05.
  • BoN samples N outputs and uses the reward model to select the best one, avoiding RL policy training but increasing inference cost.
  • Models are selected using in-distribution validation data and method-appropriate metrics for SFT, reward modelling, and RL.
  • The implementation freezes the first 80% of layers plus embedding and unembedding layers to improve computational efficiency, with a non-catastrophic performance drop.

G BEST OF N TEMPERATURE EXPERIMENT

The appendix evaluates Best-of-N sampling temperature and documents construction of a sequential instructions dataset designed for diverse, executable tasks.

  • Temperature experiment: Temperature 0.7 performs better than temperature 1 for Best-of-N sampling, motivating its use as the main-paper hyperparameter.The comparison includes Best-of-N, RLHF, and SFT.
  • Sequential instructions dataset: The sequential instructions dataset is built by adapting the AlpacaFarm Self-Instruct protocol to gather more sequential instructions.The seed instructions and prompt were adjusted for this purpose.
  • Sequential instructions dataset: The generation prompt requests 20 diverse task instructions that a GPT model can complete.The requirements target diversity in verbs, language, and task types.
  • Sequential instructions dataset: The prompt requires varied verbs, language forms, and task types, including generation, classification, and editing.It also excludes instructions that a GPT language model cannot complete.
  • Sequential instructions dataset: The dataset examples are reported in Table 5, and the dataset is made publicly accessible.The passage provides the dataset location.

I SUMMARISATION KL SWEEP RESULTS

The KL-penalty sweep tests whether this RLHF hyperparameter can balance diversity and generalisation in summarisation, but higher penalties worsen both outcomes.

  • Experimental setup: The sweep varies the KL penalty coefficient because it determines the weight of the KL penalty in the RLHF reward.The experiment directly tests whether this weight controls the tradeoff between diversity and generalisation.
  • KL sweep findings: Higher KL penalties reduce output diversity and generally worsen performance, so the KL coefficient does not provide the intended diversity–generalisation tradeoff.The comparison uses per-input and across-input diversity measures alongside performance evaluation.
  • Evaluation: Figure 11 compares GPT-4 API evaluation win rate against reference outputs, with TL;DR as in-distribution and CNN/DailyMail as out-of-distribution.The generalisation gap is defined as in-distribution minus out-of-distribution performance.
  • OOD evaluation: The OOD splits vary input length, sentiment, and subreddit, using different portions of each factor for training and testing.The length split uses 245 words as the median threshold, while sentiment uses an off-the-shelf classifier.
  • OOD evaluation: The splits are designed to produce approximately 50–50 training and testing distributions, although the subreddit split is not exact.The r/relationships subreddit contains approximately 60% of that data.
  • OOD evaluation: The authors state that these splits do not capture the full range of deployment distribution shifts, but provide a more robust evaluation across several shifts.The limitation concerns the coverage of possible distribution shifts.

J.3 GENERALISATION EVALUATION

This evaluation compares SFT, Best-of-N, and RLHF using a held-out proxy reward model across model sizes and dataset splits, then examines diversity and scaling effects.

  • Generalisation results: BoN generalises better than RLHF, which generalises better than SFT, on proxy reward scores averaged over dataset sizes.The comparison covers in-distribution performance, out-of-distribution performance, and the generalisation gap.
  • Generalisation results: RLHF outperforms BoN at middling model sizes, but BoN scales better and surpasses RLHF above 2.7b parameters.SFT has the worst absolute performance and generalisation in this comparison.
  • Best-of-N scaling: BoN begins improving over SFT beyond 350 million parameters, with the improvement increasing as model size grows.BoN also uniformly improves generalisation across model sizes, while retaining a non-zero generalisation gap.
  • RLHF scaling: Larger RLHF models show increasing performance and a clearer reduction in the generalisation gap.This trend differs from the weaker generalisation-gap trend observed for SFT models.
  • Diversity results: RLHF models have much lower per-input diversity than SFT models across all three reported diversity metrics.Per-input diversity is calculated from sampled outputs for a single input and then averaged.
  • Diversity results: Across inputs, RLHF and SFT have similar EAD scores above 125 million parameters, but RLHF remains less diverse on Sent BERT.NLI diversity for RLHF increases with model size and eventually reaches SFT levels.
  • Best-of-N scaling: Increasing Best-of-N sample count improves performance, but increases the generalisation gap at smaller model sizes.This relationship does not hold for larger model sizes.
  • Diversity–performance tradeoff: The summarisation results reinforce a tradeoff between diversity and win rate across the evaluated policy types.The comparison is shown in Figure 25.
Loading 2310.06452v3…