PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Rewarding Language Models

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston

arXiv:2401.10020v3cs.CLcs.AI

TL;DR

Human-preference alignment is limited by the available preference data and, in RLHF, by a frozen reward model. This paper trains a language model to generate and judge its own responses, using iterative DPO on self-created preference data. The authors report improvements in both instruction following and reward-modeling ability, while noting that the effect likely saturates and was tested in only one setting over three iterations.

  • Problem

    Human-preference alignment is bottlenecked by the size and quality of preference data, while RLHF also depends on the quality of a frozen reward model trained from those preferences.

  • Method

    Self-Rewarding Language Models use LLM-as-a-Judge prompting to evaluate their own generations and Iterative DPO to train on the resulting preference data.

  • Results

    The training improves both instruction-following capability and reward-modeling ability across iterations.

  • Takeaways & Limitations

    The approach provides a model that can assign rewards to its own future generations, supporting a possible virtuous circle of iterative improvement.

  • Takeaways & Limitations

    The study ran only three iterations in a single setting, and the improvement likely saturates in realistic scenarios.

Abstract

from arXiv · show

We posit that to achieve superhuman agents, future models require superhuman feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cannot then learn to improve during LLM training. In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training. We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. While there is much left still to explore, this work opens the door to the possibility of models that can continually improve in both axes.

1 Introduction

Self-Rewarding Language Models replace fixed or externally trained reward models with a language model that generates responses, judges them, and trains on the resulting preferences. Iterative DPO improves both instruction following and reward-modeling ability.

  • Human-preference alignment is bottlenecked by the size and quality of preference data, and RLHF additionally depends on a frozen reward model trained from it.
  • Self-Rewarding Language Models jointly generate responses and create and evaluate new instruction-following examples for their own training.
  • Each iteration generates candidate responses, assigns them self-rewards through LLM-as-a-Judge prompting, builds preference data, and trains the next model with DPO.
  • Iterative self-alignment is reported to improve instruction following while also increasing the model’s ability to provide higher-quality preference data to itself.

2 Self-Rewarding Language Models

The method combines instruction following with self-instruction creation so one model can generate, judge, and train on new examples. Iterative DPO repeatedly augments training data with model-generated preference pairs.

  • The approach assumes a pretrained language model and a small amount of human-annotated seed data, then trains instruction-following and self-instruction-creation skills.
  • Self-instruction creation uses LLM-as-a-Judge prompting to score the model’s candidate responses, replacing an external reward model with the model itself.
  • The procedure iteratively builds models whose generation and reward-modeling abilities can improve together rather than keeping the reward model fixed.
  • Seed instruction-following data trains the model with supervised fine-tuning, while optional evaluation fine-tuning data teaches response scoring using five additive quality criteria.
  • For each new prompt, the model samples diverse candidate responses, scores them from 0 to 5, and uses the highest- and lowest-scoring responses to construct preference pairs.
  • Iterative DPO trains M2 on AIFT(M1) data and M3 on AIFT(M2) data after M1 is initialized from seed IFT and EFT data.

3 Experiments

Across iterative Self-Rewarding training, instruction-following and reward-modeling abilities improve, with gains extending to human evaluation and AlpacaEval while mathematics, code, and reasoning improve less.

  • Setup: Llama 2 70B models use Open Assistant instruction data and ranked responses to construct instruction- and evaluation-fine-tuning data.The evaluation data trains the model to act as an LLM-as-a-Judge.
  • Instruction Following Ability: 30.5% wins versus 30.9% shows EFT+IFT performs similarly to IFT alone on instruction following.This permits EFT+IFT training to serve as Iteration 1 without reducing instruction-following performance.
  • Instruction Following Ability: 55.5% wins for M2 versus 11.7% for M1 show a large Iteration 2 improvement in head-to-head instruction following.M2 also wins 49.2% versus 14.5% against the SFT baseline.
  • Instruction Following Ability: 47.7% wins for M3 versus 12.5% for M2 show further Iteration 3 gains in head-to-head instruction following.Against the SFT baseline, M3 reaches 62.5% wins versus 9.8%.
  • Instruction Following Ability: AlpacaEval 2.0 win rates over GPT-4 Turbo rise from 9.94% in M1 to 15.38% in M2 and 20.44% in M3.M3 outperforms Claude 2, Gemini Pro, and GPT-4 0613 on this metric.
  • Instruction Following Ability: M3 outperforms many leaderboard models despite starting from a small Open Assistant seed set and generating its own targets and rewards.The comparison includes systems using proprietary alignment data or targets distilled from stronger models.
  • Instruction Following Ability: Instruction-following gains are larger in humanities, STEM, roleplay, writing, and extraction than in math, code, and reasoning.MT-Bench scores increase from 6.78 to 7.25 across training iterations.
  • Reward Modeling Ability: Adding EFT raises pairwise agreement with humans from 65.1% to 78.7% across five reward-modeling metrics.EFT supplies examples of evaluating responses with an LLM-as-a-Judge prompt.

4 Related Work

Prior work improves language models through human or AI feedback, self-created data, and iterative preference training, but generally separates judging from instruction following or keeps reward models fixed.

  • RLHF trains a fixed reward model from human preferences and uses it to optimize the language model with reinforcement learning.
  • DPO avoids training a reward model by directly optimizing the language model from human preferences.
  • Iterative preference-training methods improve over direct preference optimization, with some approaches using fixed external rewards to curate additional examples.
  • RLAIF uses an off-the-shelf language model to generate feedback, then trains a separate fixed reward model for reinforcement learning.
  • Self-Instruct creates prompts and responses to augment fine-tuning data, while other methods curate or distill examples from stronger language models.
  • LLM-as-a-Judge prompting is used to evaluate models, train reward models, and curate data, but combining judging with general instruction following is described as uncommon.

5 Conclusion

The paper introduces models that judge and train on their own generations through iterative preference-data construction and DPO. Across iterations, both instruction following and reward modeling improve, suggesting a possible path toward continual improvement beyond human-preference training.

  • Self-Rewarding Language Models judge and train on their own generations to create preference-based instruction training data.
  • Each iteration uses LLM-as-a-Judge prompting to assign rewards to self-generated responses, then applies Iterative DPO to the resulting preferences.
  • Instruction following and reward-modeling ability both improve across training iterations.
  • The authors describe the improving reward capability as a possible virtuous circle for future instruction-following improvement.
  • The authors state that improvement likely saturates in realistic scenarios, while the approach still permits possible continual improvement beyond human preferences.

6 Limitations

The authors regard the results as preliminary because they evaluated only three iterations in one setting and identified unresolved evaluation, safety, scaling, and reward-hacking questions.

  • Further evaluation should examine safety and the limits of iterative training.
  • The study ran only three training iterations in a single setting, leaving scaling across iterations, models, and capabilities unexplored.
  • Generation length increased across iterations, and its known correlation with estimated quality requires deeper analysis of the reported results.
  • The framework may permit reward hacking, but the circumstances in which it occurs remain unclear.
  • Using language models for both training rewards and some final evaluations warrants deeper analysis despite validation from human evaluation.
  • Safety training within the framework is an open direction because later iterations might evaluate and mitigate harder safety situations.

A.1 Distributions of IFT, EFT and AIFT data

The IFT and AIFT(M1) data distributions overlap substantially, whereas EFT occupies a different region of the embedding space. This distributional difference may help explain EFT's distinct effect on IFT performance.

  • Figure 6 displays the distributions of instructions and responses for IFT, EFT, and AIFT(M1).
  • EFT examples lie in a different part of the embedding space from IFT and AIFT(M1).

A.2 EFT Prompts

The experiments found that an additive score-counting prompt was more effective than the prompt adapted from Li et al. for LLM-as-a-Judge evaluation.

  • The main experiments therefore used the EFT prompt shown in Figure 2.
  • The Li et al. prompt was less effective because it framed evaluation as a multiple-choice problem that was difficult to decompose into aspect-level judgments.
  • The additive score-counting prompt produced significant improvements in EFT performance over the prompt used by Li et al.

A.3 Self-rewarding Models Using IFT Data Only

Using IFT data only, the self-rewarding loop can improve over the SFT baseline, but models trained with EFT data perform better at the same iteration count and increasingly widen the gap.

  • The IFT-only sequence uses M0 as the unfine-tuned base, M′1 after IFT SFT, then M′2 and M′3 after successive AIFT DPO training.
  • Without EFT data, models often failed to follow the scoring format or converged to scores of 4, yielding only 541 and 429 valid preference pairs across two datasets.
  • EFT data improves performance at the same iteration count, with the gap widening in later iterations.
  • M′2 and M′3 improve over the SFT baseline, but lag far behind corresponding EFT-initialized models M2 and M3.
  • Instruction complexity is scored from 1 to 10, with 10 representing questions requiring reasoning or decomposition into subproblems.
  • The evaluation categorizes AlpacaEval instructions by category, complexity, and expected response length.

A.4 Preference optimization outperforms augmenting with positive examples only

The alternative self-training procedure adds model-curated, high-quality instruction–response examples to supervised fine-tuning instead of constructing preference data. In experiments, this approach did not improve performance over the SFT baseline.

  • The alternative procedure adds model-curated instruction–response examples to the supervised fine-tuning seed set instead of constructing preference data.It follows other self-training approaches and does not use preference optimization.
  • Only candidate responses receiving a perfect score of 5 were included in this supervised fine-tuning variant.
  • 11,254 perfect-score examples yielded 29% wins versus 30% for the SFT Baseline, showing no improvement.The mixing weight was optimized during training.

A.5 Augmented Prompt Generation Using Newly Trained Models

The appendix examines prompt generation, AlpacaEval test clustering, and benchmark behavior of newly trained Self-Rewarding models. Results indicate category-specific MT-Bench gains, broadly maintained NLP benchmark performance, and limitations for text-completion evaluations.

  • A.5 Augmented Prompt Generation Using Newly Trained Models: A fixed pool of augmented prompts was created in advance with ChatLlama 70B for time efficiency.
  • A.5 Augmented Prompt Generation Using Newly Trained Models: In an interactive system, real users could ideally supply augmented prompts to align training with user requirements.
  • A.5 Augmented Prompt Generation Using Newly Trained Models: The experiments test whether newly trained Self-Rewarding models can generate new prompts through in-context learning instead of using ChatLlama 70B.
  • A.6 AlpacaEval Test Sample Clustering: GPT-4 categorized AlpacaEval test instructions by category, complexity, and expected response length, identifying 20 instruction categories.
  • A.7 NLP Benchmark Results and MT-Bench Results: Self-Rewarding models mostly maintain performance relative to Llama 2 and the SFT Baseline on the listed NLP benchmarks.These models were fine-tuned on substantially different instruction-following prompts.
  • A.6 AlpacaEval Test Sample Clustering: MT-Bench fine-grained results report performance for each problem category.
  • A.7 NLP Benchmark Results and MT-Bench Results: Self-reward is especially effective for improving writing, role-playing, extraction, and STEM tasks on MT-Bench.
  • A.7 NLP Benchmark Results and MT-Bench Results: ARC-Challenge, HellaSwag, SIQA, PIQA, and OBQA use highest-log-probability multiple-choice selection, unlike the algorithm’s optimization objective.Therefore, these results may not reflect the models’ true capabilities.
Loading 2401.10020v3…