PaperlayerView exact revision on arXiv

Source-linked AI summary

RLHF Workflow: From Reward Modeling to Online RLHF

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang

arXiv:2405.07863v3cs.LGcs.AIcs.CLstat.ML

TL;DR

The report addresses the limited availability of open-source implementations for online iterative RLHF and the infeasibility of online human feedback for resource-constrained communities. It constructs proxy preference models from diverse open-source datasets, develops and implements an online iterative RLHF workflow, and reports strong performance across chatbot and academic benchmarks.

  • Problem

    Open-source RLHF projects are largely confined to offline learning, while online human feedback is usually infeasible for resource-constrained open-source communities.

  • Method

    The report uses on-policy sampling and external preference signals from a proxy preference model trained on diverse open-source preference datasets, alongside direct preference learning algorithms.

  • Results

    The resulting model significantly improves conversation quality, outperforms comparable smaller open-source models on conversation and instruction-following benchmarks, and shows no significant regression against SFT on academic benchmarks.

  • Takeaways & Limitations

    The report provides a reproducible online iterative RLHF workflow with publicly available data, code, models, and hyper-parameter choices.

Abstract

from arXiv · show

We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature. However, existing open-source RLHF projects are still largely confined to the offline learning setting. In this technical report, we aim to fill in this gap and provide a detailed recipe that is easy to reproduce for online iterative RLHF. In particular, since online human feedback is usually infeasible for open-source communities with limited resources, we start by constructing preference models using a diverse set of open-source datasets and use the constructed proxy preference model to approximate human feedback. Then, we discuss the theoretical insights and algorithmic principles behind online iterative RLHF, followed by a detailed practical implementation. Our trained LLM achieves impressive performance on LLM chatbot benchmarks, including AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning (SFT) and iterative RLHF can obtain state-of-the-art performance with fully open-source datasets. Further, we have made our models, curated datasets, and comprehensive step-by-step code guidebooks publicly available. Please refer to https://github.com/RLHFlow/RLHF-Reward-Modeling and https://github.com/RLHFlow/Online-RLHF for more detailed information.

A Comprehensive Practical Alignment Recipe of Iterative Preference Learning

The report is authored by researchers from Salesforce AI Research and the University of Illinois Urbana-Champaign.

  • The author list includes Hanze Dong, Wei Xiong, Bo Pang, and Haoxiang Wang.
  • The author list also includes Han Zhao, Yingbo Zhou, Nan Jiang, and Doyen Sahoo.
  • Caiming Xiong and Tong Zhang are listed with dagger markers.
  • The affiliations are Salesforce AI Research and the University of Illinois Urbana-Champaign.

1 Introduction

The introduction frames RLHF as preference-based alignment and reviews offline and online approaches, emphasizing the practical limitations of offline coverage, PPO complexity, and costly human feedback. The project focuses on accessible online iterative RLHF using proxy preference models trained from diverse open-source datasets.

  • RLHF aligns LLM outputs with human values by learning from relative preferences rather than absolute reward ratings.
  • The Bradley-Terry model approximates preference signals with a reward function, although it may not fully capture complex human preferences.
  • PPO-based RLHF requires extensive tuning and simultaneously loads actor, critic, reward, and reference models, creating substantial computational and GPU-memory demands.
  • Direct preference optimization avoids explicit reward-model construction, is generally easier to tune, and uses fewer computational resources than deep RL methods.
  • Offline RLHF methods cannot query preferences during training, and finite datasets may poorly cover the prompt-response space, reducing performance on out-of-distribution data.
  • The project uses proxy preference models trained on diverse open-source datasets because online human feedback is usually unaffordable for open-source communities.

2 Reward Modeling as Human Feedback Approximation

The section constructs proxy human-feedback models from open-source preference data using Bradley-Terry reward modeling and instruction-following preference modeling. It evaluates these models on RewardBench and examines their length bias, finding that both reward models favor longer responses to some degree.

  • The training data comprises mix1, a mixture of HH-RLHF, SHP, UltraFeedback, and Summarization, and mix2, all collected open-source datasets.
  • Bradley-Terry Reward Model: The Bradley-Terry reward model initializes from an SFT model, replaces its final layer with a scalar linear head, and trains with negative log-likelihood for maximum likelihood estimation.
  • Preference Model: The preference model frames each response pair as an instruction-following classification task and randomizes response order to mitigate position bias.
  • Evaluation Result: RewardBench evaluates the models across Chat, Chat-Hard, Safety, and Reasoning, with the preference model outperforming the Bradley-Terry model on coding- and math-related reasoning tasks.
  • Evaluation Result: Both reward models show positive reward-length correlations, with mean Pearson coefficients of 0.19 for UltraRM-13B and 0.06 for the authors’ BT reward.

3 Iterative Policy Optimization

The framework combines theoretical online iterative RLHF with practical direct preference learning, using exploitation by a main agent and uncertainty-driven exploration by an enhancer. It supports hybrid offline-online data and practical variants such as temperature changes, rejection sampling, and DPO-based updates.

  • The framework develops online iterative RLHF algorithms with theoretical insights and implementation details in a direct preference learning style for stable, efficient training.
  • Algorithmic principles: The hybrid framework combines an initial offline dataset with online data collected during training and uses large batches for sparse updates.
  • Algorithmic principles: Each iteration exploits historical data through a main agent that selects the best policy under the maximum-likelihood reward estimate.
  • Algorithmic principles: The enhancer explores directions with greater uncertainty relative to the main agent while maintaining a moderate policy divergence.
  • Theoretical guarantees: For any ϵ > 0, suitable batch size m = eO(de/ϵ^2) and at most T = eO(de) iterations yield a policy with KL-regularized value gap plus exploration error ≲ϵ with high probability.
  • Practical implementation: DPO is used for simplicity, but the framework can also combine with oracle algorithms such as PPO and InfoNCA that approximate KL-regularized optimization.
  • Practical implementation: The practical BT-reward implementation applies DPO to accumulated preference data, samples responses at temperatures 1.0 and 0.7, and forms preference pairs from the best and worst responses.
  • Practical implementation: Rejection sampling explores by selecting the highest-reward response among n samples, while its policy KL divergence is upper bounded by log n − n−1/n.

4 Evaluation of the Model

The resulting model improves conversation quality over open-source and larger aligned baselines, while retaining or improving performance on several academic benchmarks. Ablations show that length penalties reduce verbosity and improve length-controlled evaluation, though they can hurt Chat-Arena-Hard performance.

  • Main Results: Online iterative RLHF significantly improves conversation quality and outperforms smaller open-source models, Tulu-2-DPO-70B, and GPT-3.5-turbo-1106 on conversation benchmarks.The iteratively trained DPO model also consistently outperforms the vanilla offline DPO baseline.
  • Academic Task: The model shows no significant regression against SFT on academic benchmarks and outperforms SFT on GSM-8K, MMLU, TruthfulQA, and ARC.The authors attribute this to iterative DPO helping the model leverage capacities acquired during pre-training and SFT.
  • Evaluation Caveats: Benchmark results require caution because finite test sets may not represent real-world scenarios, and small models may overfit or hack benchmarks.The authors also report that GPT-based evaluation depends strongly on configuration and compare one configuration with human evaluation.
  • Length Penalty Ablation: Length penalties shorten responses and improve length-control AlpacaEval-2 win rate and some academic benchmarks.The ablation evaluates response length averaged over Chat-Arena-Hard responses.
  • Length Penalty Ablation: The length-penalty model performs worse on Chat-Arena-Hard, suggesting that this benchmark may need a length-control version for more reasonable evaluation.The authors connect the intervention to a response-length bias potentially caused by reward-model bias.

5 End Note and Future Direction

The report studies an online iterative RLHF workflow using proxy preference signals and releases materials intended to make the pipeline reproducible. It identifies preference-signal quality, exploration, and response-length bias as directions for further work.

  • Contribution: The workflow uses on-policy sampling and external preference signals from a proxy preference model trained on diverse open-source preference datasets.The report provides data, code, models, and hyper-parameter choices for reproduction.
  • Future Directions: The iterative RLHF pipeline depends heavily on preference-signal quality because this project approximates human feedback with a proxy scalar reward model.The authors propose exploring multi-head rewards and classification-based activation strategies.
  • Future Directions: More effective exploration methods remain an open direction beyond rejection sampling as a heuristic exploration strategy.The report identifies exploration design as a separate unresolved component of online iterative RLHF.
  • Future Directions: Iterative RLHF amplifies the tendency of post-RLHF models to produce longer responses, motivating further algorithmic or post-training mitigation.A preliminary length-penalty study is presented as an initial response to this issue.

A Authorship and Credit Attribution

The project distributes authorship across reward modeling, SFT, iterative RLHF, evaluation, dataset preparation, coding, experimentation, and advising. Contributions cover both released models and the supporting research infrastructure.

  • Individual Contributions: HD contributed to iterative DPO and RLHF code, SFT and BT-RM training, hyper-parameter tuning, evaluation, and the released BT reward model.HD also provided preference data and conducted GPT-based evaluation of generative models.
  • Individual Contributions: WX developed Bradley-Terry and preference-model code, led reward and preference-model experiments, organized the project, and prepared the public online iterative DPO implementation.WX also contributed to dataset search, tuning, and model evaluation.
  • Individual Contributions: BP led final SFT and RLHF experiments, developed SFT recipes, tuned SFT and RLHF systems, and conducted GPT-based and academic benchmark evaluations.BP also contributed to paper writing and released SFT and RLHF models.
  • Individual Contributions: HW initiated pairwise preference-model training code, conducted reward-model experiments, curated prompt and preference data, and contributed to evaluation, analysis, writing, and figures.The contribution spans both data preparation and reward-modeling presentation.
  • Advising and Support: HZ, YZ, NJ, DS, CX, and TZ supported and advised the project, provided computational resources, and suggested experiments and writing.

B.1 Preference Datasets

The project builds preference-model training data from multiple open-source datasets covering dialogue, Reddit preferences, safety, general instruction following, coding, and complex reasoning. It filters noisy samples, compares dataset mixtures, and considers several preference-modeling approaches.

  • Dataset Composition: The training corpus mixes open-source datasets containing human, expert, or model-generated preference signals across dialogue, instruction following, safety, coding, and reasoning.Examples include HH-RLHF, SHP, HelpSteer, PKU-SafeRLHF, UltraFeedback, CodeUltraFeedback, UltraInteract, and Distilabel-Capybara.
  • Dataset Composition: HH-RLHF provides human-annotated pairwise preferences over conversation histories and alternative Claude responses.
  • Dataset Composition: SHP uses Reddit questions and comment pairs, retaining samples with a score ratio greater than 2 and at most five pairs per prompt.
  • Dataset Composition: HelpSteer supplies five human-annotated attributes, while PKU-SafeRLHF supplies expert comparisons with separate helpfulness and safety signals.These attributes or signals can be converted into pairwise comparisons for preference learning.
  • Dataset Composition: UltraFeedback covers 64k diverse prompts with four generated responses per prompt and GPT-4 preferences across instruction-following, truthfulness, honesty, and helpfulness.CodeUltraFeedback applies a similar process to coding, with annotations from GPT-3.5.
  • Dataset Composition: UltraInteract targets complex reasoning through preference trees, while Distilabel-Capybara and Distilabel-Orca provide multi-turn or instruction-focused dialogue data.
  • Data Filtering: The preprocessing removes low-quality, meaningless, empty-round, incorrectly labeled, and small-margin preference samples.The authors state that small margins tend to produce noisy preference signals.
  • Dataset Mixtures and Modeling: Mix1 combines HH-RLHF, SHP, UltraFeedback, and Summarization, whereas Mix2 includes all datasets in Table 5 and adds more reasoning and safety preference pairs.The study also compares prompting, Bradley-Terry reward modeling, and preference modeling for representing preference signals.

B.2 Benchmark Details

The project evaluates models with single-turn, multi-turn, and human-preference-oriented chatbot benchmarks, using GPT-4-based scoring and head-to-head comparison.

  • AlpacaEval-2: AlpacaEval-2 uses 805 single-turn prompts and GPT-4-Preview head-to-head judgments to compute win rates.A length-control variant is included to mitigate GPT-4's length bias.
  • MT-Bench: MT-Bench evaluates 160 prompts across eight areas through two-turn conversations scored from 1–10 by GPT-4.The final score averages the ratings for the two turns.
  • Chat-Arena-Hard: Chat-Arena-Hard contains 500 prompts from live Chatbot Arena data and evaluates specificity, knowledge, complexity, problem-solving, creativity, accuracy, and application.The benchmark also provides clearer separability among models than AlpacaEval-2 and MT-Bench.
  • Benchmark overview: The project summarizes its selected benchmarks in Table 6.

B.3 Other details

Additional project details cover prompt visualization, open-source SFT data, offline DPO settings, training parameters, and supplemental training and evaluation plots.

  • Prompt Visualization: Prompt collections are visualized with Nomic Atlas using nomic-embed-text-v1.5 embeddings.The visualization is described as part of the project's prompt-analysis materials.
  • SFT Data List: SFT training uses open-source instruction data from ShareGPT, Evol-Instruct, SlimOrca, MathInstruct, Magicoder-Evol-Instruct, GPT4-LLM, OrcaMath, GPTeacher, and UltraInteract.
  • Offline Vanilla DPO: Offline Vanilla DPO uses the Nectar dataset for one epoch with batch size 128, learning rate 5e-7, and cosine decay.
  • Hyperparameters: Training parameters are listed in Table 7.

C Case Studies

The case studies qualitatively examine responses before and after online RLHF, emphasizing improvements in detail, formatting, clarity, and structure.

  • Case Studies: After online RLHF, responses became more detailed and better formatted, often using bullet points, highlights, bold text, and enumeration.The report identifies improved clarity and structure as key factors contributing to higher win rates.
  • Case Studies: Figures 10, 11, and 12 present three online-RLHF response case studies.
Loading 2405.07863v3…