PaperlayerView exact revision on arXiv

Source-linked AI summary

OPTune: Efficient Online Preference Tuning

Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Goldstein, Tianyi Zhou, Heng Huang

arXiv:2406.07657v1cs.LGcs.CL

TL;DR

Online preference tuning is costly because it requires generating new responses during training, and randomly reusing half of them can hurt instruction-following. OPTune selectively regenerates low-reward responses and weights preference pairs by reward gap. The authors report better models than baselines with 1.27-1.56x training speedup.

  • Problem

    Online preference tuning requires repeated response generation, while randomly reusing half of generated responses degraded instruction-following performance.

    Source: §1 · pp. 1–2

  • Method

    OPTune regenerates responses for low-reward prompts and uses weighted DPO to emphasize pairs with larger reward gaps.

    Source: §1 · p. 2

  • Results

    OPTUNE trains better LLMs than baselines with 1.27-1.56x training speedup across evaluations including instruction-following tests, benchmarks, and human studies.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors propose Best-of-N and PPO as potential application settings for OPTUNE.

    Source: §6 · p. 10

    OPTUNE's performance depends heavily on reward-model accuracy and consistency; reward-model bias or missed preference nuances could lead to suboptimal policy training.

    Source: A Limitations · p. 14

Abstract

from arXiv · show

Reinforcement learning with human feedback~(RLHF) is critical for aligning Large Language Models (LLMs) with human preference. Compared to the widely studied offline version of RLHF, \emph{e.g.} direct preference optimization (DPO), recent works have shown that the online variants achieve even better alignment. However, online alignment requires on-the-fly generation of new training data, which is costly, hard to parallelize, and suffers from varying quality and utility. In this paper, we propose a more efficient data exploration strategy for online preference tuning (OPTune), which does not rely on human-curated or pre-collected teacher responses but dynamically samples informative responses for on-policy preference alignment. During data generation, OPTune only selects prompts whose (re)generated responses can potentially provide more informative and higher-quality training signals than the existing responses. In the training objective, OPTune reweights each generated response (pair) by its utility in improving the alignment so that learning can be focused on the most helpful samples. Throughout our evaluations, OPTune'd LLMs maintain the instruction-following benefits provided by standard preference tuning whilst enjoying 1.27-1.56x faster training speed due to the efficient data exploration strategy.

1 Introduction

OPTune addresses the costs of online preference tuning with selective response regeneration and reward-gap-weighted learning. The authors report better models than baselines alongside faster training.

1 Introduction

  • Response generation and policy training are the most time-consuming online DPO steps; Table 1 reports per-procedure time shares under optimized GPU batches and 512-token response limits.Source: §1 · p. 1, Table 1 · p. 2
  • Randomly reusing half of generated responses degraded instruction-following performance, motivating the search for a fixed-budget alternative that maintains online PT performance.Source: §1 · p. 2
  • OPTune regenerates responses only for prompts whose current responses receive the lowest rewards, reusing higher-reward responses in the training set.This reward-based selection is intended to reduce generation cost without compromising instruction-following or alignment quality.Source: §1 · p. 2
  • Weighted DPO assigns greater weight to response pairs with larger reward gaps, focusing learning on samples that may contribute more to online alignment.The method addresses the loss of reward-gap information when scalar rewards become binary preference labels.Source: §1 · p. 2
  • OPTUNE trains better LLMs than baselines with 1.27-1.56x training speedup.The evaluation includes instruction-following tests, benchmarks, and human studies.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe RLHF’s three-stage pipeline, reward modeling from pairwise preferences, and policy optimization using reward feedback. They also distinguish offline from iterative online DPO and preview OPTUNE’s selective regeneration and weighted training.

2 Preliminaries

  • RLHF comprises supervised fine-tuning, reward-model training on preference data, and reinforcement learning from the SFT checkpoint using reward-model feedback.The standard pipeline is attributed to Ziegler et al. [2019a] and subsequent works.Source: §2 · p. 2
  • The reward model uses pairwise comparisons to learn scalar rewards, with a loss that is the negative log-likelihood of preferring the chosen response over the rejected one.The Bradley-Terry model provides the probabilistic framework for these preferences.Source: §2 · pp. 2–3
  • OPTUNE’s pipeline regenerates responses for low-reward examples, reuses high-quality examples, and applies weighted DPO to focus training on high-utility samples.The figure defines πt as the policy at iteration t, R as the reward model, and ρ as the prompt-selection ratio for regeneration.Source: Figure 1 · p. 3
  • RL fine-tuning maximizes generated-response reward while KL regularization constrains deviation from the SFT model.The KL term is defined on the conditional response distribution; α controls regularization strength and Dp supplies prompts for sampling.Source: §2 · p. 3
  • Offline DPO uses preference data collected before training, whereas online DPO samples two responses per prompt each iteration and updates preferences continuously.The sampled responses are scored by the reward model to identify preferred and dispreferred outputs.Source: §2 · p. 3

3 Method

OPTUNE combines reward-based prompt selection with a weighted DPO objective to improve data-generation and training efficiency in online preference alignment. Its selection strategy regenerates low-scoring prompts while retaining existing responses for other prompts, and wDPO emphasizes pairs with larger reward gaps.

3 Method

  • OPTUNE targets both costs of online preference alignment: iterative data regeneration and policy training.Its proposed method combines reward-based prompt selection with a weighted DPO loss variant.Source: §3 · p. 4

3.1 Data generation efficiency: Reward-based prompt selection

  • The bottom-50% ranked prompts yield more reward gains than the top-50% ranked prompts.This comparison motivates prioritizing low-reward prompts for regeneration.Source: Figure 2 · p. 4
  • OPTUNE’s reward-based selection regenerates responses for prompts whose current responses receive the lowest reward, aiming for larger subsequent reward gains.The method selects a subset of prompts for each iteration and reuses existing responses for prompts not selected.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 Training efficiency: Weighted DPO Loss

  • Online DPO reduces scalar rewards to binary preference labels, losing reward-gap information that wDPO incorporates explicitly to prioritize pairs with larger gaps.The objective weights response pairs using their explicit reward difference.Source: §3.2 · pp. 4–5
  • At ρ = 0.5 and ρ = 0.7, OPTUNE policies are comparable with ρ = 1 while achieving 30% to 50% generation efficiency.Figure 3 reports higher win scores than random-selection models at the same selection ratio.Source: Figure 3 · p. 6

4 Experiment

OPTUNE reduces response-generation effort through selective prompt regeneration while preserving strong performance against random selection and fully online training. Evaluations also compare DPO and weighted DPO training, with benchmark, AlpacaEval, and human-study results reported for the trained policies.

4.1 Experiment setup

  • Experiments use 48k sampled UltraChat prompts, start from Zephyr-7b-sft-full, and compare against offline, fully online, and random-selection baselines.The setup also evaluates free-form instruction following, benchmarks, and human ratings.Source: §4.1 · pp. 6–7

4.2 Results on Generation Efficiency

  • 30–50% generation efficiency accompanies OPTUNE’s performance above random-selection baselines and comparable to fully online training (ρ = 1).The reported trend is that win score rises with larger online-sample ratios.Source: §4.2 · p. 7
  • OPTUNE with DPO matches or surpasses vanilla online DPO in iterations 1 and 3, lags slightly in iteration 2, and achieves 1.27–1.56x training speedup.It saves 30% to 50% on generation time and consistently outperforms random selection; Figure 4 notes that ρ = 0.7 maintains performance while saving 30% generation cost.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 Results on Training Efficiency

  • With OPTUNE, wDPO significantly surpasses DPO and reaches the same win score faster under the compared prompt-selection ratios.Training configurations are held the same across the two losses at each ratio, and their training times are considered almost the same.Source: §4.3 · pp. 7–8, Figure 6 · p. 8
  • Figure 5 reports win score against total training time, and its caption says regenerating responses for half the prompts reaches a score on par with ρ = 1.The reported time includes generation, rewarding, and wDPO training.Source: §4.2 · p. 7, Figure 5 · p. 8

4.4 Evaluation Results on AlpacaEval, Benchmarks, Human Study

  • On AlpacaEval, OPTUNE outperforms random selection and no selection (ρ = 1.0); the table caption identifies ρ = 0.7 as the best setting.The comparison uses iteration-3 wDPO policies and an 805-prompt test set.Source: §4.4 · p. 8, Table 2 · p. 8
  • ρ = 0.7 policies exceed offline and vanilla-online policies on “Average” and score highest on TruthfulQA and GSM8k.The authors associate the latter scores with gains in math problem-solving and factuality.Source: §4.4 · p. 8, Table 3 · p. 8
  • In human ratings, OPTUNE responses are preferred over random-selection responses 24.07% versus 14.81% of the time, and are similar to full-generation outputs in 51.56% of comparisons.Against full generation, OPTUNE is ranked better 23.44% of the time and full generation 25.0%.Source: §4.4 · p. 9

5 Related Work

The related work situates OPTUNE within online RLHF, where on-the-fly sampling is costly, and distinguishes its prompt-selection strategy from response filtering and inference-speedup methods. It also describes the instruction-following and benchmark evaluations used in the paper.

5 Related Work

  • Online RLHF requires costly on-the-fly policy sampling, while offline methods keep preference datasets unchanged and may underperform online approaches.Online DPO/wDPO updates less frequently than PPO, which updates per batch.Source: §5 · p. 9
  • Existing data-selection methods focus on response quality rather than selecting prompts for efficient exploration in iterative RLHF.OPTUNE shares their aim of improving training-data quality but targets prompt selection.Source: §5 · p. 9
  • OPTUNE speeds generation by reusing prior responses, unlike inference methods that reduce KV-cache memory or decoding steps; those methods can also be used on selected prompts.Source: §5 · p. 9
  • The evaluations combine GPT-4-judged instruction-following tests, human side-by-side comparisons, and commonly used language-model benchmarks.The listed benchmarks include TruthfulQA, MMLU, GSM8K, and Hellaswag.Source: §5 · p. 9

6 Discussions & Conclusion

OPTUNE selectively regenerates the lowest-reward responses and weights preference pairs by reward gap. The paper reports improved efficiency while maintaining or improving alignment, and suggests application to other online RLHF algorithms.

6 Discussions & Conclusion

  • OPTUNE reports up to 2x generation efficiency and 1.56x training-efficiency speedup while maintaining or improving alignment with human preferences.Source: §6 · p. 10
  • OPTUNE selectively regenerates the lowest-reward responses and explicitly represents reward gaps in its wDPO objective.Source: §6 · p. 10
  • The authors propose extending prompt selection to Best-of-N and PPO, using PPO's replay buffer to select low-reward prompts.They leave this application for future work.Source: §6 · p. 10

A Limitations

OPTUNE's performance depends heavily on the accuracy and consistency of its reward model.

A Limitations

  • OPTUNE relies heavily on reward-model accuracy and consistency.Source: A Limitations · p. 14
  • A reward model that misses human-preference nuances or has biases may lead to suboptimal policy training despite efficiency gains.Source: A Limitations · p. 14

B Broader Impact

The paper presents OPTUNE as improving alignment with human preferences and training efficiency, with potential relevance to sensitive applications and organizations with limited computational resources.

B Broader Impact

  • OPTUNE is described as improving alignment with human preferences, supporting the paper's stated relevance to sensitive applications.Source: B Broader Impact · p. 14
  • The paper states that OPTUNE accelerates training without compromising output quality, potentially making high-performing models more accessible to resource-limited organizations.Source: B Broader Impact · p. 14

C Hyperparameters

DPO and wDPO training use RMSProp with specified optimization settings, while data generation uses a separate sampling configuration.

C Hyperparameters

D Human Study

The human study compares OPTUNE policies with each other and against random selection, using participant judgments of response helpfulness across prompts from the LIMA test set.

D Human Study

  • Eight volunteer students rated 100 unique response pairs drawn from 50 LIMA test prompts, with each participant judging 50 randomly selected pairs.Each prompt generated one OPTUNE ρ = 0.7 versus OPTUNE ρ = 1.0 pair and one OPTUNE ρ = 0.7 versus random-selection ρ = 0.7 pair; pairs were rated by four people on average.Source: D Human Study · p. 14, Table 4 · p. 15
  • 400 human ratings were collected, with 200 for each comparison, and the results are shown in Table 4.Source: D Human Study · p. 14, Table 4 · p. 15
  • Participants saw the prompt and two model generations, then indicated a preference or a tie.Source: D Human Study · p. 14, Figure 7 · p. 15
  • The guidelines ask raters to assess intent alignment, clarity and precision, relevance, and efficient brevity.Source: D Human Study · p. 15

E The benchmark settings

The benchmark settings section identifies the metrics and few-shot demonstrations used for each benchmark; the following rating-prompt section describes the GPT-4 evaluation prompt.

E The benchmark settings

  • The benchmark settings table lists metrics and few-shot demonstrations for each benchmark under the standard LM-Harness-Evaluation repository setting [Gao et al., 2023].Source: Table 5 · p. 16

F Rating prompt

Loading 2406.07657v1…