PaperlayerView exact revision on arXiv

Source-linked AI summary

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Yuda Song, Gokul Swamy, Aarti Singh, J. Andrew Bagnell, Wen Sun

arXiv:2406.01462v2cs.LGcs.AIcs.CL

TL;DR

The paper asks how online RLHF and offline contrastive preference methods differ statistically, given limited rigorous theoretical separation. It analyzes dataset coverage, proves stronger requirements for offline methods than online RL, and introduces HyPO, which combines offline preference optimization with online KL control and outperforms DPO empirically.

  • Problem

    Prior work lacked a rigorous theoretical separation between online RLHF and offline contrastive preference methods despite empirical evidence favoring online approaches.

    Source: §1 · p. 2

  • Method

    The paper analyzes convergence and reverse-KL guarantees under global and partial coverage, then derives HyPO using offline data for the DPO objective and online samples for reverse-KL control.

    Source: §1 · p. 2

  • Results

    HyPO outperforms DPO across all reported TL;DR summarization metrics and on general chat benchmarks, while online RL requires only weaker partial coverage and can control reverse KL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Coverage is critical to preference-learning convergence, and the theoretical separation motivates hybrid methods that retain offline computational simplicity while adding online KL control.

    Source: §1 · p. 2

    The offline analysis relies on a strong global-coverage assumption for the Bradley–Terry loss, limiting what can generally be guaranteed from offline data.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques -- online reinforcement learning (RL) such as Proximal Policy Optimization (PPO) and offline contrastive methods such as Direct Preference Optimization (DPO) -- were positioned as equivalent in prior work due to the fact that both have to start from the same offline preference dataset. To further expand our theoretical understanding of the similarities and differences between online and offline techniques for preference fine-tuning, we conduct a rigorous analysis through the lens of dataset coverage, a concept that captures how the training data covers the test distribution and is widely used in RL. We prove that a global coverage condition is both necessary and sufficient for offline contrastive methods to converge to the optimal policy, but a weaker partial coverage condition suffices for online RL methods. This separation provides one explanation of why online RL methods can perform better than offline methods, especially when the offline preference data is not diverse enough. Finally, motivated by our preceding theoretical observations, we derive a hybrid preference optimization (HyPO) algorithm that uses offline data for contrastive-based preference optimization and online data for KL regularization. Theoretically and empirically, we demonstrate that HyPO is more performant than its pure offline counterpart DPO, while still preserving its computation and memory efficiency.

1 Introduction

The paper studies why online RLHF and offline contrastive preference methods can differ despite sharing offline preference data. Through dataset coverage, it proves distinct convergence and KL guarantees and proposes HyPO as a more performant hybrid approach.

1 Introduction

  • Prior work treated online RL and offline contrastive methods as theoretically equivalent, but recent empirical findings motivate a rigorous analysis of their statistical separation.The paper frames dataset coverage as the key lens for explaining when the methods differ.Source: §1 · pp. 1–2
  • Global coverage is necessary for offline contrastive methods such as DPO to converge optimally, whereas weaker partial coverage suffices for online RLHF.The separation is attributed to differences in reward modeling and online versus offline regularization.Source: §1 · p. 2
  • Offline methods can have infinite reverse KL under partial coverage, while online RLHF can directly control reverse KL using online samples.Thus RLHF has stronger guarantees for staying close to the reference policy in realistic partial-coverage settings.Source: §1 · p. 2
  • HyPO combines offline DPO optimization with online samples for explicit reverse-KL control, preserving computational simplicity while addressing offline-method deficiencies.The method uses offline data for the DPO objective and online data for KL regularization.Source: §1 · p. 2
  • HyPO outperforms DPO on all reported TL;DR summarization metrics, including GPT4 win-rate and reverse KL, and also improves general chat performance.It additionally mitigates overfitting observed in offline contrastive methods.Source: §1 · p. 2
  • Function approximation explains how DPO and RLHF can reduce the likelihood of both preferred and rejected responses while extrapolating toward unseen optimal actions.Without function approximation, DPO can instead increase the likelihood of sub-optimal actions.Source: §1 · p. 2

2 Related Work

Related work has compared online RLHF and offline contrastive preference methods, studied preference-data effects, and reported advantages for on-policy sampling. This paper adds a coverage-based theoretical explanation and leaves hybrid-method analysis beyond its scope for future work.

Preference Fine-Tuning

  • The paper assumes preferences arise from an underlying utility function without intransitivities and leaves extending its analysis to hybrid methods with online preference labels for future work.This scope boundary appears alongside prior work on hybrid preference-learning methods.Source: §2 · p. 3

Understanding PFT

  • Prior studies report better performance from online methods and attribute it to on-policy sampling or the ability to eliminate more undesirable policies.Other work finds that combining contrastive losses with interactive training is preferable in practice.Source: §2 · p. 3
  • The paper complements mostly empirical comparisons with a rigorous coverage-based explanation of online–offline differences and a new algorithm targeting offline contrastive weaknesses.Earlier work derived both method families from the same minimum relative entropy objective and sometimes treated them as equivalent.Source: §2 · p. 3
  • DPO’s simultaneous decrease in preferred and rejected response likelihoods is explained as natural when offline data contains only sub-optimal responses and function approximation enables extrapolation.This analysis highlights function approximation as important to offline contrastive success.Source: §2 · p. 3

Coverage

  • Coverage measures how well an offline distribution covers the support of a policy of interest and is central to analyses of offline, online, and hybrid RL.Both online RLHF and offline contrastive methods rely on offline preference data, motivating coverage analysis.Source: §2 · p. 3

3 Preliminaries

The paper formulates preference fine-tuning as a contextual-bandit problem that balances human reward against deviation from a reference policy. It contrasts online RLHF, which requires online sampling and auxiliary modeling, with offline contrastive objectives such as DPO.

Preliminaries

  • The contextual-bandit formulation treats prompts as contexts, generated sequences as actions, and policies as response distributions over a finite but combinatorially large action space.The prompt distribution is ρ, and the reward assigns value to each context–response pair.Source: §3 · p. 3
  • Preference data consists of context, preferred-response, and rejected-response triplets sampled from an offline response distribution and labeled by a Bradley–Terry model.The model uses an underlying human implicit reward and rules out intransitive preferences.Source: §3 · pp. 3–4
  • The objective maximizes expected human implicit reward while penalizing reverse KL divergence from the reference policy.The optimal policy is defined as a maximizer of this reward–regularization objective.Source: §3 · p. 4
  • Reverse KL requires samples from the policy being optimized, whereas forward KL can be evaluated using offline reference-policy samples.This distinction is central to separating online RLHF from offline contrastive methods.Source: §3 · p. 4

Online RLHF Algorithms

  • Online RLHF trains a reward model from preference data and then optimizes the policy against that learned reward using methods such as PPO.The policy-optimization stage requires extensive online sampling and may require an additional critic model.Source: §3 · p. 4

Offline Contrastive Algorithms

  • Offline contrastive methods avoid separate reward modeling and directly optimize a ranking objective on the offline preference dataset; DPO is the primary example considered.IPO is also discussed, with its technical details deferred to the appendix.Source: §3 · p. 4

4 Offline Contrastive Methods Require a Stronger Coverage Condition than Online RL Methods

The analysis separates offline contrastive methods from online RL through dataset coverage: DPO and IPO require global coverage, whereas online RL can achieve guarantees under weaker local KL-ball coverage. The section also motivates HyPO, which combines offline contrastive learning with online KL regularization.

4 Offline Contrastive Methods Require a Stronger Coverage Condition than Online RL Methods

  • Global coverage is necessary for offline contrastive methods such as DPO and IPO, while local KL-ball coverage suffices for online RL methods.This separation is attributed to differences in reward modeling and offline versus online regularization.Source: §4 · p. 5, §4.3 · p. 8

4.1 Global Coverage is Necessary for Offline Contrastive Algorithms

  • Without global coverage, DPO can generate out-of-distribution responses and cannot guarantee performance, even under strong in-distribution reward-learning assumptions.The paper gives a construction with a returned DPO policy whose objective value is J(π) = −∞.Source: §4.1 · pp. 5–7
  • IPO likewise fails under partial coverage: there exists an in-distribution population-loss minimizer with J(π) = −∞.The result holds despite assuming the returned policy achieves the smallest in-distribution population error.Source: §4.1 · p. 6

4.2 Global Coverage is Sufficient for Offline Contrastive Algorithms

  • Under global coverage, DPO obtains a performance guarantee, and the proof extends to reward-learning guarantees that preserve relative rewards between response pairs.The sufficient result applies when the DPO policy satisfies the stated in-distribution learning condition.Source: §4.2 · p. 7

4.3 Online RL Method Under Partial Coverage

  • Under bounded learned rewards and local KL-ball coverage, online RL keeps reverse KL bounded and guarantees performance without requiring global coverage.The resulting coverage factor is no larger than the global-coverage factor, so the RLHF regret bound can be strictly better than DPO’s.Source: §4.3 · pp. 7–8
  • HyPO uses offline data for the DPO loss and online samples for explicit KL regularization, updating the policy with ℓ_dpo − λℓ_kl.The algorithm samples offline preference triplets and unlabeled responses generated by the current policy.Source: §4.3 · p. 8
  • The analysis leaves single-policy coverage as an open boundary: relaxing local KL-ball coverage may require computationally difficult pessimism to obtain meaningful regret guarantees.The paper conjectures that bounded KL alone may be insufficient because density ratios can still be exponentially large.Source: §4.3 · p. 8

5 Hybrid Preference Optimization: Regularizing Offline Learning with Unlabeled Online Samples

HyPO combines offline DPO training with online reverse-KL regularization to bridge offline contrastive and online RL methods. Experiments report improved control of divergence and performance relative to DPO, including reduced overfitting during extended training.

5.1.2 General Chat

  • The general-chat evaluation compares the base model, DPO, and HyPO across AlpacaEval 2.0, MT-bench, and Open LLM leaderboard tasks.Results are reported in Table 2 for AlpacaEval and MT-bench and in Table 3 for MMLU, GSM8K, Arc, TruthfulQA, and HellaSwag.Source: Table 2 · p. 9, Table 3 · p. 10, §5.1.2 · p. 10

5 Hybrid Preference Optimization: Regularizing Offline Learning with Unlabeled Online Samples

  • HyPO theoretically guarantees partial-coverage performance by explicitly constraining reverse KL, whereas unrestricted DPO can have unbounded reverse KL.The practical filtering procedure is replaced by a constrained optimization formulation whose Lagrangian yields HyPO.Source: §5 · p. 9
  • HyPO uses offline samples for the DPO loss and unlabeled online samples to optimize reverse-KL regularization.The method implements the constrained objective through a Lagrangian, with λ treated as a hyperparameter because the reward bound is unknown.Source: §5 · p. 9
  • HyPO’s reverse-KL term requires online sampling from the learned policy, unlike related methods that regulate DPO with forward KL from reference-policy samples.Unlike iterative RLHF methods that label online samples with an additional reward model, HyPO uses unlabeled online samples.Source: §5 · p. 9

5.1.1 Summarization

  • HyPO outperforms DPO on TL;DR in GPT4 win-rate and reverse KL, while avoiding the additional reward model and value network required by PPO.HyPO remains below PPO in win-rate in the cited comparison but preserves DPO’s computational advantages.Source: Table 1 · p. 9, §5.1.1 · p. 10

5.1.3 HyPO Mitigates Overfitting in Contrastive Methods

  • HyPO better prevents validation-KL deviation from the reference policy than DPO when both train for five epochs on summarization.The experiment uses Pythia-2.8B and evaluates the first 300 validation examples, targeting overfitting from excessive training.Source: §5.1.3 · p. 10

6 Function Approximation Coverage: Can Fine-tuned Policies Extrapolate?

Under global coverage with linear function approximation, preference fine-tuning can extrapolate toward unseen optimal responses; without function approximation, DPO may assign probability to unseen suboptimal responses.

6 Function Approximation Coverage: Can Fine-tuned Policies Extrapolate?

  • Figure 1 evaluates median validation reverse KL for DPO and HyPO after five epochs on TL;DR across three random seeds.Shaded regions show the minimum and maximum across repetitions.Source: Figure 1 · p. 11
  • The illustrative linear example uses three responses, with data covering only y1 and y2 while the ground-truth reward is linear in their feature representations.The reference policy is uniform, and the optimal response is absent from the covered support.Source: §6 · p. 11
  • The construction shows that learned reward parameters can reduce likelihood for observed responses while increasing likelihood for an unseen response under the softmax policy.This behavior follows from the linear reward parameterization and the policy relation π(y) ∝ πref(y) exp(br(y)/β).Source: §6 · p. 11
  • Function approximation and dataset coverage let DPO extrapolate by increasing unseen optimal-response likelihood while decreasing likelihood for observed preferred and rejected responses.The paper presents this as an explanation for how preference fine-tuning can improve SFT models even when optimal responses are absent from training data.Source: §6 · pp. 10–11

6.1.1 Extrapolation with Function Approximation

  • Both online RL and DPO increase the probability of an unseen optimal response under linear function approximation, while preferred-response probability decreases.The synthetic experiment uses global coverage in a 16-dimensional linear feature space and tracks preferred responses against the best response.Source: §6.1.1 · p. 12, §6.1 · p. 12

6.1.2 Extrapolation without Function Approximation

  • Without function approximation, DPO can increase probability for unseen suboptimal responses rather than the unseen optimum, demonstrating failed extrapolation.In the multi-armed-bandit setting, actions are independent, so observing one action provides no information about another.Source: §6.1.2 · p. 12, Figure 2 · p. 12

7 Discussion

The discussion identifies theoretical, empirical, and coverage-related limitations, while noting that online samples are used only for reverse-KL regularization and not additional preference acquisition.

7 Discussion

  • The theoretical analysis addresses statistical properties rather than optimization, so it is complementary to optimization-focused work.This limits the scope of the paper’s explanation of algorithmic behavior.Source: §7 · p. 13
  • HyPO remains below online RLHF empirically, suggesting the theory does not explain every component of online RLHF’s benefit.The paper hypothesizes that learned reward functions may generalize better.Source: §7 · p. 13
  • The necessity of KL-ball coverage for online RL remains unclear, and the paper conjectures that single-policy coverage is insufficient under reverse-KL regularization.The conjecture is motivated by possible exponential error amplification even when reverse KL is bounded.Source: §7 · p. 13
  • HyPO uses online samples only to regularize reverse KL; using them for new preference labels or length control is left open.These are proposed directions rather than components evaluated in the presented method.Source: §7 · p. 13
  • The method may not explicitly address hallucinations or toxic behavior, a common shortcoming of general-purpose fine-tuning algorithms.The paper presents this as a further limitation and direction.Source: §7 · p. 13

B.1 Results for IPO

The appendix gives negative results for IPO and DPO under incomplete coverage, showing that distributional guarantees can coexist with unbounded objective or KL behavior.

B.1 Results for IPO

  • There exists an IPO-returned policy satisfying the in-distribution guarantee for which the objective J(π) equals −∞.This result is established when the reference policy fails the relevant coverage assumption.Source: §B.1 · p. 19
  • Under partial coverage, IPO can have unbounded reverse KL from the reference policy even when its response probabilities satisfy the stated preference relation.The construction sets πipo(y2)=α and assigns the remaining mass to y3 under the displayed condition.Source: §B.1 · p. 19

B.2 DPO Has Vacuous Forward KL

  • For DPO, the normalization term in the forward-KL decomposition can be vacuous without additional assumptions.The paper identifies this term as the source of the difficulty in relating forward KL to reward-learning guarantees.Source: §B.2 · pp. 19–20
  • There exist DPO policies satisfying the stated reward-learning assumption whose forward KL KL(πref||πdpo) is arbitrarily large.Thus a good in-distribution guarantee alone does not control the forward divergence.Source: §B.2 · p. 20

C.1 Proof of Proposition 4.1

The proof constructs a DPO policy that under incomplete reference support assigns positive probability outside that support, making reverse KL to the reference unbounded.

C.1 Proof of Proposition 4.1

  • When the reference policy covers only y1 and y2, a DPO policy can assign positive probability to y3 while remaining within the stated in-distribution guarantee.The proof derives π(y1)<π*(y1) and π(y2)<π*(y2), forcing π(y3)>0.Source: §C.1 · pp. 20–21
  • Because πref(y3)=0 while π(y3)>0, KL(π||πref) is unbounded.The divergence failure follows directly from support mismatch.Source: §C.1 · pp. 20–21

C.2 Proof of Theorem 4.2

The proof establishes Theorem 4.2 under its stated assumptions and then extends the same partial-coverage guarantee to HyPO. The HyPO argument reuses the theorem’s proof structure while addressing its distinct reward-function and constraint conditions.

C.2 Proof of Theorem 4.2

  • Theorem 4.2 is proved for any reference policy satisfying Assumption 4.2 and any RLHF policy satisfying the stated event condition.The proof invokes Lemma 4.1 and related lemmas under Assumptions 4.2–4.4.Source: §C.2 · pp. 21–22
  • Lemma 4.1 supplies the intermediate result needed to invoke Assumption 4.2 in the proof of Theorem 4.2.The derivation uses the RLHF policy’s exponential reparameterization relative to the reference policy and Jensen’s inequality.Source: §C.2 · p. 21

D.1 Theoretical guarantee

  • HyPO inherits a partial-coverage guarantee for policies satisfying the event condition in Lemma A.2, with the same reference-policy requirement used in Theorem 4.2.The guarantee is stated for the constrained optimization version of HyPO and its implicit reward-function reparameterization.Source: §D.1 · p. 22
  • The HyPO proof adapts Theorem 4.2 by replacing the unavailable global boundedness condition with the constraint used by HyPO.The argument also verifies the premise of Lemma A.1 and uses the policy–implicit-reward reparameterization to invoke Lemma A.2.Source: §D.1 · p. 22
  • The HyPO discussion connects its guarantee to DPO’s negative result by noting that the KL constraint prevents the construction underlying Proposition 4.1.This observation is made for inputs with positive coverage density under the stated KL condition.Source: §D.1 · p. 22

D.2 Experiment details

The experiments use Pythia models and the TL;DR preference data for summarization, then Llama3-8B-Instruct and Ultrafeedback for general chat. HyPO uses online KL optimization and requires more training time than DPO in both settings.

D.2.1 Summarization

  • The summarization experiments use Pythia 1.4B and 2.8B models with the TL;DR human-reference and preference datasets.The reference data includes 117k training, 6.45K validation, and 6.55K testing examples; the preference data includes 92.9K training and 83.8K validation examples.Source: §D.2.1 · p. 23
  • HyPO optimizes online KL with RLOO, using two generations per prompt and trajectory-level KL in the summarization setup.Reward evaluation and KL computation use the full reference-dataset validation split, while GPT win rate uses 600 validation samples.Source: §D.2.1 · p. 23
  • DPO takes 1–2 hours and HyPO 4–5 hours in the summarization experiments, using four GPUs in parallel.The runs use a mixture of Nvidia A6000 and L40 GPUs with 48 GB VRAM.Source: §D.2.1 · p. 23

D.2.2 General Chat

  • The general-chat experiments use Llama3-8B-Instruct with the Ultrafeedback dataset and add a length penalty alongside the KL penalty.The online penalty is defined using the generation length term α|y|.Source: §D.2.2 · p. 23
  • DPO takes 3 hours per epoch and HyPO 18 hours per epoch for general chat on eight Nvidia A100 80GB GPUs.These timings describe the reported training setup for one epoch.Source: §D.2.2 · p. 23
Loading 2406.01462v2…