PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, Aviral Kumar

arXiv:2404.14367v3cs.LG

TL;DR

Preference fine-tuning has unclear practical requirements because RL, contrastive, and supervised methods can behave differently under limited coverage and computation. The paper analyzes these methods across bandit and LLM settings, finding that on-policy sampling and negative gradients generally outperform likelihood-only objectives, with mode-seeking behavior explaining the difference. It translates these findings into guidance about method choice and preference-data coverage.

  • Problem

    Existing results leave unclear whether RL, contrastive, or supervised objectives are sufficient and whether preference data should be collected on-policy.

  • Method

    The paper studies fine-tuning methods across didactic bandits, synthetic LLM tasks, and full-scale LLM problems, varying geometric alignment, data coverage, and optimization choices.

  • Results

    On-policy sampling and negative gradients generally outperform offline supervised objectives, especially when high-reward responses lie in less-likely regions of the reference policy.

  • Takeaways & Limitations

    Mode-seeking objectives unify on-policy sampling and negative gradients, while practitioners should consider reward-policy geometry, data coverage, and computational tradeoffs.

  • Takeaways & Limitations

    The study does not rigorously derive a statistical framework, simplifies coverage relative to the reference policy, and does not examine reward-model quality.

Abstract

from arXiv · show

Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning. Different methods come with different implementation tradeoffs and performance differences, and existing empirical findings present different conclusions, for instance, some results show that online RL is quite important to attain good fine-tuning results, while others find (offline) contrastive or even purely supervised methods sufficient. This raises a natural question: what kind of approaches are important for fine-tuning with preference data and why? In this paper, we answer this question by performing a rigorous analysis of a number of fine-tuning techniques on didactic and full-scale LLM problems. Our main finding is that, in general, approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i.e., employ a "negative gradient") outperform offline and maximum likelihood objectives. We conceptualize our insights and unify methods that use on-policy sampling or negative gradient under a notion of mode-seeking objectives for categorical distributions. Mode-seeking objectives are able to alter probability mass on specific bins of a categorical distribution at a fast rate compared to maximum likelihood, allowing them to relocate masses across bins more effectively. Our analysis prescribes actionable insights for preference fine-tuning of LLMs and informs how data should be collected for maximal improvement.

1. Introduction

The paper studies why preference fine-tuning methods differ in practice, focusing on on-policy sampling, negative gradients, and their relationship to reward geometry and data coverage. Across didactic and LLM settings, it argues that mode-seeking approaches can relocate probability mass toward high-reward responses more effectively than likelihood-based objectives.

  • Motivation: Preference fine-tuning methods differ in performance because practical training lacks the full data coverage and infinite computation required for equivalent optimal policies.This motivates comparing RL, contrastive, and supervised objectives together with on-policy versus offline data collection.
  • Approach: The study analyzes fine-tuning methods across didactic bandits, synthetic LLM problems, and full-scale LLM problems under varying coverage and geometric conditions.The framework uses a surrogate expected-reward objective regularized by KL divergence from a reference policy.
  • Main findings: On-policy sampling and negative gradients generally outperform offline supervised objectives, especially when high-reward responses lie in less-likely regions of the reference policy.The paper reports complementary benefits from combining on-policy sampling with explicit likelihood reduction on selected responses.
  • Mechanism: On-policy RL and certain contrastive objectives exhibit mode-seeking behavior, accumulating probability mass faster on a subset of high-reward responses than mode-covering supervised objectives.The paper formalizes this contrast by comparing reverse KL with forward KL for categorical distributions.
  • Practical implications: Practitioners should select methods according to reward-policy geometry and preference-data composition, while balancing on-policy sampling against gradient updates and computational tradeoffs.The paper also links data composition to collection strategies for preference fine-tuning.

2. Related Work

Related work studies preference fine-tuning through parameter effects, generalization, policy optimization, and comparisons between contrastive and supervised methods. The paper positions its contribution as an analysis of when different algorithms work under coverage and geometric conditions.

  • Prior analysis: Prior analyses examine how preference fine-tuning affects network parameters on selected tasks, whereas this work studies conditions under which different algorithms perform well.The authors describe their analysis as complementary and applicable to downstream tasks more broadly.
  • Policy optimization: Concurrent work suggests REINFORCE may suffice without PPO, while this paper reports that PPO is more robust to sample reuse than REINFORCE.The authors characterize these conclusions as mostly complementary.
  • Contrastive versus supervised learning: Other work compares contrastive and supervised fine-tuning on LLM-generated data but does not study preference-data coverage or geometric conditions.These omitted factors are central variables in the present analysis.

3. Characterizing And Unifying Preference Fine-Tuning Methods

The paper organizes preference fine-tuning methods by on-policy sampling, sample reuse, and negative gradients, then relates these choices to a KL-constrained reward-optimization objective. On-policy RL and contrastive methods differ from likelihood-only methods through sampling and explicit likelihood reduction.

  • Preliminaries and notation: Preference fine-tuning seeks a policy maximizing expected reward while penalizing deviation from a reference policy with a KL regularizer.The regularization weight controls the policy’s deviation from the reference under reverse KL.
  • Method taxonomy: The paper characterizes methods along three axes: on-policy sampling, on-policy sample reuse, and negative gradients.These axes distinguish how data are collected, reused, and weighted during optimization.
  • On-policy RL: On-policy RL samples responses from the current policy, scores them with a reward model, and updates policy parameters using a policy-gradient objective.REINFORCE performs one update per sampled response, while PPO can reuse responses across multiple updates.
  • On-policy RL: Normalized reward estimates reduce policy-gradient variance, while negative normalized rewards push down likelihood for corresponding responses.The latter gives on-policy RL a negative-gradient component absent from maximum-likelihood training.
  • On-policy supervised methods: On-policy supervised methods repeatedly maximize likelihood on reward-weighted or reward-filtered responses, using sample reuse but no negative-gradient effect.RWR, ReST, and SuperHF update on batched on-policy data before drawing new samples.
  • Offline methods: Fully offline contrastive methods such as DPO and IPO use preference pairs without on-policy sampling, whereas Pref-FT trains by supervised learning on preferred responses.Offline methods generally do not use a reward model in the same way as model-based on-policy approaches.

4. Research Questions and Analysis Setup

The analysis asks when on-policy sampling and negative gradients help, and whether their benefits are complementary. It evaluates these questions across controlled bandits, synthetic LLM tasks, and full-scale preference datasets while varying reward-policy geometry, data coverage, and sample reuse.

  • Research questions: The study asks when on-policy sampling improves offline fine-tuning, whether sample reuse helps, when negative gradients outperform maximum likelihood, and whether the two benefits are complementary.These questions directly define the experimental comparisons.
  • Analysis factors: Performance is analyzed through reward-policy geometric alignment and preference-data coverage relative to the reference policy.The first factor concerns where high reward lies relative to reference density; the second concerns the density of preference responses.
  • Tasks and datasets: The didactic N-dimensional contextual bandit uses 100-token vocabularies, 10-token responses, and known rewards to isolate geometric effects without reward-model estimation error.The setup compares reward functions whose optima occupy low- versus high-density regions of the reference policy.
  • Tasks and datasets: The bandit hypothesis is that on-policy sampling is crucial when the reward optimum lies in a low-likelihood reference region, whereas offline or maximum-likelihood methods may suffice when it aligns with the reference mode.These cases are represented by reward functions R1 and R2.
  • Tasks and datasets: Synthetic LLM tasks use hand-crafted rewards and reward models so training can be evaluated against ground-truth objectives before scaling to real preference datasets.The full-scale experiments draw on AlpacaFarm and related real preference data, including settings with distinct response-length distributions.
  • Experimental control: The unified algorithm varies on-policy sample quantity and gradient steps per sampled dataset to distinguish stale-data updates from explicit sample reuse.Larger datasets and more inner steps induce different degrees of off-policy updating.

5. Empirical Analysis Results

Across bandit, synthetic, and full-scale LLM experiments, more on-policy sampling and negative-gradient objectives generally improve preference fine-tuning, with benefits depending on reward–reference geometry and data coverage. Combining on-policy data with negative-gradient losses provides complementary gains, while sample reuse can trade exploration for computational efficiency.

  • 5.1.1. Takeaway 1: On-Policy Sampling in the Reward Model Improves Performance: More frequent sampling from recent policies improves best-of-N performance and convergence under a fixed data budget, especially when reward peaks are poorly aligned with the reference policy.Larger batches create more off-policy updates and greater divergence from the data-collection policy; degradation is milder for R2 when reward and reference-policy peaks overlap.
  • 5.1.1. Takeaway 1: On-Policy Sampling in the Reward Model Improves Performance: In synthetic Min Length and Skew Length problems, smaller sampling batches generally produce higher reward, faster proxy-reward optimization, and better PPO performance.The same direction appears across PPO, REINFORCE, and RWR; in Min Length, it persists despite inaccuracies in the proxy reward model.
  • 5.1.1. Takeaway 1: On-Policy Sampling in the Reward Model Improves Performance: When reward peaks lie in highly likely reference-policy regions, varying on-policyness has little effect and more off-policy configurations can perform similarly.This pattern appears in the synthetic Mode Length setting and matches the bandit result for R2.
  • 5.1.1. Takeaway 1: On-Policy Sampling in the Reward Model Improves Performance: On-policy sampling transfers to AlpacaFarm, where using only on-policy samples yields higher gold reward for both on-policy RWR and REINFORCE.The real-preference result agrees with the didactic and synthetic experiments.
  • 5.1.2. Takeaway 2: On-Policy Sample Reuse Can Enable Leveraging Off-Policy Data: Sample reuse can reduce dependence on on-policy sampling, but it also reduces exploration of the response space.This establishes a computational tradeoff rather than a uniformly superior sampling strategy.
  • 5.2.1. Takeaway 1: Negative Gradient Enables Faster Convergence Amongst Offline Methods: Negative-gradient objectives outperform offline maximum-likelihood methods when reward peaks occur in less likely reference-policy regions, while their advantage is smaller when those regions overlap.Best-of-N plus a negative gradient outperforms Best-of-N and closes the gap to IPO; DPO similarly outperforms Pref-FT in Min Length but performs similarly in Mode Length.
  • 5.2.2. Takeaway 2: Mechanisms Explaining the Behavior of the Negative Gradient: Negative gradients increase the preferred–dispreferred likelihood margin, but recovered probability mass may move to other responses rather than the preferred response.The outcome depends on response similarity, model capacity, reference-policy choice, dataset size, and composition; observed preferred-response likelihood trends differ across AlpacaFarm and UltraFeedback.
  • 5.3. Question 3: On-Policy Sampling and Negative Gradients are Complementary: On-policy sampling and negative-gradient losses are complementary: on-policy data improves response-space coverage, while the negative-gradient loss supplies a stronger learning signal.Using an effective negative-gradient loss on on-policy data can improve over on-policy RL or supervised learning and may offer computational benefits.

6. Conceptual Unification and Theoretical Analysis

The paper unifies on-policy sampling and negative-gradient contrastive methods as mode-seeking objectives, contrasting them with mode-covering weighted maximum-likelihood methods. Theoretical analyses explain how these objectives redistribute categorical probability mass differently during training.

  • On-policy RL and on-policy weighted-likelihood methods optimize regularized reverse KL and are therefore mode-seeking.
  • Offline contrastive objectives use negative gradients to allocate probability mass disproportionately toward the current policy’s highest-probability responses.
  • Offline weighted maximum-likelihood methods such as RWR, ReST, and BoN optimize forward KL and are mode-covering.
  • Reverse KL modifies probability mass more aggressively than forward KL and concentrates updates on selected categories during intermediate training.The analysis considers categorical distributions parameterized by independent logits and compares gradient-step dynamics.
  • Reverse KL can quickly redistribute probability mass toward only a subset of target categories even when the model can fully represent the target distribution.

7. Discussion, Conclusion, and Limitations

The study concludes that on-policy sampling and negative gradients are important components of preference fine-tuning, with their value depending on reward geometry and data coverage. It also identifies theoretical, coverage, and reward-model limitations.

  • On-policy sampling is especially important when high-reward responses lie in less-likely regions of the reference-policy initialization.
  • Preference datasets need broader coverage than the reference policy for on-policy sampling to work effectively in the studied setting.
  • Negative gradients can accelerate convergence and complement on-policy sampling.
  • The study does not derive rigorous statistical guarantees, and the statistical interpretation of negative gradients remains incompletely developed.
  • The coverage analysis simplifies the data context by omitting pre-training-distribution coverage and does not examine reward-model quality or parameterization.

A. Connections to Existing Fine-Tuning Results

The framework is presented as consistent with several existing LLM fine-tuning findings, including gaps between supervised objectives and RL or contrastive methods. Related work also supports the efficacy and complementarity of on-policy sampling and negative gradients.

  • On AlpacaFarm, the framework corroborates a performance gap between supervised objectives and RL or contrastive methods.
  • Several methods combine on-policy or rejection sampling with contrastive training, supporting the complementarity of these components.
  • Prior work finding that offline DPO underperforms on-policy PPO is consistent with this paper’s analysis of on-policy fine-tuning.

B. Computational vs Wall-Clock Time Tradeoff for Various Methods

The paper compares on-policy and offline contrastive methods in reward and wall-clock convergence time. Its results indicate that on-policy contrastive approaches can achieve better outcomes with less wall-clock time than offline contrastive and on-policy RL alternatives.

  • On-policy contrastive approaches generally achieve superior reward and wall-clock time compared with offline DPO/IPO and on-policy RL.
  • 0.4 hours versus 1.3 hours: on-policy DPO converges faster than offline DPO to the same solution in the Min Length scenario.
  • The appendix frames on-policy methods and offline contrastive methods as mode-seeking, unlike supervised maximum-likelihood methods, to explain their experimental behavior.

C.1.1. On-policy Methods Are Mode-Seeking

The analysis identifies on-policy RL and contrastive updates with negative gradients as mode-seeking, while offline maximum-likelihood methods are mode-covering. Reverse-KL optimization reallocates probability mass toward selected high-reward responses more rapidly than forward-KL optimization.

  • On-policy methods: On-policy RL methods optimize a regularized reverse-KL objective and therefore exhibit mode-seeking behavior.The reward-model partition function does not depend on the policy, making RL-loss minimization equivalent to reverse-KL optimization.
  • Contrastive approaches: Contrastive updates with negative gradients accelerate probability-mass increases on preferred responses relative to offline weighted maximum likelihood.The theoretical result holds for appropriate negative responses, model classes, and reference initializations.
  • Offline supervised methods: Offline supervised maximum-likelihood methods minimize a re-weighted forward-KL divergence and exhibit mode-covering behavior.They increase likelihood across high-reward responses rather than concentrating mass on a selected subset.
  • Forward and reverse KL: Forward and reverse KL differ in how gradient updates redistribute probability mass across categorical outcomes.The analysis studies their update differences through categorical distributions, logit gradients, and several cases.

E.1. Standardized Parameters (Consistent for all Methods)

The experimental setup standardizes algorithm, sampling, and model configurations while comparing offline and online data collection. Best-of-N, IPO, REINFORCE, PPO, and RWR are defined using fixed prompts and either reference-policy or current-policy samples.

  • Algorithms: The didactic bandit experiments compare Best-of-N, IPO, REINFORCE, PPO, and RWR.Best-of-N selects the highest-reward action among N sampled responses and trains on its negative log-likelihood.
  • Sampling: Offline datasets sample actions from the reference policy, whereas online datasets are regenerated from the current policy after T gradient steps.The previous online dataset is discarded after each collection cycle.
  • Negative gradient: The negative-gradient variant adds a term that minimizes the likelihood of dispreferred actions.The implementation thresholds this term because the loss can otherwise become unstable and approach negative infinity.
  • RL objectives: REINFORCE uses normalized reward estimates to reduce policy-gradient variance and support faster convergence.The paper notes that high-variance gradients can slow convergence and sometimes produce suboptimal solutions in deep RL.

G.3. Experiment Details

The synthetic experiments use standardized GPT-Nano and repeated data collection, then examine algorithm performance and policy recency across length-based setups. Mode Length shows similar algorithm performance, while Skew Length benefits from more on-policy sampling.

  • Experiment configuration: The experiments use N = 10, ten prompts, repeated policy-data collection, and varied T values for on-policy sampling.The setup performs 100 data-collection cycles and varies the number of gradient steps between collections.
  • Experiment configuration: The experiments use a 0.9M-parameter GPT-Nano transformer architecture.The implementation is taken from the minGPT repository.
  • Mode Length: All algorithms perform similarly in the Mode Length setup, and varying on-policyness generally does not degrade performance.Figure 20 measures distance to the reference-policy completion-length mode.
  • Skew Length: More on-policy sampling results in better performance in the Skew Length setup.Figure 22 reports average completion length and proxy reward against gradient steps.

H.3. Sample Reuse in Synthetic LLM Settings

Sample reuse can improve efficiency in the Skew Length setting, but excessive reuse can harm performance. PPO with importance clipping tolerates reuse better than Best-of-N.

  • Algorithm comparison: PPO with importance clipping is much better at sample reuse than Best-of-N.The comparison is reported for the Skew Length setting.
  • Sample reuse: T = 2 and T = 4 outperform T = 1, showing that limited sample reuse can improve sample efficiency.T denotes the number of inner iterations on the same data batch.
  • Sample reuse: T = 8 becomes unstable for PPO, showing that excessive sample reuse can hurt performance.Larger T makes training more off-policy.
Loading 2404.14367v3…