Source-linked AI summary
Policy Optimization in RLHF: The Impact of Out-of-preference Data
Ziniu Li, Tian Xu, Yang Yu
TL;DR
The paper examines how data sources affect generalization when optimizing policies from human preferences. It analyzes reward-model-free and reward-model-based approaches and tests them in controlled experiments. The results show that out-of-preference data improves generalization, even with a shared good feature representation.
Problem
The paper asks which policy-optimization data sources are more effective and how they affect generalization when reward models are learned from human preferences.
Source: §1 · p. 2Method
The paper analyzes policy-optimization errors in a contextual-bandit framework and conducts experiments with linear and neural function approximation.
Source: §1 · p. 2Results
Out-of-preference data improves generalization even when the policy and reward model share a good feature representation.
Source: §1 · p. 2Takeaways & Limitations
The results underscore the importance of optimizing policies on out-of-preference data and using a reward model to provide supervision signals.
Source: §5 · p. 10The theoretical analysis uses supremum-norm errors, omits a separate RMB-PO+ analysis, and identifies tighter finite-sample guarantees as future work.
Source: §2.1 · p. 5
Abstract
from arXiv · showhide
Aligning intelligent agents with human preferences and values is important. This paper examines two popular alignment methods: Direct Preference Optimization (DPO) and Reward-Model-Based Policy Optimization (RMB-PO). A variant of RMB-PO, referred to as RMB-PO+ is also considered. These methods, either explicitly or implicitly, learn a reward model from preference data and differ in the data used for policy optimization to unlock the generalization ability of the reward model. In particular, compared with DPO, RMB-PO additionally uses policy-generated data, and RMB-PO+ further leverages new, preference-free data. We examine the impact of such out-of-preference data. Our study, conducted through controlled and synthetic experiments, demonstrates that DPO performs poorly, whereas RMB-PO+ performs the best. In particular, even when providing the policy model with a good feature representation, we find that policy optimization with adequate out-of-preference data significantly improves performance by harnessing the reward model's generalization capabilities.
1 Introduction
The paper asks whether reward-model-free or reward-model-based policy optimization is more effective, and how their data sources affect generalization. It analyzes these methods and reports that out-of-preference data improves generalization, including when policy and reward models share a good feature representation.
1 Introduction
- The paper compares reward-model-free methods that optimize directly from preference annotations with reward-model-based methods that train a reward model before policy optimization.DPO is an example of the first class; RLHF with PPO is a popular example of the second.Source: §1 · p. 2
- The paper asks whether preference-dataset pairs or newly generated prompt-response pairs are more effective for policy optimization and generalization.Reward-model-free methods use previously collected pairs, while reward-model-based methods generate new responses to prompts.Source: §1 · p. 2
- The analysis frames alignment as a contextual bandit and treats policy optimization methods as Monte Carlo approximations of expected-reward maximization.It finds that including out-of-preference data improves the approximation's accuracy.Source: §1 · p. 2
- Out-of-preference data improves generalization even when policy and reward models share the same good feature representation.Other experiments also support this claim.Source: §1 · p. 2
2 Problem Formulation
The section formulates alignment as reward-maximizing policy optimization from pairwise preferences, then contrasts preference-data and policy-generated data for practical optimization. Its error analysis separates reward, action-distribution, and state-distribution estimation errors, and notes that the analysis is basic and lacks finite-sample guarantees.
2 Problem Formulation
- In contextual bandits, states and actions define the decision problem; for language models, they correspond to prompts and responses.The policy seeks to maximize reward over the state distribution.Source: §2 · pp. 2–3
- Because rewards are unknown and only pairwise preferences are observed, Bradley-Terry-based maximum-likelihood training encourages higher scores for preferred actions.The preference dataset contains states and paired actions, with one action marked as preferred.Source: §2 · p. 3
- The ideal policy maximizes recovered reward over the state distribution with a KL penalty intended to mitigate reward hacking and over-optimization.The state distribution is denoted by ρ, and the reference policy is πref.Source: §2 · pp. 2–3
- DPO uses preference-dataset state-action pairs to approximate the distributions and combines the corresponding procedures into one objective without explicitly training a reward model.This is the reward-model-free approach described in the section.Source: §2 · p. 3
- RMB-PO samples policy actions at preference-data states, while RMB-PO+ also uses preference-free states; the action expectation may be approximated by sampling multiple actions.The paper excludes this action-sampling approximation error, noting it can be mitigated with computational power.Source: §2 · pp. 3–4
2.1 Theoretical Analysis
- The analysis distinguishes reward-evaluation error, finite-sample action-distribution error, and finite-sample state-distribution error.Finite preference data contributes the first error; RMB-PO aims to mitigate the second, and RMB-PO+ further reduces the third without increasing preference-data sample complexity.Source: §2.1 · pp. 4–5
- The analysis is basic because it uses supremum-norm errors, omits a separate RMB-PO+ analysis, and leaves tighter finite-sample guarantees for future work.The stated reason for omitting the RMB-PO+ analysis is that it is similar to the RMB-PO analysis.Source: §2.1 · p. 5
3 Experiments
Across linear and neural bandit experiments, out-of-preference data improved policy generalization. In the neural bandit, RMB-PO methods outperformed DPO, while additional preference-free data beyond a modest amount added little.
3 Experiments
- In the reported experiments, preference data size was n=20 and preference-free data size m=10n; reward-model training accuracy ranged from 60% to 80% across 10 experiments, with results averaged over 10 random seeds.Source: §3 · p. 6, §3.1 · p. 7
3.1 Linear Bandit
- With shared features, RMB-PO benefits from out-of-preference data; under feature mismatch, RMB-PO+ beats RMB-PO, and both RMB-PO variants outperform DPO off-support.The learned action distributions align with the optimal policy in preference-covered regions for DPO, while RMB-PO methods perform better in uncovered, out-of-distribution regions.Source: Figure 2 · p. 6, Figure 3 · p. 6, §3.1 · p. 7
3.2 Neural Bandit
- With preference data fixed at n=50, RMB-PO and RMB-PO+ significantly outperform DPO; preference-free data twice as large already improves performance over RMB-PO, while further scaling helps little.Source: §3.2 · p. 8
4 Discussion
The discussion connects reward-model-guided policy optimization to imitation learning and reinforcement learning, highlighting the role of optimization on data beyond expert or preference datasets. It also notes practical advantages and challenges of reward-model-based methods.
4 Discussion
- Behavioral cloning maximizes expert-data likelihood, a mechanism the paper describes as similar to DPO’s treatment of positively and negatively preferred actions.Behavioral cloning trains a policy by maximizing the likelihood of expert data; the cited discussion compares its mechanism with DPO [Rafailov et al., 2023].Source: §4 · p. 9
- Adversarial imitation learning usually performs better than behavioral cloning, and theoretical work finds it can address distribution shift through optimization on out-of-expert data.The paper relates reward recovery followed by policy optimization in imitation learning to the RLHF framework.Source: §4 · p. 9
- Transition-model-based RL findings align with the paper’s observation that additional policy optimization on transition-model-generated data is helpful.The discussion notes empirical successes in sample complexity [Luo et al., 2019, Janner et al., 2019] and points to work on data augmentation [Hafner et al., 2020, Schrittwieser et al., 2020, Yu et al., 2020, Luo et al., 2023].Source: §4 · p. 9
- RMB-PO avoids extra preference annotation, but its large action space creates an optimization challenge that computationally efficient methods can address.The discussion also notes that optimizing with prompts-only data can improve performance, a setting DPO cannot achieve [Li et al., 2023].Source: §4 · p. 10
5 Conclusion
The paper analyzes errors in preference-based policy optimization and validates its claims with experiments. Its results underscore the importance of out-of-preference policy optimization and reward-model supervision.
5 Conclusion
- The paper analyzes errors in policy optimization methods for alignment from preferences.It also conducts experiments to validate its claims.Source: §5 · p. 10
- The results underscore the importance of optimizing policies on out-of-preference data.Source: §5 · p. 10
- The results also underscore the power of using a reward model to provide supervision signals.Source: §5 · p. 10