Source-linked AI summary
Generalized Preference Optimization: A Unified Approach to Offline Alignment
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, Bilal Piot
TL;DR
Offline preference optimization needs a unified account of how pairwise-data losses relate to RLHF and its regularization. GPO provides that account through convex loss functions, recovers existing algorithms, and finds similar trade-offs across variants while showing that their regularization strengths and optimal β values differ.
Problem
Offline preference optimization uses pairwise datasets efficiently, but the relationships among its losses and their induced regularization require a unified analysis.
Method
GPO parameterizes offline preference optimization with convex functions, deriving losses from supervised binary classification and encompassing DPO, IPO, and SLiC.
Results
Different GPO variants achieve similar regularization–performance trade-offs, while their distinct regularization strengths produce different optimal β values.
Takeaways & Limitations
Offline regularization generally differs from KL divergence, although the two losses are correlated near the reference policy and KL enforcement through offline optimization appears possible.
Takeaways & Limitations
The framework covers reward-maximization RLHF and contrastive offline losses, but not complex preference structures or supervised-learning-based losses.
Abstract
from arXiv · showhide
Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al 2023, we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners.
1. Introduction
Offline preference optimization replaces interactive RLHF components with direct learning from pairwise datasets. GPO unifies these methods through convex losses and analyzes the regularization they induce, finding similar performance–regularization trade-offs across variants but different hyper-parameter optima.
- Offline preference optimization directly aligns models from pairwise comparison datasets without sampling from the learned model or training reward models.
- GPO parameterizes offline preference optimization losses with convex functions, recovering DPO, IPO, and SLiC as special cases.
- The convex function’s tail behavior governs effective regularization between πθ and πref, which generally differs from the KL divergence intended by RLHF.
- The framework connects reward modeling to supervised binary classification, enabling systematic construction of existing and new offline losses.
- Different GPO variants show similar regularization–performance trade-offs, but their best β values can differ substantially because their regularization strengths differ.
2. A general family of offline preference optimization losses
GPO derives offline preference optimization objectives by translating supervised binary-classification losses into functions of policy log-ratio differences. This recovers DPO, IPO, and SLiC while enabling additional variants from other convex losses.
- Pairwise preference data labels responses as winner and loser, and the framework does not require a Bradley–Terry preference model.
- GPO replaces reward differences in regularized policy optimization with policy log-ratio differences measured against a reference policy.
- Reward learning is framed as supervised binary classification, allowing a general convex loss f to define an offline objective E[f(βρθ)].
- DPO uses logistic loss, IPO uses squared loss, and SLiC uses hinge loss within the generalized formulation.
- Exponential, truncated quadratic, and Savage losses provide examples of convex classification losses that can produce novel offline preference optimization algorithms.
- At a global minimizer, the offline objective corresponds to the optimal regularized policy for a reward function that globally minimizes the reward-model loss.
3. Reward modeling viewed as a binary classification problem
The paper recasts reward modeling as supervised binary classification, using convex surrogate losses on pairwise response preferences. This perspective yields the GPO formulation and clarifies how loss choice affects learned rewards and downstream regularization.
- Because the 0-1 classification loss is non-convex, the framework instead uses smooth convex functions f as surrogate losses.
- Reward modeling represents pairwise response preferences as binary classification, assigning label 1 when y1 is preferred to y2 and -1 otherwise.
- Pointwise reward models parameterize the classifier prediction as the difference between rewards for the two responses.
- Under separable preferences, logistic loss increases prediction magnitude toward infinity, whereas fast-decaying or upward-tail losses such as hinge and squared loss alleviate this behavior.
- The convex-loss analysis assumes f′(0) < 0, which places the minimizer at a positive reward difference and relates to recovering preference signs under Bayes consistency.
4. Understanding regularization in offline preference optimization
Offline preference losses implicitly regularize the policy toward the reference policy through the convex loss used by GPO. The resulting offline regularization can differ from KL regularization, especially when optimization moves far from the reference policy, although variants show similar empirical trade-offs.
- The tail behavior of the convex function f determines the effective regularization strength between πθ and πref, while differing generally from KL divergence.
- Near ρθ = 0, Taylor expansion makes GPO losses resemble squared-loss optimization, combining preference improvement with regularization toward πref.
- When the approximation is valid, GPO corresponds to IPO with effective regularizer |f′′(0)/f′(0)| · β.
- Fast-decaying or upward-tail losses constrain deviations more strongly, whereas slower-decaying losses can keep increasing winning-action probabilities and approach deterministic policies in pathological cases.
- DPO requires a higher β than IPO or SLiC to enforce the same regularization level because the convex loss determines natural regularization strength.
- The KL gradient uses on-policy samples from πθ, whereas offline squared regularization uses samples from μ, creating a structural difference between the objectives.
- The squared loss can have local minima distinct from KL minima, so local descent may fail to decrease KL even though both losses are globally minimized at πθ = πref.
- 10^-0.5 → 10^1.5, KL divergence varies more than μ-weighted squared loss, which ranges from 10^-1.5 → 10^0.5 in the correlated regime.
5. Empirical study of GPO variants
Experiments compare GPO variants through performance–KL trade-offs and show broadly similar curves, while the convex loss changes the regularization strength and therefore the best β. In summarization, choosing β appears more important than choosing among variants.
- Performance–KL trade-offs: GPO variants exhibit similar overall performance–KL trade-off curves, with comparable peak performance reached at similar KL divergence levels.This suggests that suitable choices of β and training duration can target a specified performance level.
- Performance–KL trade-offs: For fixed β, training traces can reach specified KL-divergence and win-rate levels as optimization progresses across training steps.The data points for different β values appear to connect smoothly at their boundaries.
- Impact of β: Different GPO variants peak at different β values: squared and truncated squared loss near β=1, versus roughly β∼10 for most other variants.The reported variation in peak win rate may not be statistically significant.
- Impact of β: At large β, squared and truncated squared losses enforce stronger regularization and incur smaller KL divergence than the other variants.At small β, the variants show little distinction in KL divergence.
- Summarization experiments: In the summarization experiments, best performance generally occurs for β∈[0.1, 1], and performance drops more when β becomes large.Pairwise comparisons show variants are generally on par, making the choice of β appear more critical.
6. Discussions and conclusion
The paper presents GPO as a convex-function framework that unifies offline preference optimization and clarifies its regularization behavior. Experiments find similar performance–regularization trade-offs across variants, but the framework remains limited in scope.
- Contributions: GPO parameterizes a continuous spectrum of offline preference losses through convex functions, with DPO, IPO, and SLiC as special cases.The formulation is derived using supervised binary-classification ideas.
- Regularization: Offline regularization and the KL regularization targeted by RLHF are generally different, although empirical correlations from the origin suggest KL enforcement through offline optimization may be possible.The paper characterizes this possibility as more challenging than direct KL regularization.
- Empirical conclusions: Different convex losses induce distinct regularization strengths, so their optimal β values differ even when their performance–regularization trade-offs are similar.The paper specifically notes that squared loss needs a smaller β than logistic loss.
- Limitations: The framework is based on reward maximization and therefore retains theoretical issues when the ground-truth preference structure is complex.It also covers only contrastive offline losses, not supervised-learning-based losses.
A.1. Bandit experiment
The bandit experiment illustrates how GPO loss tails affect the regularization imposed on the learned policy. Losses with insufficiently vanishing tails can yield greedy behavior despite regularization.
- Setup: The experiment uses a 3-action bandit to examine the regularization properties of different GPO variants.Policies are softmax-parameterized and trained from uniformly sampled offline pairs.
- Regularization behavior: With β=1, logistic, exponential, and Savage losses converge to the greedy action because their tails do not vanish fast enough.This behavior occurs even with regularization.
- Regularization behavior: The other three losses keep πθ(y1) closer to πref(y1) because they impose stronger regularization.
- Counterexample: The counterexample constructs related distributions as three-mode Gaussian mixtures, with numerical counterexamples obtainable using at most five simulations.The parameter choices are described as ad hoc, with other hyper-parameters expected to produce counterexamples as well.
A.3. Language modeling experiments
Language-modeling experiments evaluate GPO variants on summarization using KL divergence, win rate, and offline regularization traces. The variants show similar trade-off patterns and peak performance at the same KL divergence despite differing β values.
- Experimental setup: The summarization experiments use an open-source human-feedback dataset with 700M-parameter T5X models and evaluate responses with prompted PaLM-2 judgments over 2,000 examples.Training uses a constant learning rate of 10^-5, batch size 32, Adafactor, and 2×10^5 steps.
- Regularization traces: For each fixed convex function and β, the experiments trace μ-weighted squared loss on the learner against KL divergence on the evaluator during training.Evaluation occurs every 2,000 steps, with denser evaluation during the first 2,000 steps.
- Trade-offs: Different GPO variants exhibit similar win-rate–KL trade-off patterns and similar dependence on β.Figure 9 groups data points by regularization coefficient for each algorithmic variant.
- Trade-offs: All algorithmic variants reach peak performance at the same KL divergence but require different β values.The difference is attributed to the distinct natural regularization strengths of the loss functions.
- Regularization traces: The tracing plot separates individual GPO variants to visualize KL divergence against μ-weighted squared loss.
A.5. Model-based side by side evaluation
The summarization experiments evaluate GPO variants using model-based side-by-side comparisons against the supervised fine-tuned reference and against one another. Performance is generally best for β∈[0.1, 1], while variants show mostly similar results.
- Evaluation setup: The study trains models on an open-source summarization dataset and uses a PALM-2 model for side-by-side evaluation.The evaluation setup follows prior studies of the KL-divergence versus win-rate trade-off.
- Results: Win rate against the supervised fine-tuned checkpoint is usually highest at β∈[0.1, 1], with similar performance across different convex functions.When β becomes too large, win rate declines more quickly across all methods.
- Results: At β=0.1, the GPO variants perform mostly similarly in side-by-side comparisons against one another.This checkpoint was selected because all variants showed reasonable performance against the supervised fine-tuned checkpoint.
B. Proof and derivations of theoretical results
The proof establishes that a global minimizer of the offline preference-optimization loss corresponds to the optimal regularized policy for a reward function minimizing the reward-model loss. The argument reparameterizes the reward through the policy log ratio and transfers global optimality between the objectives.
- Theoretical result: The theorem identifies the global minimizer of the offline preference-optimization loss with the optimal regularized policy for a globally loss-minimizing reward function.This connects offline preference optimization to the regularized policy objective.
- Proof: The proof reparameterizes the reward as a scaled log ratio between the policy and reference policy, plus a normalizing constant.The reparameterization is expressed as rϕ(y) = β log πθ(y) / πref(y) + z.
- Proof: Under this reparameterization, a globally minimizing reward function induces a policy that globally minimizes the offline preference-optimization loss.The conclusion follows by relating the reward-model objective to the offline loss.
B.1. Derivation of the gradient of KL divergence and 𝜇-weighted squared loss
The derivations relate the KL-divergence gradient to a μ-weighted squared loss and characterize how convex GPO losses locally reduce to the squared-loss, or IPO, form. Bayes consistency explains why the resulting preference classifier recovers the preference sign under convexity conditions.
- Gradient derivation: The gradient derivation uses score-function cancellation, exchangeability, independence, and identical-distribution arguments to establish the relation between KL divergence and μ-weighted squared loss.The related result connects the KL gradient with the gradient of the variance of the log ratio.
- Taylor expansion: A second-order Taylor expansion of a convex GPO loss recovers the squared loss with β′ = f′′(0)β / |f′(0)|.The squared loss is effectively the IPO loss, up to constants under the expansion.
- Bayes consistency: The Bayes-optimal classifier predicts the preference sign according to sign(2p(ℓ=1|z)−1).This classifier minimizes the 0-1 classification error.
- Bayes consistency: For convex f with f′(0)<0, a global minimizer of the binary classification loss is Bayes consistent.The condition ensures that the learned classifier recovers the correct sign of the preference probability.
C.1. Discussion of pairwise preference model
The pairwise preference model is analyzed as a general bivariate predictor, with Bayes consistency ensuring correct preference signs under the considered convex losses. However, different convex losses can produce substantially different prediction magnitudes even when their signs agree.
- Model formulation: The pairwise preference predictor is parameterized as a general bivariate function of two responses rather than as a difference of two univariate rewards.The paper leaves precise transfer of these results to pointwise reward models for future work.
- Bayes consistency: Bayes consistency implies that minimizing the considered convex losses yields predictions whose signs match the preference probability sign.The right-hand side of Eqn (9) is the Bayes-optimal classifier for the classification loss.
- Loss dependence: Different convex losses can produce drastically different prediction values even when they produce the same preference sign.The paper illustrates this distinction through logistic-loss and hinge-loss case studies.