PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Alignment of Large Language Models through Online Preference Optimisation

Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, Bilal Piot

arXiv:2403.08635v1cs.LGcs.AIstat.ML

TL;DR

Aligning language-model outputs with human preferences motivates methods with different objectives and data regimes. This paper connects online IPO to self-play in a regularised preference game and introduces IPO-MD, which uses mixture-policy data. On summarisation, online IPO and IPO-MD are statistically indistinguishable and consistently outperform the other evaluated algorithms.

  • Problem

    Preference-optimisation methods differ in how they use data and combine algorithmic properties, and the paper identifies online contrastive optimisation as a potentially advantageous combination not yet represented.

    Source: §3 · p. 4

  • Method

    The paper introduces online IPO and IPO-MD, using online sampling and a regularised mixture data distribution, and analyses their connection to self-play in a preference game.

    Source: §1 · p. 2, §4.2 · p. 5, §5.1 · p. 6

  • Results

    On summarisation, IPO and IPO-MD are statistically indistinguishable and consistently outperform the other evaluated algorithms.

    Source: §6.1 · p. 8

  • Takeaways & Limitations

    Online IPO's expected update is equivalent to self-play in the regularised game, while IPO-MD and online IPO are promising on the evaluated summarisation task.

    Source: §4.2 · p. 5, §7 · p. 9

    The reported empirical results are limited to summarisation, and further experiments are needed to validate them on a general conversational agent.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task.

1. Introduction

The section contrasts reward-model-based alignment and DPO with IPO and Nash-MD-PG, which directly target preference probabilities but differ in data regime and objective. The paper bridges these approaches, introduces online IPO and IPO-MD, and compares them experimentally.

1. Introduction

  • Learning from feedback is used to align agents with human preferences, and RLHF has become a common paradigm for fine-tuning large language models.The passage cites Knox & Stone (2008), Griffith et al. (2013), Christiano et al. (2017), Warnell et al. (2018), Glaese et al. (2022), and OpenAI (2022).Source: §1 · p. 1
  • Traditional RLHF learns a Bradley-Terry reward model before reinforcement learning, while DPO bypasses reward learning but retains the Bradley-Terry model.The population-loss minimiser of DPO is mathematically equivalent to that of the traditional approach.Source: §1 · p. 1
  • IPO directly optimises preference probabilities against a fixed data distribution, whereas Nash-MD-PG uses online policy updates to seek a Nash equilibrium.Despite these differences in data and objective, the methods motivate the paper's theoretical bridge.Source: §1 · p. 1
  • The paper identifies variation in offline versus online data, contrastivity, and equilibrium structure as key differences between IPO and Nash-MD-PG.It uses these differences to motivate online IPO and IPO-MD, and establishes an online IPO connection to self-play.Source: §1 · pp. 1–2
  • The paper introduces IPO-MD, which uses Nash-MD-PG's lagged data distribution to interpolate between offline and online IPO variants.The introduction also previews experiments comparing the proposed methods with baselines.Source: §1 · p. 2

2. Background

The background frames preference optimisation as learning a policy that selects preferred actions, then reviews reward-model-based RLHF and direct preference-optimisation methods. These methods differ in their objectives, data assumptions, and use of regularisation.

2.1. Preference optimisation in bandits

  • The paper formulates preference optimisation in a finite-action, non-contextual bandit setting and notes that its ideas extend to contextual or sequential settings.A preference function gives pairwise preference probabilities and is assumed to satisfy p(y′ ≻ y) = 1 − p(y ≻ y′).Source: §2.1 · p. 2

2.2. RLHF with a Bradley-Terry reward model

  • RLHF fits a Bradley-Terry reward model from observed preferences, then maximises expected learned reward subject to a KL constraint against the reference policy.The temperature parameter controls regularisation toward the reference policy.Source: §2.2 · p. 2

2.3. Direct preference optimisation (DPO)

  • DPO reparameterises the optimal reward in terms of the optimal policy, bypassing reward learning while yielding the same optimal policy as regularised RLHF.Both approaches rely on the Bradley-Terry model.Source: §2.3 · pp. 2–3

2.4. Sequence Likelihood Calibration (SLiC)

  • SLiC is presented as an offline loss on preference data and is interpreted as a hinge-loss variation of DPO.The cited work first proposed SLiC as an alternative to RLHF and later normalised policy probabilities by reference-policy probabilities.Source: §2.4 · p. 3

2.5. Identity policy optimisation (IPO)

  • IPO directly optimises preference probabilities against a fixed policy, and its quadratic loss regularises policy log-probability ratios toward those of the reference policy.The passage contrasts this regularisation property with the minimiser of the DPO empirical loss.Source: §2.5 · p. 3

2.6. Nash-MD-PG

  • Nash-MD-PG treats preference optimisation as a two-player constant-sum game and seeks a Nash equilibrium with sampling between self-play and the data-generating policy.Its parameter β controls that tradeoff, with β = 0 for self-play and β = 1 for sampling from µ.Source: §2.6 · p. 3

3. Comparative discussion of preference optimisation algorithms

The comparison identifies distinct strengths across preference-optimisation methods and combinations not yet represented. It motivates algorithms that combine properties of existing approaches.

3. Comparative discussion of preference optimisation algorithms

  • IPO uses contrastive updates through both preferred and non-preferred samples, while Nash-MD-PG directly updates only the sampled action's policy probability.Contrastive algorithms may be more data-efficient because they use both samples in each update.Source: §3 · p. 4
  • IPO uses fixed offline data, whereas Nash-MD-PG samples from the current policy and a geometric mixture; online data can improve learned-policy performance when available.Offline data can limit preference evaluation outside the data distribution, while regularisation toward a reference policy is one mitigation.Source: §3 · p. 4
  • IPO has a supervised objective with a closed-form optimiser, while Nash-MD-PG's changing data distribution makes its stationary points self-referential.Nash-MD-PG permits mixture sampling between the policy and the data-generating distribution.Source: §3 · p. 4
  • The discussion identifies online contrastive optimisation as one potentially advantageous combination missing from existing methods.Table 1 overviews methods by their properties, including how the paper's methods combine existing strengths.Source: §3 · p. 4

4. Online IPO

Online IPO replaces IPO's fixed data distribution with data generated by the current policy. Its expected gradient matches self-play in a regularised preference game, while online DPO generally has a different objective in the two-action case.

4.1. Algorithm

  • Online IPO replaces the fixed data distribution in the IPO population loss with data generated by the current policy.The sample-based loss otherwise coincides with offline IPO, and the data-generation process is stop-gradient.Source: §4.1 · p. 4

4.2. Analysis

  • The minimiser of the online IPO objective is the Nash equilibrium of the regularised game, and its expected gradient matches the game's self-play update.In expectation, this corresponds to Nash-MD-PG with β = 0; online IPO's contrastive updates may reduce gradient-estimate variance.Source: §4.2 · p. 5

4.3. Online DPO

  • For two actions, online DPO and online IPO are different objectives except in the uniform-preference case.Under the Bradley-Terry assumption, the RLHF solution is a stationary point of online DPO, coinciding with offline DPO's stationary point.Source: §4.3 · p. 5

5. IPO-MD

IPO-MD extends Online IPO by sampling from a geometric mixture of the online and reference policies, and its fixed points match Nash-MD-PG’s. Its gradients align with Nash-MD-PG at β=0 but differ for β>0.

5.1. Algorithm

  • IPO-MD(β) uses a geometric mixture of online and reference policies, ranging from Online IPO self-play at β=0 to a fixed-reference variant at β=1.Source: §5.1 · pp. 5–6

5.2. Analysis

  • Nash-MD-PG is on-policy, using actions sampled under the current policy, whereas IPO-MD is off-policy, using actions sampled under the mixture policy.Source: §5.2 · p. 6
  • IPO-MD(β) and Nash-MD-PG(β) share fixed points, which correspond to equilibria of the regularized game with a modified regularization parameter.The analysis also identifies the Nash equilibrium condition for that game.Source: §5.2 · p. 6
  • At β=0, IPO-MD and Nash-MD-PG gradients align; for β>0, their gradients differ even though their fixed points remain the same.Source: §5.2 · p. 6

6. Experiments

The experiments evaluate online preference-optimisation methods on summarisation and examine their performance against RL and across training and regularisation settings. IPO and IPO-MD are the strongest methods in the reported comparison, while the findings remain limited to summarisation.

6. Experiments

  • Experiments compared online RL, IPO, DPO, SLiC, Nash-MD-PG, and IPO-MD on summarisation, with PaLM2 judging generated responses side by side.The preference and reward models were trained on pairwise TL;DR summary preferences, and policy prompts came from XSum.Source: §6 · pp. 6–7

6.1. Main results

  • Considering standard deviations, IPO and IPO-MD are statistically indistinguishable and consistently beat the other algorithms in side-by-side summarisation comparisons.IPO has the highest mean performance; Table 2 reports average preferences and standard deviations across seed comparisons.Source: §6.1 · p. 8, Table 2 · p. 8
  • Results are limited to summarisation; the authors call for more experiments to validate them on a general conversational agent.The paper describes summarisation as a test bed for human-alignment algorithms.Source: §6.1 · p. 8

6.2. Ablations and Additional Results on Summarisation

  • At small τ, IPO and DPO behave similarly; at larger τ, IPO's score decays faster, consistent with its stronger regularisation effect.Figure 1 sweeps τ against RL on summarisation.Source: §6.2 · p. 8, Figure 1 · p. 8
  • As regularisation strengthens, Online IPO requires more training time to reach its best performance against RL.Figure 2 plots performance against RL over training steps.Source: §6.2 · p. 8, Figure 2 · p. 8

7. Conclusion

The conclusion presents Online IPO and IPO-MD as algorithms combining aspects of IPO and Nash-MD-PG. It reports an expected-update equivalence between Online IPO and self-play, and promising summarisation results while noting the work's current scale and task limits.

7. Conclusion

  • Online IPO and IPO-MD combine IPO's loss with Nash-MD's online sampling and regularised data distribution.The paper presents both as new preference-optimisation algorithms.Source: §7 · p. 9
  • Theoretical analysis shows Online IPO's expected update is equivalent to self-play in a regularised two-player preference game, a property not shared by online-DPO.Source: §7 · p. 9
  • On summarisation, Online IPO and IPO-MD are reported as the most robust algorithms; the work is limited to a 770M model and one task.Future work is described as scaling to a full conversational agent with a model larger than 100 billion parameters.Source: §7 · p. 9

APPENDICES

The appendices situate preference optimisation among RLHF and direct methods, then report summarisation sweeps and selected hyperparameters. The sweep discussion finds online methods outperform offline methods in this setup and notes that non-optimal settings often still benefit from IPO-MD's mixture ratio.

A. Related Work

  • Related work discusses RLHF and direct preference optimisation, highlighting reward over-optimisation concerns and the practical distinction between online and offline data.Source: A. Related Work · p. 13

B.1. Regularisation Sweep for Online and Offline

  • Online versions significantly outperform offline versions in the reported sweeps, a setting that starts from a policy already fine-tuned on summarisation data.The authors explain that this initialization lets online methods sample good summaries from the first checkpoint.Source: §B.1 · p. 13, Figure 3 · p. 13, Figure 4 · p. 14

B.2. Mixing ratio curve

  • In most plotted cases, changing IPO-MD's mixture ratio β still helps improve win rate over RL, despite using learning rates and steps different from its optimal checkpoint.Figure 5 fixes τ = 1 and plots three learning-step settings.Source: §B.2 · p. 14, Figure 5 · p. 14

B.3. Best hyperparameters found

  • The reported best settings use β = 0.125 for IPO-MD and Nash-MD-PG; their τ values are 1.0 and 0.008, respectively.The table also reports each method's learning rate.Source: §B.3 · p. 14

C. Proofs

The proofs establish that Online IPO's expected gradient matches the self-play update direction. They also lay out update expressions for the Nash-MD-PG and IPO-MD algorithms.

C. Proofs

  • Proposition 4.2 states that Online IPO's expected gradient equals the self-play update direction in the specified preference game.Source: C. Proofs · p. 14
  • The proof derives Online IPO's update from the negative loss derivative, separates terms involving τ from those without it, and compares it with the self-play update.It concludes that the expected Online IPO update direction is exactly the self-play direction.Source: C. Proofs · pp. 14–16
  • Proposition 5.1 gives gradients for Nash-MD-PG and IPO-MD; the proof defines π′ as a geometric mixture of π and πref and defines ρπ using log-policy ratios.Source: C. Proofs · pp. 16–17

D. Comparison of the variance of contrastive versus non-contrastive gradient estimates

The contrastive and non-contrastive gradient estimates have the same expectation, but their variances can differ. Under a sufficient condition on the policy representation and preference model, the contrastive estimate has no greater variance.

D. Comparison of the variance of contrastive versus non-contrastive gradient estimates

  • Online-IPO uses the contrastive estimate, while self-play as implemented by Nash-MD-PG (β = 0) uses the non-contrastive estimate.The estimates have the same expectation, although their variances may differ.Source: D. Comparison of the variance of… · p. 18
  • When the stated condition on the policy representation and preference model holds, the contrastive estimate’s variance is no greater than the non-contrastive estimate’s.The passage identifies the condition as sufficient for this variance comparison.Source: D. Comparison of the variance of… · p. 18
  • The contrastive estimate has variance 2 (Var(X1) + Cov(X1, X2)), so negative covariance makes it lower than Var(X1).The passage describes this covariance-based reduction as the principle of antithetic variates.Source: D. Comparison of the variance of… · p. 18

E. Tabular example

The tabular example plots trajectories for online, offline, and MD variants of IPO across values of β. The supplementary study finds that online DPO and online IPO are generally distinct in two-action problems, while the RLHF solution is stationary for online DPO under a Bradley–Terry preference model.

E. Tabular example

  • Figure 6 plots trajectories for IPO variants across β values, with τ = 0.1 and a uniform reference policy.The plot is presented as intuition for IPO-MD; its caption identifies online, offline, and MD variants.Source: E. Tabular example · p. 19, Figure 6 · p. 19

F. Supplementary Theoretical Study of Online DPO

  • For two-action problems, online DPO and online IPO are different objectives except when preferences are uniform; the condition for equivalence is difficult to satisfy.The study reports no two-action problem satisfying the condition apart from the uniform-preference case.Source: F. Supplementary Theoretical Study of… · p. 20
  • Under the Bradley–Terry model assumption, the RLHF solution is a stationary point of online DPO.The passage also notes that offline DPO can have multiple solutions when the sampling distribution assigns zero probability to some actions.Source: F. Supplementary Theoretical Study of… · p. 20

F.1. Results and Proofs

The supplementary results characterize the offline DPO objective, its stationary points, and conditions connecting online DPO to a regularized-game equilibrium. They also identify non-uniqueness when the sampling distribution omits actions.

F.1. Results and Proofs

  • Offline DPO’s objective is the expected DPO loss over preference-labeled pairs sampled from the distribution μ.The formulation assumes a finite action space and symmetric pairwise preferences.Source: §F.1 · pp. 20–21
  • Remark F.4’s interior-simplex restriction is not essential to Equation (15), which holds for any exp ◦ f with real-valued f.The objective depends on ratios of policy probabilities and is unchanged by positive rescaling.Source: §F.1 · p. 21
  • The regularized-game Nash equilibrium is a stationary point of online DPO only when the stated condition in Lemma F.5 holds.The proof reduces this question to whether the equilibrium solves offline DPO with sampling distribution μ = π*; the supplied passages do not state the condition itself.Source: §F.1 · pp. 21–22
  • Theorem F.6 states that no 2-action regularised game given in Equation (9) has a Nash equilibrium satisfying Equation (19), except regularised games with p(y1 ≻ y2) = 1.Source: §F.1 · p. 22
  • Under a Bradley–Terry preference model, the RLHF solution is a critical point of offline IPO for any μ and a stationary point of online DPO.For an interior sampling distribution, the passage says this solution is the only offline DPO solution.Source: §F.1 · pp. 23–24
  • When μ assigns zero probability to some actions, offline DPO can have infinitely many solutions, including ones assigning arbitrarily small probabilities to sampled actions.This non-uniqueness follows from the objective’s insensitivity to actions outside the sampling support.Source: §F.1 · p. 24

Related papers

Loading 2403.08635v1…