PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Play Preference Optimization for Language Model Alignment

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, Quanquan Gu

arXiv:2405.00675v5cs.LGcs.AIcs.CLstat.ML

TL;DR

Existing RLHF approaches face challenges from non-transitive human preferences and from efficiently scaling self-play methods to large-scale language-model alignment. SPPO formulates alignment as a constant-sum game, uses iterative self-play updates to approximate a Nash equilibrium, and improves performance across multiple benchmarks without stronger-model supervision.

  • Problem

    RLHF methods based on parametric preference models can fail to capture non-transitive human preferences, while efficiently applying self-play algorithms to large-scale language-model alignment remains unclear.

  • Method

    SPPO formulates RLHF as a constant-sum two-player game and uses multiplicative-weight-based iterative self-play updates to provably approximate its Nash equilibrium.

  • Results

    SPPO improves DPO and IPO across AlpacaEval 2.0, MT-Bench, Arena-Hard, and the Open LLM Leaderboard without external supervision from GPT-4 or stronger language models.

  • Takeaways & Limitations

    SPPO provides an alignment approach that combines theoretical convergence guarantees with a simple optimization objective and strong multi-benchmark performance.

  • Takeaways & Limitations

    Performance declines after the first or second alignment iterations, limiting continued gains in general language-model capabilities.

Abstract

from arXiv · show

Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium. Additionally, we propose a new SPPO objective which is both strongly motivated by theory and is simple and effective in practice. In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench, Arena-Hard, and the Open LLM Leaderboard. Starting from a stronger base model Llama-3-8B-Instruct, we are able to achieve a length-controlled win rate of 38.77%. Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models. Codes are available at https://github.com/uclaml/SPPO.

1 Introduction

The paper motivates preference-probability-based alignment by the limits of score-based RLHF for inconsistent human preferences, then proposes SPPO as a theoretically grounded, scalable self-play method. SPPO models RLHF as a constant-sum game and reports consistent gains across alignment benchmarks without stronger-model supervision.

  • Motivation: Human preferences can be inconsistent, non-transitive, and even irrational, challenging RLHF methods built around a single numerical score.The paper cites preference loops and decision inconsistency as evidence that a monotonic, transitive hierarchy may not represent human judgments.
  • Method: SPPO formulates RLHF as a constant-sum two-player game whose goal is to identify a Nash equilibrium policy.Each player is an LLM producing responses and maximizing its probability of being preferred over its opponent.
  • Method: SPPO uses multiplicative-weights updates and self-play fine-tuning on policy-generated responses annotated by a preference model.The framework approximates each online-learning step by having the current policy play against the previous-round policy on synthetic data.
  • Theory: SPPO provides provable convergence to an approximate Nash equilibrium and a simple loss function for optimization.The algorithm starts from an exponential-weight update known to converge to the equilibrium of a constant-sum game.
  • Objective: SPPO’s objective avoids pairwise comparisons and is connected to policy-gradient theory, implicitly encouraging a token-level optimal value function.The objective is presented as a theoretically motivated alternative to symmetric pairwise losses such as DPO and IPO.
  • Results: SPPO improves Mistral-7B-Instruct-v0.2 and Llama-3-8B-Instruct by over 11% in length-controlled win rate against GPT-4-Turbo on AlpacaEval 2.0.The paper also reports strong performance across MT-Bench, Arena-Hard, and the Open LLM Leaderboard without external supervision from GPT-4 or stronger models.

2 Related Work

Related work spans reward-model RLHF, general-preference objectives, theoretical analyses, and self-play fine-tuning. The paper positions SPPO as a scalable self-play approach for general preferences with a distinct optimization objective.

  • Reward-model RLHF: Classical RLHF learns a reward model from human feedback and then applies reinforcement learning to maximize that reward.InstructGPT exemplifies this pipeline for instruction-following language models.
  • Reward-model RLHF: Parametric reward models such as Bradley-Terry assign scalar scores, while DPO and related methods still implicitly rely on score-based assumptions.These approaches differ in optimization form but retain an underlying numerical preference representation.
  • General preferences: General-preference methods model pairwise preference probabilities directly because human preferences may not be strictly transitive.IPO seeks to maximize a policy’s expected winning probability over a reference policy.
  • Theory: Theoretical RLHF work reduces general-preference alignment to two-player zero-sum Markov games and studies guarantees under different reward or preference settings.Related analyses include pessimistic reward learning, reward-based reductions, and reverse-KL-regularized contextual bandits.
  • Self-play: Self-play fine-tuning repeatedly generates new data from the latest policy and trains a successor policy, with or without human preference data.Prior frameworks include reward-annotated EM procedures and supervised self-play using synthesized preference pairs.
  • Positioning: SPPO differs from concurrent methods in its approximation of the log-partition factor and its preference-optimization formulation.The paper contrasts SPPO with SPO, DNO, and REBEL, while noting that tabular computation of normalization can be prohibitive.

3 Preliminaries

The paper frames preference learning for language models by contrasting reward-model RLHF with general preference probabilities. Under non-transitive preferences, it targets the von Neumann winner as a symmetric Nash equilibrium in a constant-sum game.

  • Preference-learning setup: An autoregressive language model generates responses token by token, while a preference oracle indicates which of two responses is preferred for a prompt.The preference probability is the expectation of the oracle’s binary feedback.
  • RLHF with reward models: Bradley-Terry RLHF models pairwise choices through a scalar reward and optimizes reward subject to reverse-KL regularization against a reference policy.The resulting policy optimization is commonly solved with reinforcement-learning algorithms such as PPO.
  • RLHF with reward models: Direct preference optimization derives a closed-form policy solution from the regularized reward formulation and converts it into a loss over winner–loser response pairs.This approach still inherits the scalar-score structure implied by Bradley-Terry modeling.
  • RLHF with general preferences: Human preferences can be inconsistent, non-transitive, or cyclic, so a single numerical score may not sufficiently represent them.The paper therefore motivates working directly with pairwise preference probabilities.
  • RLHF with general preferences: The general-preference formulation seeks the von Neumann winner, defined here as the symmetric Nash equilibrium of a two-player constant-sum game.Each player is a policy, and policy-level winning probabilities average response-level preferences over the prompt distribution.

4 Self-Play Preference Optimization(SPPO)

SPPO uses multiplicative self-play updates to approximately solve the constant-sum preference game, then fits a tractable square-loss objective using sampled responses and preference win rates. Its objective preserves both preference gaps and absolute winner–loser likelihood movement, with convergence guarantees for the average policy.

  • Theoretical framework: SPPO is derived from an iterative framework that asymptotically converges toward the optimal policy on average in a constant-sum game.The method starts from exponential or multiplicative weight updates.
  • Theoretical framework: Each update increases a response’s policy weight according to its average advantage over the current policy.The next policy is formed from the current policy through multiplicative weighting and normalization.
  • SPPO objective: SPPO directly approximates the ideal update with an L2 objective instead of canceling the log normalizing factor through pairwise differencing as DPO and IPO do.The objective is motivated both by the game-theoretic update and by policy-gradient theory.
  • Finite-sample algorithm: For each prompt, SPPO samples K responses from the current policy, estimates their win rates through preference queries, and optimizes the resulting dataset.Including all K responses can require O(K^2) preference-oracle queries.
  • Finite-sample algorithm: The finite-sample expectation can be estimated with B new samples using O(KB) preference-oracle queries, and the resulting optimization is efficiently tractable.As K approaches infinity, the finite-sample objective recovers the population objective.
  • Guarantees: Theorem 4.1 bounds convergence of the time-averaged policy toward the Nash equilibrium using the duality gap.The proof adapts a theorem from Freund and Schapire’s analysis of constant-sum games.
  • Connection to policy gradient: SPPO also admits a semi-online policy-gradient interpretation because it samples from the iteration-start policy rather than sampling on-policy at every gradient step.Its win rate acts as the reward, while the normalizing term corresponds to a soft value-function baseline approximated by 1/2.
  • Comparison and interpretation: SPPO’s square loss pushes the winner log-ratio toward 1/2 and the loser log-ratio toward −1/2, not merely their gap toward 1.This encourages higher winner likelihood and lower loser likelihood relative to the reference policy, addressing data sparsity concerns.

5 Experiments

Experiments evaluate SPPO with PairRM on UltraFeedback prompts across AlpacaEval 2.0, MT-Bench, Arena-Hard, and the Open LLM Leaderboard, comparing it with base and iterative DPO/IPO models. SPPO generally improves benchmark performance across iterations, while later alignment iterations can reduce general-task scores.

  • AlpacaEval 2.0: 28.52% is Mistral-7B-SPPO Iter3’s length-controlled win rate against GPT-4-Turbo, alongside a 31.02% overall win rate on AlpacaEval 2.0.The reported gains across iterations are 7.69%, 2.10%, and 1.64%, respectively.
  • AlpacaEval 2.0: 38.77% is Llama-3-8B-SPPO’s length-controlled win rate, with a 39.85% overall win rate after applying SPPO to the stronger base model.Its iteration gains are 8.81%, 3.42%, and 3.62%, totaling 15.85%.
  • AlpacaEval 2.0: SPPO’s 28.53% length-controlled win rate exceeds DPO’s 26.39% and IPO’s 25.45% on AlpacaEval 2.0.DPO and IPO increase average output lengths to 2736 and 2654, while SPPO reaches 2163 from a base length of 1676.
  • MT-Bench and Arena-Hard: 7.59 is Mistral-7B-SPPO Iter3’s average MT-Bench score, exceeding all baselines and improving over the base model despite early iteration drops.On Arena-Hard, SPPO Iter3 achieves an average score of 23.3 and shows steady gains across iterations.
  • Open LLM Leaderboard: 66.75 is SPPO’s state-of-the-art average Open LLM Leaderboard score for Mistral-7B, but subsequent DPO, IPO, and SPPO iterations decline.The supplied results attribute this boundary to a possible alignment-tax effect when optimizing against PairRM preferences.
  • Ablation Study: SPPO remains robust to noise in win-rate estimation: the initial K = 5 advantage over K = 2 diminishes in later iterations, while K = 2 increases output length less.Here, K denotes the number of sampled responses used for estimating the win rate.

6 Conclusions

SPPO is introduced as an approach for fine-tuning LLMs from human or AI feedback, improving over DPO and IPO across several benchmarks while retaining theoretical and experimental limitations.

  • SPPO fine-tunes LLMs from human or AI feedback using a preference model and a new optimization objective.
  • SPPO improves over DPO and IPO across AlpacaEval 2.0, MT-Bench, Arena-Hard, and the Open LLM Leaderboard.
  • Theoretical approximation relies on sufficient model expressiveness and generated-data coverage.
  • Experiments use one dataset and a limited set of benchmarks because of computational constraints.

A Approximating the Normalizing Factor

The paper approximates the normalizing factor under maximally disordered and ordered preference assumptions, choosing η/2 for experiments motivated by intransitive and irrational preferences.

  • The analysis considers two extreme preference regimes: fair-coin preferences and a strict ordering among responses.
  • The most “disordered” case: Under the disordered regime, pairwise preferences are modeled as independent fair coin tosses, with deterministic feedback derived from preference probabilities.
  • As K →∞, the expectation approaches 1 while the variance and covariance approach 0.
  • For large η ≈1e3, the approximation is characterized using the chosen constant and can be tuned as a hyperparameter.
  • Choice of η: The normalizing factor lies between η/2 and η depending on preference disorder, motivating the experimental choice η/2.

B Proof of Theorem 4.1

The proof analyzes an exponential-weight policy update and shows that, with suitable learning rate and iterations, the resulting mixture policy approaches the minimax-optimal Nash equilibrium.

  • The policy update multiplies the current policy by exp(ηP(y ≻πt|x)) and renormalizes across responses.
  • The proof invokes a no-regret exponential-weight result for arbitrary preference oracles and sequences of mixed policies.
  • Finite support of the autoregressive base policy provides a bounded KL-divergence term used in the analysis.
  • For appropriately chosen η and T, the mixture policy is close to the minimax-optimal policy, or Nash equilibrium.

C Response Examples in Different Iterations

This section provides generation examples for SPPO fine-tuned models across different iterations.

  • Table 4 presents a generation example from the SPPO fine-tuned model at different iterations.
  • Table 5 presents another generation example from the SPPO fine-tuned model at different iterations.
Loading 2405.00675v5…