PaperlayerView exact revision on arXiv

Source-linked AI summary

A Minimaximalist Approach to Reinforcement Learning from Human Feedback

Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, Alekh Agarwal

arXiv:2401.04056v2cs.LG

TL;DR

Reward-model RLHF assumes preferences can be represented by a total order and dueling approaches can be unstable with function approximation. SPO instead frames preference learning as a zero-sum game and uses self-play; it almost exactly computes the Minimax Winner in an intransitive example, while reward modeling forces a distant corner solution.

  • Problem

    Reward modeling assumes a total order over behavior, conflicting with intransitive and stochastic preferences, while dueling approaches inherit adversarial-training instabilities with function approximation.

  • Method

    SPO frames RLHF as a two-player zero-sum game, replaces reward modeling with pairwise preference comparisons, and uses a single agent in self-play to compute the Minimax Winner.

  • Results

    SPO almost exactly computes the Minimax Winner across sub-population weightings, whereas reward modeling forces a total order and typically converges far from it.

  • Takeaways & Limitations

    SPO provides a simple approach that the paper evaluates theoretically and across diverse control tasks, while remaining robust to noisy, intransitive, and non-Markovian preferences.

  • Takeaways & Limitations

    The experiments do not explore the contextual setting, and scalable approximations for modern generative modeling are left for future work.

Abstract

from arXiv · show

We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training and is therefore rather simple to implement. Our approach is maximalist in that it provably handles non-Markovian, intransitive, and stochastic preferences while being robust to the compounding errors that plague offline approaches to sequential prediction. To achieve the preceding qualities, we build upon the concept of a Minimax Winner (MW), a notion of preference aggregation from the social choice theory literature that frames learning from preferences as a zero-sum game between two policies. By leveraging the symmetry of this game, we prove that rather than using the traditional technique of dueling two policies to compute the MW, we can simply have a single agent play against itself while maintaining strong convergence guarantees. Practically, this corresponds to sampling multiple trajectories from a policy, asking a preference or teacher model to compare them, and then using the proportion of wins as the reward for a particular trajectory. We demonstrate that on a suite of continuous control tasks, we are able to learn significantly more efficiently than reward-model based approaches while maintaining robustness to the intransitive and stochastic preferences that frequently occur in practice when aggregating human judgments.

1. Introduction

RLHF uses comparative feedback, but reward-model and dueling approaches face limitations with intransitive, stochastic, and complex preferences. SPO addresses these issues through a reward-model-free, self-play formulation.

  • Motivation: RLHF optimizes policies from relative behavioral feedback rather than absolute scores.Comparative feedback is easier for human raters to provide than absolute scoring.
  • Existing approaches: Reward-based RLHF trains a reward model on preferred and dis-preferred behavior, then optimizes the learned reward.This is a two-stage procedure built around a classification objective.
  • Limitations: Reward modeling assumes a total order, excluding intransitive preferences and potentially contributing to generation collapse under stochastic human preferences.Aggregating preferences across raters also makes transitivity unlikely to hold.
  • Existing approaches: Dueling methods frame RLHF as a two-player zero-sum game but inherit adversarial-training instability and have mostly been applied to bandit problems.The cited literature includes some more recent sequential exceptions.
  • SPO: SPO removes reward modeling and adversarial training by using symmetric self-play between a policy and itself.It samples multiple trajectories, compares pairs with a preference or teacher model, and rewards each trajectory by its win rate.

1. We derive SPO: an algorithm for RLHF that avoids reward modeling, compounding errors, and adversarial

SPO uses the Minimax Winner framework and game symmetry to replace two-policy dueling with single-agent self-play. Its analysis gives convergence guarantees, while experiments report stronger sample efficiency across diverse preference settings.

  • Algorithm: SPO frames preference optimization as a two-player zero-sum game based on the Minimax Winner from social choice theory.The payoff symmetry then supports training a single agent against itself.
  • Theory: With intransitive preferences, SPO converges to an approximate Minimax Winner at the underlying no-regret algorithm’s rate.When a reward function exists, it converges to the optimal policy at a fast rate matching standard techniques.
  • Experiments: SPO is more performant and sample-efficient than reward-model-based approaches across continuous-control tasks and varied preference setups.The setups include Markovian, stochastic, non-Markovian, and aggregated intransitive preferences.
  • Comparison: Table 1 provides a taxonomy of RLHF algorithms and the preference-related issues to which they are robust.The supplied table text does not specify individual rows or comparisons.

2. Reinforcement Learning from Human Feedback via Game Solving

The paper formulates preference-based RLHF as a zero-sum game and uses Minimax Winners to handle preferences that reward models cannot represent. SPO exploits game symmetry to compute an approximate Minimax Winner with one self-playing agent, while retaining guarantees under both intransitive and reward-consistent preferences.

  • Preference Modeling: Preference-based RLHF queries comparative judgments over pairs of trajectories, which may be supplied by humans or a preference model.The preference function is antisymmetric and can represent relative preferences beyond differences of scalar rewards.
  • Preference Modeling: Reward-model approaches impose a total order, so they cannot represent cycles such as a ≻ c, c ≻ d, and d ≻ a.Their tie-breaking can select an option that leaves part of the population consistently unsatisfied.
  • Minimax Winners: The Minimax Winner selects a policy in the two-player zero-sum preference game and always exists, unlike a unique Copeland Winner.When a reward function explains preferences, the Minimax Winner coincides with the reward-optimal policy.
  • Self-Play Preference Optimization: SPO uses antisymmetry to replace dueling two-policy training with a single agent trained against its own iterates.This avoids adversarial training while preserving the game-solving objective.
  • Self-Play Preference Optimization: Theorem 2.3 guarantees that averaging the iterates of one no-regret algorithm yields a 2Reg(T)/T-approximate Minimax Winner.The result is a reduction from computing Minimax Winners to no-regret online learning and approaches an exact winner with sufficient iterations.
  • Guarantees: SPO converges to the optimal policy at a fast rate when a clearly optimal policy exists, while reward-based RLHF and DPO can fail to compute the Minimax Winner.The paper also gives an approximate-winner guarantee for the Hedge-based implementation under a margin condition.
  • Practical Scope: The practical RL setting requires choosing a no-regret method that can optimize SPO losses over large, often continuous policy classes.The paper leaves contextual experiments and scalable approximations for modern generative modeling applications to future work.

3. Experimental Evaluation

The evaluation compares SPO with iterative reward modeling in the context-free, online-oracle setting across several preference conditions.

  • The experiments compare SPO and iterative RM along multiple evaluation axes in a context-free, online-oracle setting.

1. Can SPO compute MWs when faced with intransitive

The intransitive-preference evaluation aggregates three internally transitive populations and measures SPO against the exact Minimax Winner and a continuous-control qualitative analogue.

  • The study aggregates three populations with internally transitive preferences and measures how far SPO is from the exact Minimax Winner.It also reports qualitative results on a Mujoco continuous-control task where computing the Minimax Winner is infeasible.

2. How sample efficient is SPO on problems with unique

Across ground-truth, noisy, intransitive, and non-Markovian preferences, the experiments examine SPO’s efficiency and robustness against reward-model approaches. The reported results include strong performance under challenging preference structures, while reward modeling can help in some stochastic settings.

  • Ground-truth reward preferences: SPO matches or improves upon reward-model approaches when preferences select trajectories with higher ground-truth reward.The comparison is shown in Figure 6(a).
  • Online updating: Freezing the reward model partway through training causes a consistent decline in RM performance.This result moves the baseline toward an offline RLHF recipe and highlights the role of continued updating in the tested setup.
  • Intransitive preferences: SPO nearly computes the Minimax Winner across sub-population weightings, whereas RM imposes a total action order and often converges far from it.This evaluation uses cyclic preferences formed by aggregating internally transitive sub-populations.
  • Noisy preferences: Under Bernoulli-flipped preference feedback, SPO performs comparably to RM on some environments but noticeably worse on others.The authors note that learning a parametric preference model can provide an empirical benefit in some stochastic situations.
  • Non-Markovian preferences: SPO consistently crosses the 4·rmax threshold in the non-Markovian task, while RM often struggles to do so.The task requires maximizing early rewards before switching to more conservative behavior under a last-quarter constraint.

4. Discussion

The discussion positions SPO as a reward-model-free, self-play reduction for preference optimization, relates it to prior dueling and offline methods, and identifies remaining theoretical, computational, and evaluation boundaries.

  • Limitations: Preference-model methods require sampling multiple times to compute generation scores, creating a computational limitation relative to reward-model methods in contextual settings.
  • Related work: Unlike prior dueling approaches requiring adversarial training, SPO derives a single-player algorithm from the structure of the preference game.The related-work comparison also distinguishes SPO from methods relying on reward-function assumptions or strong linearity assumptions.
  • Scope: The evaluation assumes online access to preference queries and leaves the contextual, offline-dataset setting for future work.The paper separately notes that direct human or expert-model querying may not support online labels in every setting.

C.3. Proof of Theorem 2.4

This proof constructs a preference matrix with a unique Minimax Winner and shows that standard reward-based RLHF and DPO do not recover it in general. The argument compares their induced solutions and losses under a uniform reference policy.

  • The proof uses a preference matrix with a unique Minimax Winner and sets the reference policy πref to uniform.The construction contrasts the Minimax Winner with the Copeland Winner selected by a Bradley-Terry reward model.
  • The Bradley-Terry reward model peaks at the Copeland Winner b rather than the Minimax Winner.The reward-model outputs are assumed to lie in [0, 1].
  • For any finite non-negative β, the optimal reward-based policy plays a and c equally often and therefore cannot play the Minimax Winner.
  • DPO assumes unweighted preferences and transforms log likelihood into cross-entropy, whose constraints cannot all be satisfied for the constructed preference matrix.The proof invokes Gibbs’ inequality to characterize the cross-entropy minimum.
  • For every β ∈ (0, ∞), the DPO loss of the Minimax Winner exceeds the loss of πref, so DPO never selects the Minimax Winner.

C.4. Proof of Corollary 2.6

The proof derives SPO’s convergence guarantee by applying no-regret analysis to history-dependent policies and using the finite-horizon Performance Difference Lemma. The resulting mixture policy is an approximate Minimax Winner.

  • The analysis optimizes over the full history-dependent policy class, whose convexity and compactness enable application of the stated theorem.
  • The finite-horizon Performance Difference Lemma transfers the history-wise regret bound into an overall policy-performance bound.
  • SPO’s no-regret policy update is equivalent to running Hedge with bounded per-history losses.The proof uses losses in [0, 1] and notes that action-dependent terms are interchangeable up to a per-state constant under softmax.
  • The resulting mixture policy is an 8H log(|A|)/T-approximate Minimax Winner.

C.6. Proof of Corollary 2.5

This proof establishes convergence rates for SPO under a gap condition and extends the result to bandit feedback and contextual policies. The analysis uses no-regret updates, importance sampling, and symmetry between the two players.

  • Assumption C.1 separates a target policy set Π⋆ from other policies by a preference gap Δ while bounding within-set and outside-set comparisons.
  • Under the gap condition, the convergence rate improves from eO(1/√T) for Hedge to a faster gap-dependent rate.
  • After T Hedge updates, the mixture policy is a (1+2|Π| ln T)/(ΔT)-approximate Minimax Winner.
  • The proof bounds the probability of policies outside Π⋆ and obtains a fast rate by choosing η = 1 and ε = 1/T.
  • With bandit feedback, importance-weighted losses remain unbiased, allowing a single player to simulate game solving while inheriting the underlying no-regret rate.
  • The contextual extension samples multiple trajectories from one context for comparison and preserves the main theoretical guarantees.

D. Compounding Errors in RLHF

Offline preference learning can suffer from compounding errors because off-policy likelihoods may not distinguish policies that behave differently after trajectory deviations. The problem worsens with longer horizons, although frequent on-policy preference collection may mitigate it.

  • Compounding errors arise when a sequential predictor enters states absent from offline training data and has no demonstrated action for correcting the deviation.This reflects covariate shift between the training-state distribution and the agent’s induced test-time distribution.
  • In the two-step sentence-completion example, two policies have identical preferred and dis-preferred generation probabilities despite differing in how they respond to prior words.
  • Offline approaches such as DPO cannot distinguish those policies when their losses depend only on off-policy likelihoods.
  • The issue is fundamental to offline approaches and becomes worse as task horizons increase, while frequent on-policy preference collection may mitigate it.

E. Additional Results

Additional experiments cover reward-maximization, noisy, non-Markovian, and intransitive preferences across control and bandit settings, with supplementary ablations of reward-model design choices. SPO uses fewer preference-oracle calls than RM in the reported experiments.

  • Supplementary results cover larger control-environment suites with reward-maximization, noisy, and non-Markovian preferences.
  • SPO and RM are compared on larger bandit environments with intransitive preferences, while additional seeds examine SPO in Ant.
  • Additional ablations vary reward-model snippet length and whether the reward model is frozen during training.
  • SPO’s oracle calls equal collected episodes times queue length B, and the experiments report fewer oracle calls for SPO than RM.
  • The bandit study uses three arms and preference functions with unique Minimax Winners, including a Rock-Paper-Scissors case when a = b = c.
  • Bandit experiments run PPO for 50M steps, use B = 1000 for SPO, and use distinct reward-model update and entropy settings.

F.2. Continuous Control Experiments

Continuous-control experiments use SAC with shared function-approximation settings, while SPO and RM receive separate queue, replay, and update configurations. The Ant-v3 setup evaluates an intransitive preference combining distance and angular components.

  • All continuous-control methods use SAC with Adam, three-layer width-256 networks, and ReLU actor and critic activations.
  • SPO uses queue length B = 10 for non-Markovian and intransitive preferences, B = 100 for noisy max-reward preferences, and replay buffers of 1M or 100k.
  • The reward model uses Leaky ReLU activations with a final tanh, updates every 256 policy updates, and uses batch size 64 with learning rate 3e-4.
  • The Ant-v3 preference combines distance and angular preferences, with the angular component favoring trajectories at an angle θ = π/4 in front.
  • The angular preference compares wrapped angular differences against a threshold of θ/2, while the distance preference directly compares distances above the threshold.
Loading 2401.04056v2…