Source-linked AI summary
Nash Learning from Human Feedback
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, Bilal Piot
TL;DR
NLHF addresses limitations of reward models by learning a pairwise preference model and seeking its Nash equilibrium. The paper introduces Nash-MD and deep-learning variants, with intermediate mixture parameters performing best in a text-summarization comparison.
Problem
Existing reward models may not capture the full spectrum of human preferences and can depend on the sampling distribution.
Method
NLHF learns a pairwise preference model and optimizes policies toward its Nash equilibrium using Nash-MD and deep-learning variants.
Results
Nash-MD-PG models with β ∈[0.125, 0.375] emerge as the best-performing methods in the text-summarization pairwise comparison.
Takeaways & Limitations
NLHF offers an alternative to RLHF that uses preference models without specific assumptions such as Bradley-Terry and can preserve models across new data iterations.
Takeaways & Limitations
The experiments are intended to illustrate practical implementation rather than establish superiority over RLHF, requiring a larger and more intensive comparison.
Abstract
from arXiv · showhide
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human preferences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
1. Introduction
NLHF replaces reward-model optimization with a preference model over response pairs and a Nash-equilibrium objective. It is designed to capture richer preferences, reduce distribution sensitivity, and support practical mirror-descent and deep-learning algorithms.
- RLHF commonly learns a Bradley-Terry reward model from pairwise preferences before optimizing the language-model policy against it.
- BT-based reward models may fail to represent the full spectrum of human preferences and can misalign reward maximization with winning probabilities.
- NLHF learns a preference model conditioned on two responses and seeks a policy preferred over responses generated by any alternative policy.
- Because it makes no Bradley-Terry assumption, the learned preference model can capture diversity and richness present in the training data.
- Preference models are less sensitive to data distribution than reward models because they directly compare two responses rather than implicitly comparing one response with a training distribution.
- Nash-MD computes regularized Nash equilibria with mirror descent, while its deep-learning variants use competing policies formed from current, initial, or moving-average policies.
2. Prior work
Prior work spans preference-based reinforcement learning, trajectory-level pairwise feedback, theoretical guarantees, and direct policy optimization without an explicit reward function. NLHF is situated among approaches that learn or optimize from preferences rather than relying solely on scalar rewards.
- Preference-based reinforcement learning directly uses pairwise human preferences instead of a hand-designed or learned scalar reward.
- Pairwise trajectory feedback has been applied to summarization, question answering, and language-based assistants because it is simple to implement.
- Theoretical preference-learning work has studied histories, state-action rankings, policy comparisons, policy distributions, dueling reinforcement learning, and Nash equilibria.
- Function-approximation results in preference-based reinforcement learning existed before this work but included a computationally inefficient algorithm.
- Direct Preference Optimization and related methods optimize policies through losses based on Bradley-Terry rewards, ranking, or sequence contrasts.
3. The preference model and its Nash equilibrium
The paper models pairwise response preferences as an antisymmetric two-player constant-sum game and defines alignment through its Nash equilibrium. This formulation directly optimizes preferences and can represent preference structures that scalar reward models miss.
- The preference model assigns P(y ≻ y′|x) between 0 and 1 and assumes antisymmetry: P(y ≻ y′|x) = 1 − P(y′ ≻ y|x).
- Policy preference is obtained by averaging response-level preferences over alternative-policy responses and the context distribution.
- A policy wins against another when its policy-level preference is at least 1/2, and seeking a policy that wins against every alternative defines the objective.
- The objective forms an antisymmetric constant-sum two-player game whose equilibrium is supported by the minimax theorem.
- Unlike most RLHF objectives, this formulation uses preferences directly and compares policies through Nash equilibrium rather than optimization against a fixed behavior.
- Preference models can represent non-transitive preferences and other structures that Bradley-Terry reward models cannot fully capture.
- In the three-action example, the Nash equilibrium assigns probabilities near uniformly across actions, whereas the RLHF reward optimum deterministically selects one action.
- Ideal preference-model learning is independent of the response-generation distribution, while reward-model solutions depend on the policy that generated training data.
4. Regularized preference model
The regularized preference model incorporates proximity to a reference policy when preferences are better estimated near that policy or safety requires staying close to it. The resulting model has a unique Nash equilibrium.
- Regularization is motivated by settings where preference estimates are more accurate near a reference policy or where solutions should remain close to a known safe policy.
- The model defines regularized action preferences and a KL-regularized preference between policies.
- The regularized preference model has a unique Nash equilibrium.
5. Algorithms for approximating the Nash eq.
The paper situates Nash equilibrium approximation within constant-sum games, contrasting average-convergent methods with methods designed to converge in the last iterate.
- Convergence on average: Fictitious play repeatedly computes best responses against the uniform mixture of the opponent’s past strategies.
- Convergence on average: In constant-sum games, fictitious play’s mixture policy converges to the Nash equilibrium, while function-approximation variants have also been studied.
- Online convex optimization: Online convex optimization frames each player as minimizing a convex loss in convex-concave constant-sum games.
- Online convex optimization: Some online methods achieve faster convergence rates, including O(1/t), than methods whose rates are expressed through average regret.
- Convergence of the last iterate: Extragradient, optimistic mirror descent, and related methods target convergence of the last iterate to a Nash equilibrium.
6. Analysis of a tabular algorithm: Nash-MD
Nash-MD is a mirror-descent algorithm that regularizes policy updates toward a reference policy and proves last-iterate convergence to a regularized Nash equilibrium.
- Algorithm: Nash-MD uses a geometric mixture of the current policy and reference policy as its regularized opponent policy.
- Algorithm: The update improves preference against the regularized opponent while limiting deviation through a KL penalty.
- Convergence guarantee: With η_t = 2/(τ(t + 2)), the last iterate’s KL distance to the regularized Nash equilibrium decays at speed O(1/T).
- LLM-relevant properties: Unlike fictitious play, Nash-MD needs only one geometric mixture rather than the full mixture of past policies.
- LLM-relevant properties: Last-iterate convergence avoids storing a mixture policy, reducing a memory burden relevant to LLM applications.
- Comparison with OMD: Nash-MD differs from ordinary OMD by optimizing preference against the regularized mixture instead of the current opponent policy.
- Contextual bandits: The analysis extends to contextual bandits by conditioning policies and preferences on prompts and defining KL divergence as an expectation over prompts.
7. Deep learning implementation of NLHF
For deep policies, the paper derives policy-gradient implementations that compare current-policy responses with responses from either a geometric mixture or an exponential moving-average policy.
- Policy-gradient algorithms: The gradient algorithms sample a prompt, a response from the current policy, and a response from an alternative policy.
- Alternative policies: Nash-MD-PG uses an alternative policy defined as a geometric mixture of the current policy and reference policy.
- Alternative policies: Nash-EMA-PG uses an alternative policy whose parameters are an exponential moving average of past parameters.
- Variance reduction: Subtracting the self-preference baseline 1/2 reduces gradient variance without requiring a learned value function.
8. Experiments on a text summarization task
The summarization experiments compare NLHF variants with RLHF using pairwise PaLM 2 preferences; Nash-MD-PG performs best, especially at intermediate mixture values, but the comparison is illustrative rather than definitive.
- Experimental setup: The experiment compares Self-Play, Best-Response, Nash-MD-PG, Nash-EMA-PG, and an RLHF baseline on text summarization.
- Results: Nash-MD-PG models, especially those with β ∈ [0.125, 0.375], are the best-performing methods in the pairwise comparison.
- Results: Intermediate β values from 0.125 to 0.375 consistently outperform Self-Play and Best-Response.
- Caveat: NLHF and RLHF use different learned models, so model quality must be included in a full comparison.
- Caveat: The experiments illustrate practical NLHF implementation rather than establish superiority, which would require a larger and more intensive evaluation.
- Caveat: The methods were not individually hyperparameter-optimized, and shared parameters may explain differences from another study’s pairwise results.
9. Conclusion and future work
NLHF is presented as an alternative to RLHF that uses preference models and Nash-equilibrium optimization, with Nash-MD as a self-improvement algorithm. The paper establishes tabular convergence, develops deep-learning variants, and identifies mixture strategies for future work.
- Conclusion: NLHF uses supervised preference learning and Nash equilibrium as an alternative to reward-model-based RLHF.The approach avoids Bradley-Terry assumptions when learning from pairwise preferences.
- Conclusion: Nash-MD improves the current model by comparing it with a geometric mixture of the current and a past policy.The mixture coefficient β ranges from 0, self-play, to 1, play against a fixed policy.
- Conclusion: In tabular policy representations, the last Nash-MD iterate converges to the regularized preference-model Nash equilibrium.Deep variants Nash-MD-PG and Nash-EMA-PG extend these ideas to neural policies and LLMs.
- Conclusion: The authors report applying the deep variants to LLM fine-tuning on a text-summarization task.
- Future work: Future work includes mixtures of current policies with past checkpoints and decaying β toward zero in deep Nash-MD variants.
A. Maximizing expected Elo vs maximizing probability of winning
The appendix shows that expected Elo reward and probability-of-winning optimization can disagree under policy constraints, even when preferences fit Bradley-Terry exactly. A three-action example demonstrates the distinct optima.
- Example setup: A Bradley-Terry reward model can perfectly represent the example’s preferences, with Elo scores R(y1)=0, R(y2)=log 9, and R(y3)=log 2.
- Constrained optima: Under the constraint π(y1)=2π(y2), the expected-reward optimum is πdef=(2/3, 1/3, 0), while the preference optimum is πdef=(0, 0, 1).
- Objective disagreement: The reward objective favors the mixed policy, whereas the preference objective favors y3 with winning probability 50/99 > 1/2.The reported reward comparison is 0 × 2/3 + log(9) × 1/3 > log(2).
- Conclusion: Thus, constrained optimization can produce different policies when maximizing expected Elo reward versus maximizing probability of winning.The constraint may also be imposed softly through KL regularization toward a reference distribution.
B. Sensitivity of reward models w.r.t. the sampling distribution
The paper argues that Bradley-Terry reward models can depend on the sampling distribution and that aggregated human preferences can be non-transitive. It also develops regularized Nash formulations and deep-learning optimization approaches.
- Sampling-distribution sensitivity: The sensitivity arises because Bradley-Terry fitting matches preferences under sampled responses rather than preserving the full pairwise preference model.The proof uses sampled human-preference comparisons and the optimization loss over π.
- Sampling-distribution sensitivity: When a preference model is not perfectly captured by Bradley-Terry, the optimal Bradley-Terry reward model depends explicitly on the sampling distribution π.The paper states that changing π can change reward differences and the corresponding Bradley-Terry preference model.
- Non-transitive aggregation: Even transitive individual human preferences can aggregate into a non-transitive preference model.In the example, each human has an ordering, but the aggregate satisfies P(y1 ≻y2)=P(y2 ≻y3)=P(y3 ≻y1)=2/3.
- Non-transitive aggregation: The paper’s preference model represents pairwise outcomes as probabilities over human or outcome-level comparisons, allowing non-transitive aggregate preferences.The indicator-based formulation assigns half probability when the underlying scores are equal.
- Regularized Nash formulation: Regularized preference mappings are concave in one policy and convex in the other, supporting Nash-equilibrium existence; strict convexity or concavity supports uniqueness.
- Deep-learning optimization: Deep-learning algorithms are developed to compute Nash equilibria, including a model-free variant that replaces the learned preference model with immediate human comparisons.The model-free estimate requires access to human preferences immediately after generating both responses.
F.3. Choice of the alternative policy π′
The parametric NLHF implementations choose alternative policies through geometric mixtures or exponential moving averages, while using gradient descent rather than directly implementing tabular mirror descent. Experiments compare these algorithms and model types on summarization preference data.
- Alternative-policy algorithms: Nash-MD-PG and Nash-EMA-PG use alternative policies inspired by mirror ascent and fictitious play with an exponential moving average, respectively.Nash-MD-PG uses a geometric mixture, while Nash-EMA-PG uses an exponential moving average of past parameters.
- Implementation differences: The parametric policy-gradient implementation differs from tabular Nash-MD because it takes a gradient step instead of fully solving the inner optimization problem.A more faithful implementation would use a two-timescale algorithm with several inner gradient steps before updating the policies.
- Regularization: The KL regularization in Nash-MD-PG can be expressed relative to the reference policy, with an additional mixture parameter β tuned independently of τ.The equivalence uses the mixture definition and a normalizing constant independent of the generated response.
- Nash-MD-PG: Nash-MD-PG improves the current policy against a geometric mixture of the current policy and a reference policy.The geometric-mixture formulation makes the policy-gradient implementation capture the central Nash-MD idea.
- Preference and reward models: Preference models score the probability that one summary is preferred to another, whereas reward models assign scalar values and derive pairwise preferences through Bradley-Terry.Both model types are trained on pairwise preference data; the preference model directly receives the two summaries being compared.
- Experiments: 0.78 versus 0.76: the T5X-XL preference model reached slightly higher peak test accuracy than the same-size reward model on summarization.Preference-model accuracy increased with model size, but gains from T5X-XL to T5X-XXL were relatively small, so later experiments used T5X-XL.
G.7. Analysis of the results
Nash-MD performs especially strongly at intermediate mixing values, while self-play and best-response extremes are suboptimal. The RLHF baseline is strong, but Nash-MD with β = 0.125 surpasses the other evaluated models, whereas Nash-EMA underperforms Nash-MD and RLHF.
- 99% win rate against SFT makes the RLHF baseline a very strong comparator.
- Self-play performs strongly overall but is vulnerable to exploitation by certain Nash-MD models and does not guarantee reaching a Nash equilibrium.
- Nash-MD models with β ≤0.5 perform very strongly, with β = 0.125, β = 0.25, and β = 0.375 outperforming all other models, including RLHF.
- All Nash-EMA variants are outperformed by Nash-MD for β ≤0.5 and by RLHF, suggesting EMA may be poorly suited here.
- Both β = 0 self-play and β = 1 best-response against SFT are suboptimal compared with intermediate Nash-MD values, especially β = 0.125, β = 0.25, and β = 0.375.