PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie

arXiv:2404.03715v1cs.LGcs.AIcs.CL

TL;DR

Reward-based RLHF cannot represent general intransitive or cyclic preferences, while scalable methods for directly optimizing such preferences remain unclear. The paper introduces Direct Nash Optimization, a batched on-policy regression method with convergence and monotonic-improvement guarantees. DNO produces a 7B Orca-2.5 model achieving 33% win-rate against GPT-4-Turbo on AlpacaEval 2.0 after length control.

  • Problem

    Reward-based RLHF cannot express general intransitive or cyclic preferences, and scalable optimization of general preferences remains unclear.

  • Method

    Direct Nash Optimization uses batched on-policy iterations with a regression-based contrastive objective to optimize general preferences.

  • Results

    33% win-rate against GPT-4-Turbo was achieved by the 7B Orca-2.5 model on AlpacaEval 2.0 after controlling for response length.

  • Takeaways & Limitations

    DNO combines scalable contrastive optimization with general-preference modeling and supports iterative self-improvement beyond a strong teacher.

  • Takeaways & Limitations

    Errors in critical components such as preference annotation can propagate across iterations, requiring rollback and potentially wasting time and cost.

Abstract

from arXiv · show

This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy optimization. However, such a reward maximization approach is limited by the nature of "point-wise" rewards (such as Bradley-Terry model), which fails to express complex intransitive or cyclic preference relations. While advances on RLHF show reward learning and policy optimization can be merged into a single contrastive objective for stability, they yet still remain tethered to the reward maximization framework. Recently, a new wave of research sidesteps the reward maximization presumptions in favor of directly optimizing over "pair-wise" or general preferences. In this paper, we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm that marries the simplicity and stability of contrastive learning with theoretical generality from optimizing general preferences. Because DNO is a batched on-policy algorithm using a regression-based objective, its implementation is straightforward and efficient. Moreover, DNO enjoys monotonic improvement across iterations that help it improve even over a strong teacher (such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of 33% on AlpacaEval 2.0 (even after controlling for response length), an absolute gain of 26% (7% to 33%) over the initializing model. It outperforms models with far more parameters, including Mistral Large, Self-Rewarding LM (70B parameters), and older versions of GPT-4.

1 Introduction

RLHF traditionally optimizes scalar reward models, which cannot represent all general preferences, motivating DNO as a scalable contrastive approach for directly optimizing such preferences. DNO combines batched on-policy regression with theoretical guarantees and achieves strong empirical performance.

  • Scalar reward functions cannot express all pairwise preferences, including intransitive or cyclic relations.
  • DNO targets general preferences while seeking an efficient implementation comparable to contrastive reward-based optimization.
  • DNO decomposes optimization into batched on-policy iterations, each using a simple regression objective that is easy to implement at scale.
  • DNO converges to the intended Nash equilibrium on average and can improve monotonically across iterations, with bounded finite-sample approximation error.
  • DNO’s practical algorithm uses online policy samples, GPT-4 preference judgments, teacher comparisons, and large-margin training pairs.
  • 33% win-rate against GPT-4-Turbo was achieved by the 7B Orca-2.5 model on AlpacaEval 2.0 after controlling for response length.

2 Preliminaries

The preliminaries distinguish reward-based RLHF from direct optimization of general preferences. General preferences may be cyclic, so the learning objective is formulated through a Nash equilibrium, while existing algorithms face scalability challenges from on-policy estimation.

  • Batched on-policy learning collects data iteratively in large batches, permits other offline data, and allows more substantial policy updates than purely on-policy learning.
  • RLHF with reward models learns a scalar reward from preference data, then fine-tunes the language model with reinforcement learning.
  • DPO combines reward learning and policy optimization into a single objective using a policy’s internal reward representation.
  • General preference optimization allows intransitive relations, such as a preferred over b, b over c, and c over a.
  • The learning goal for general preferences can be defined as the Nash equilibrium of a two-player zero-sum game whose payoffs are preference probabilities.

3 Direct Nash Optimization

DNO replaces unstable or costly direct soft-policy updates with batched on-policy regression that approximates each iteration’s target. Its analysis establishes bounded approximation error and monotonic policy improvement under the stated assumptions.

  • Algorithm design: DNO returns a uniform mixture of iterates, while validation data can select the best policy among the learned iterations for deployment.The theoretical algorithm returns the mixture policy, but practical deployment can use validation to choose a policy from the iteration sequence.
  • Algorithm design: The regression operates on response pairs and uses binary cross-entropy, avoiding the partition-function and on-policy sampling difficulties of direct policy optimization.The learning objective regresses a prediction to a preference-derived goal from tuples (x, y1, y2).
  • Algorithm design: DNO uses batched on-policy updates and a regression-based objective to approximate explicit soft policy iteration.The algorithm avoids directly pushing the policy toward the soft policy-iteration target, instead regressing an internal reward toward preference-based rewards.
  • Monotonic improvement: DNO’s batched on-policy design enables monotonic improvement across iterations, unlike alternatives whose iterative updates may be undefined, unstable, or unable to approximate the target accurately.The guarantee depends on learning each soft policy-iteration target accurately enough.
  • Theoretical analysis: Finite-sample analysis tightly bounds the total-variation approximation error between the learned policy and the soft policy-iteration target under realizability and boundedness assumptions.The bound depends on a concentrability coefficient that informs the choice of sampling policies for sample efficiency.
  • Theoretical analysis: The sampling distributions should be near on-policy because fitting the soft policy-iteration target differs from Bradley-Terry reward estimation.The analysis identifies on-policy sampling as important beyond policy-optimization-style RLHF algorithms.

4 Practical Algorithm – Iterative Contrastive Self-Improvement

DNO-Prct is an iterative, batched on-policy contrastive-learning implementation of DNO that uses sampled preference comparisons and large-margin pairs for practical self-improvement. Its design preserves scalability while addressing bounded reward targets and supports extensions beyond iterative DPO.

  • DNO-Prct overview: DNO-Prct performs iterative self-improvement through batched on-policy contrastive learning, using prompt-specific sampled responses and preference-based pair construction.The algorithm initializes a reference policy, samples multiple outputs from the current policy, ranks them with the general preference function, and learns the next policy contrastively.
  • On-policy sampling: DNO-Prct avoids explicit reward computation by sampling current-policy responses and using their preference outcomes to estimate the internal reward signal.The current policy supplies the response samples, while preference-function samples provide the pairwise comparisons used in training.
  • Preference pair construction: Large-margin pairs replace unavailable exact preference probabilities, making the contrastive approximation more reliable when reward targets are bounded.Scaling reward and regularization together preserves the soft policy iteration target, but increasing the regularization parameter can worsen sample complexity; large-margin filtering helps address this trade-off.
  • Relationship to DPO: DNO-Prct resembles iterative DPO by design, but the broader DNO framework can extend to regularized preferences and other sampling techniques.The shared soft policy iteration structure does not make DNO and DPO theoretically identical: DNO is framed around general preferences and online-learning principles.
  • Experimental comparison: Figure 2 compares post-training methods under matched 7B Orca-2.5 initialization and mostly batched on-policy settings, with SFT and Offline DPO treated as epochs.The figure caption identifies DNO as the most effective method in this comparison.

5 Experiments

Experiments evaluate DNO through iterative on-policy training, controlled benchmarks, comparisons with baselines, pair-construction variants, scaling, and documented limitations. DNO improves over off-policy and supervised alternatives, while performance and data-quality dynamics introduce practical caveats.

  • Experimental setup: DNO iterations sample current-policy outputs, obtain preference annotations, and train the next policy with newly constructed pairs.
  • Experimental setup: AlpacaEval 2.0 measures head-to-head win-rate against GPT-4-Turbo on 805 prompts and reports a length-controlled variant.
  • Baselines: Offline contrastive training methods outperform additional SFT, indicating that positive-negative differences provide more training signal than positives alone.
  • On-policy versus off-policy: On-policy methods, especially DNO, outperform four-epoch Offline DPO despite using only three iterations and one-third of UltraFeedback inputs per iteration.
  • Comparisons: 24.97 win-rate versus 20.44 on AlpacaEval 2.0, and 7.46 versus 7.25 on MT-Bench, are reported for DNO and the 70B Self-Rewarding model respectively.
  • Pair construction: DNO constructs teacher-student pairs even when the student is preferred, whereas DNO-Restrictive and related methods exclude these comparisons.
  • Scaling: DNO-More-Data scales with a tenfold expansion of instruction data distributed across six non-overlapping partitions.
  • Limitations: Preference-annotation anomalies can propagate across iterations, while contrastive training produces an initial output-length spike of at least twofold over the initializing SFT model.

6 Related Work

Related work spans reward-based, contrastive, iterative, and general-preference approaches across offline and online settings. DNO is positioned against methods that use self-play or weaker annotators without a more powerful teacher.

  • Online RLHF: Online RLHF is unstable and memory-intensive because training requires policy, reward, and advantage models on device.
  • Offline preference learning: Offline contrastive methods such as DPO and SLiC optimize pairwise preferences without explicit reward learning, while RSO, LIPO, and PRO extend preference construction or scope.
  • Iterative reward-based finetuning: Iterative reward-based methods sample from the current policy, filter outputs with rewards or AI feedback, and use selected data for subsequent improvement.
  • On-policy contrastive learning: Self-Rewarding Language Models iteratively train on preferences from sampled outputs but use the policy itself as an initially weak annotator.
  • On-policy general preference optimization: General-preference methods study Nash equilibria or related game-theoretic objectives, including multi-agent RL, Nash-MD, and online AI-feedback approaches.
  • Teacher comparisons: Existing self-play methods compare student outputs with other student outputs or the initial reference policy, while omitting student-versus-teacher preferences may hinder performance.

7 Conclusion

DNO targets general preference models while retaining a practical, scalable implementation and finite-sample analysis. Its theory establishes convergence properties, and experiments show monotonic iterative improvement in a 7B model.

  • DNO combines general-preference optimization with a batched regression-based contrastive objective designed to approximate soft policy iteration more stably.
  • The analysis establishes an eO(1/N) concentration bound on squared total variation error between the learned policy and its soft policy iteration target at iteration t.
  • DNO converges to the Nash equilibrium on-average and empirically exhibits monotonic improvement across iterations.

A Extension to Regularized Preferences

The regularized-preference extension adapts DNO and its practical implementation to KL-regularized objectives. It supports alternative convergence options, smoothed-policy sampling, batched ranking, pair construction, and contrastive updates.

  • Extension: The extension applies DNO to regularized preferences and provides corresponding theoretical and practical algorithms.
  • Relation to prior methods: SPO differs from Nash-MD by using the last-iteration policy for reward construction and soft policy iteration, whereas Nash-MD uses its smoothed version.
  • Algorithm 3: Algorithm 3 takes a general preference function, learning rate, KL-regularization coefficient, iteration count, and prompt distribution as inputs.
  • Algorithm 4: Its practical version samples batched on-policy responses, ranks them using pairwise win rates, filters large-margin preference pairs, and applies contrastive learning.
  • Algorithm 4: The implementation can use either the current policy or a smoothed policy for sampling, corresponding to on-average or last-iteration convergence options.
  • Implementation consideration: Sampling from the smoothed policy introduces an implementation consideration, with token-level mixtures between the current and reference policies offered as an alternative.

B Detailed Proofs

The proofs analyze DNO under realizability, bounded log-probability, feasible-policy, and concentrability assumptions. They convert regression guarantees into policy-space error bounds using concentration, Pinsker’s inequality, and f-divergence arguments.

  • Proof framework: The theoretical analysis intentionally simplifies version-space and concentrability ideas from reinforcement learning literature, leaving exhaustive analysis outside the paper’s primary scope.
  • Feasible solution space: At each iteration, the feasible solution space contains the policy produced by Algorithm 1 regardless of data-sampling randomness.
  • Feasible solution space: The feasible space is analogous to a version space containing policies with small empirical loss that can yield small population loss through concentration.
  • Concentrability: The concentrability coefficient extends an offline reinforcement-learning concept to the general-preference setup over the iteration-specific feasible solution space.
  • Assumptions: The realizability assumption requires the soft-policy iteration update to remain representable within the feasible solution space.
  • Assumptions: The boundedness assumption constrains log-probability ratios between candidate and current policies to lie within [−Rmax, Rmax].
  • Proof steps: The proof first relates logarithmic-loss regression to squared error, then uses policy-space concentration to bound distance between learned and target policies.
  • Proof steps: Pinsker’s inequality and an f-divergence construction connect regression error to total variation distance between policies.

C Additional Experimental Details

The additional experiments describe batched GPT-4 preference annotation, the additive scoring rubric, and costs for scaling DNO to 600k training inputs.

  • Preference Annotation: GPT-4 evaluates all candidate responses side-by-side using an additive 6-point scoring rubric rather than separate pairwise annotation requests.The rubric assigns points for answer format, relevance, and coverage of the user's question.
  • Cost Analysis: The scaled-up experiment applies the cost analysis to 600k training inputs across six iterations, covering sampling, GPT-4 annotation, and training.The major line items are sampling outputs, annotating them to construct training pairs, and training the next iteration.
  • Cost Analysis: Sampling five outputs for 100k examples took about 18–24 hours and cost about $6,000 on ten 8xA100 80GB pods.The estimate is based on spot pricing and varied with average output length.
  • Cost Analysis: GPT-4 annotation used about 450M prompt tokens and 60M completion tokens on average across iterations, costing about $34,000.The estimate reflects the endpoint version used for annotation.
  • Preference Annotation: The annotation prompt requires a brief justification and a separate total score for each candidate in a structured dictionary format.Each answer is scored independently, with the output restricted to the specified dictionary format.
Loading 2404.03715v1…