PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, Tong Zhang

arXiv:2402.07314v3cs.LGstat.ML

TL;DR

RLHF commonly relies on a reward function and Bradley–Terry preferences, which assume transitivity despite evidence of intransitive human preferences. This paper formulates RLHF as a KL-regularized minimax game under a general preference oracle and proposes offline and online algorithms. Its case study reports much higher reasoning-task test accuracy for the preference model, with comparable results on other tasks.

  • Problem

    Reward-based RLHF commonly assumes a ground-truth reward and transitive Bradley–Terry preferences, although evidence of intransitive human preferences challenges that assumption.

    Source: §1 · p. 2

  • Method

    The paper formulates RLHF as a KL-regularized minimax game under a general preference oracle and proposes sample-efficient offline and online algorithms.

    Source: §1 · p. 3, §8 · p. 11

  • Results

    The preference model achieved much higher reasoning-task test accuracy and comparable results on other tasks; its ranking-accuracy advantage also improved reward-based algorithms including iterative DPO.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The framework supports reward-model-free learning under general preferences, with finite-sample guarantees under coverage and exploration conditions and practical designs using approximated computational oracles.

    Source: §1 · p. 3, §8 · p. 11

    The preference-model experiment reused a dataset mixture and hyperparameters mainly tuned for the BT model, which may leave its results short of what refined tuning could achieve.

    Source: §1 · p. 2

Abstract

from arXiv · show

We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle. In particular, we do not assume the existence of a reward function and an oracle preference signal drawn from the Bradley-Terry model as most of the prior works do. We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework.

1 Introduction

The introduction motivates RLHF without assuming a reward function or Bradley–Terry preferences, and presents a general-preference framework with offline and online algorithms. A case study reports stronger reasoning-task accuracy for the preference model, while the paper also describes benefits for ranking-based and reward-based algorithms.

1 Introduction

  • BT-based RLHF assumes a ground-truth reward and transitive preferences, an assumption contradicted by evidence about human decision-making.The paper notes that practical RLHF BT models achieve around 70% accuracy when approximating human preferences.Source: §1 · p. 2
  • The preference model achieves much higher reasoning-task test accuracy and comparable results on other tasks than the BT reward model; Table 1 compares their Reward-Bench accuracy.Both models use the same Gemma-2B-it checkpoint and preference dataset.Source: §1 · p. 2, Table 1 · p. 2
  • The general preference oracle is strictly more general than BT and can represent intransitive preferences, including feedback from AI evaluators.The introduction notes that preference-model construction can be more efficient in ranking accuracy than reward modeling.Source: §1 · p. 2
  • The introduction reports that preference-model ranking-accuracy advantages benefit ranking-dependent algorithms and improve reward-based algorithms, including iterative DPO.The open-source preference dataset mixture and hyperparameters were mainly tuned for BT, so the authors expect refined tuning could further improve the preference model.Source: §1 · pp. 2–3
  • The paper studies general-preference RLHF with KL regularization in offline and online settings, proposing sample-efficient algorithms with finite-sample guarantees under coverage and exploration conditions.It also proposes practical designs using reasonable approximations of computational oracles.Source: §1 · p. 3

2 Problem Formulation

The formulation treats RLHF as a KL-regularized minimax game between two policies evaluated by a general preference function. Its objective seeks a max-player policy that remains preferred against any competing policy, while the KL term trades off preference and closeness to the initial model.

2 Problem Formulation

  • The target is the preference function itself; under BT, relative preference log-odds equal reward differences, recovering the reward-based objective as a special case.The authors say their analysis techniques also apply to the relative-log-odds target with slight modifications.Source: §2 · p. 3
  • The game uses two policies with the same support as the initial policy; its unique Nash equilibrium is also called the Minimax Winner or von Neumann Winner.The two Nash policies coincide and are best responses to each other.Source: §2 · pp. 3–4
  • The analysis assumes a finite preference-function class containing the target; results extend to infinite classes with bounded covering number by discretization.The preference-function class is antisymmetric across swapped responses.Source: §2 · p. 4
  • The learning objective seeks an ε-approximate Nash policy for the max-player, consistently preferred under the KL-regularized preference against any competing policy.When the KL penalty is roughly omitted, the policy is consistently preferred by the underlying preference oracle.Source: §2 · p. 4
  • The KL penalty trades off preference under the oracle against staying close to the initial model, and also ensures a stochastic policy under a suitable initial checkpoint.The coefficient η is typically treated as a hyperparameter and adjusted by parameter search.Source: §2 · pp. 3–4
  • By objective symmetry, an ε-suboptimal max-player policy yields a duality-gap bound of 2ε when the min-player uses that same policy.The formulation decomposes the gap into the two players' suboptimalities.Source: §2 · p. 4

3 Improved Algorithms in Offline Setting

The offline section frames learning from a fixed preference dataset as a minimax policy problem and proposes pessimistic planning to address unreliable preference estimates. Its guarantee depends on dataset coverage, while large distribution shifts motivate online data collection.

3.1 Setup

  • The offline setup learns from a pre-collected dataset without further preference-oracle queries and evaluates suboptimality by the gap between Nash and best-response values.The dataset’s comparison samples are assumed to be independently collected as in equation (1).Source: §3.1 · p. 5
  • Finite offline preference data may poorly estimate preferences outside the behavior policy’s coverage, making optimization of the learned model vulnerable to reward over-optimization.The paper describes this as exploiting unreliable estimates to improve a proxy metric while worsening performance under the ground-truth preference.Source: §3.1 · p. 5

3.2 Learning with Pessimism

  • PELHF builds a version space containing the true preference function with high probability, then evaluates each main-agent policy using the most conservative preference function and best response.It solves the minimax game using this conservative value estimate rather than planning directly with the empirical maximum-likelihood estimate.Source: §3.2 · p. 5
  • Theorem 1 gives a high-probability suboptimality guarantee under Assumption 1, with the stated choices of λ and β².The theorem’s bound depends on a coverage coefficient describing how the target policies are covered by the offline dataset; this coefficient resembles unilateral coverage for Markov games [86].Source: §3.2 · p. 6
  • The authors argue that large policy distribution shifts can make offline coverage coefficients impractically large, motivating online oracle queries to enrich the dataset during training.They cite an average density ratio above exp(25) in prior RLHF training and an average KL divergence of 456.4 in a Gemma case study.Source: §3.2 · p. 6

4 Iterative RLHF with Online Exploration

Iterative RLHF alternates sparse policy updates with batches of new oracle-labeled comparisons. The online method pairs a greedy main agent with an uncertainty-seeking enhancer, yielding a high-probability approximate Nash policy guarantee governed by preference-model complexity.

4.1 Setup of Iterative RLHF

  • Each iterative RLHF batch updates the policy pair using historical data, then collects m prompt-response comparisons and queries the oracle before the next update.The paper distinguishes this batch-online procedure from sequential online learning, where the policy updates after each preference pair.Source: §4.1 · p. 6

4.2 Learning with Exploration

  • The information ratio measures worst-case out-of-sample error relative to historical in-sample error, and the eluder coefficient bounds generalization to unseen distributions.The coefficient is defined from the information ratio over the preference-model and policy spaces.Source: §4.2 · pp. 6–7
  • The online algorithm’s main agent exploits historical data while the enhancer seeks policies with high uncertainty relative to the main agent, guiding exploration.The information ratio compares out-of-sample error with in-sample error, while the eluder coefficient limits surprise on new distributions given collected data.Source: §4.2 · p. 7
  • Theorem 2 guarantees that suitable iteration and batch-size choices yield an ε-approximate Nash policy with probability at least 1−δ.The required iteration count depends on the eluder coefficient, which characterizes preference-model complexity.Source: §4.2 · p. 7
  • Searching the entire policy space for the most uncertain enhancer can be challenging in practice, and the enhancer itself has no theoretical guarantee.The paper proposes restricting the exploration step to a subset of policies as a modification.Source: §4.2 · p. 8
  • The restricted exploration set trades off KL proximity to the main agent’s policy against greater information gain, and the enhancer is stated to converge to π*.The authors defer details of this convergence claim to Appendix D.2.Source: §4.2 · p. 8

5 Practical Implementation of Preference Model and Iterative RLHF

The practical implementation approximates the main agent with self-play IPO and the enhancer with rejection sampling, while using an LLM as a generative preference model. The reported comparison says Online ELHF-IPO outperforms the baselines.

Main agent approximates Nash equilibrium oracle via self-play IPO

  • Self-play IPO approximates the Nash-equilibrium oracle by optimizing a loss whose minimizer is stated to be the unique Nash policy.The implementation stops gradients through the policy-generated data.Source: §5 · p. 8

Enhancer explores via rejection sampling

  • The enhancer uses rejection sampling: it samples n responses from the main agent for a prompt, ranks them by a tournament procedure, and keeps the best response.The method is presented as a practical heuristic because neural-network uncertainty estimators generally lack a closed form.Source: §5 · p. 8

Preference model construction

  • Preference pairs are formatted as instruction-following examples, with random response-order switching to mitigate position bias and the probability of decoding A used as the preference estimate.A rubric may be added to the instruction template, but the authors use the current version to focus on the general preference structure.Source: §5 · pp. 8–9

Evaluation

  • Online ELHF-IPO outperforms the baselines in the reported comparison of RLHF algorithms.Table 3 reports gold win rates on 3K UltraFeedback test prompts using Offline DPO as the reference.Source: Table 3 · p. 9

6 Experiments

The experiments compare Online ELHF-IPO with offline DPO, offline IPO, and SFT using preference-model and GPT-4-judged win rates. The reported main result is that Online ELHF-IPO outperforms the baselines.

Model, Dataset, and Evaluation

  • The evaluation uses LLaMA3-8B-based preference win rates on a held-out UltraFeedback test set and GPT-4 Preview win rates on the out-of-distribution AlpacaEval2 prompts.For the first evaluation, the best DPO model is the reference; for AlpacaEval2, GPT-4 Preview serves as both reference and judge.Source: §6 · p. 9

Method and Competitors

  • Online ELHF-IPO runs three iterations, retraining the preference model on historical data each time and restarting self-play IPO from the initial checkpoint.Offline DPO, offline IPO, and SFT are the baselines; the chosen KL coefficient for offline IPO and Online ELHF-IPO is η = 0.1.Source: §6 · p. 9

Simulation framework

  • Offline DPO and IPO use synthetic comparisons from two sampled responses per training prompt, while Online ELHF uses n = 4 in rejection sampling.The rejection-sampling ranking procedure is tournament-style, with complexity linear in n.Source: §6 · pp. 9–10

IPO, DPO, and Online ELHF-IPO

  • Online ELHF-IPO outperforms the baselines in the main reported comparison.The paper identifies Table 3 as the main result.Source: §6 · p. 10

7 Related Work

Theoretical RLHF work has largely studied reward-based learning, while this paper situates its general-preference approach among work on offline, online, and regularized preference learning. It distinguishes its setting from related general-preference methods by the regularization, learning paradigm, and feedback available.

7 Related Work

  • Reward-based RLHF theory spans dueling bandits, online RL, and offline learning, but prior work still optimizes rewards and can admit deterministic optimal policies [73] [48] [51] [88] [80] [40].Xiong et al. formulate reward-based RLHF as a reverse-KL regularized contextual bandit with offline, online, and hybrid guarantees.Source: §7 · p. 10
  • General-preference studies differ here in using reverse-KL regularization, large or infinite state-action spaces, batch online updates, and statistically learning the preference model [22] [65].Dudík et al. [22] study finite actions and sequential updates; IPO [3] and Nash learning [46] optimize against a fixed, known oracle.Source: §7 · p. 10
  • Swamy et al. [58] study non-regularized sequential online learning for continuous control, observing preference scores rather than the binary preference signals used here.Their algorithms reduce to no-regret methods such as Hedge.Source: §7 · pp. 10–11

8 Conclusion

The conclusion presents RLHF with a general preference oracle as a nontransitive, reverse-KL regularized minimax game and reports statistically efficient offline and online algorithms. The accompanying material reviews related RLHF approaches, records author contributions, and points readers to a notation table.

8 Conclusion

  • The proposed general-preference framework captures nontransitive preferences and provides statistically efficient offline and online algorithms with practical approximations of computational oracles.It formulates RLHF as a KL-regularized minimax game between two LLMs.Source: §8 · p. 11

A Authorship and Credit Attribution

  • Contribution notes assign theoretical and offline work to CY and YZ, the online iterative algorithm to WX, practical implementation and most experiments to HD, and advising and resources to NJ and TZ.WX also contributed to proof, writing, and experiments; YZ contributed to proof and writing.Source: A Authorship and Credit Attribution · p. 16

B Notation Table, Related Work, Experimental Details

  • Table 4 collects the notation used throughout the paper.Source: Table 4 · p. 16

B.1 More Related Work

  • Reward-ranked finetuning and iterative DPO are presented as supervised-learning alternatives, but these methods remain within reward-based RLHF [18] [77] [60] [30] [33] [53].The related-work discussion also notes PPO's tuning burden and forgetting reported for reward-ranked finetuning [56] [60] [42] [13].Source: §B.1 · pp. 16–17

B.2 Details of Experiments

The experiments report a state-of-the-art RewardBench test accuracy for the LLaMA3-8B-it preference model, which is used as a stable oracle in simulation. The appendices describe the simulation data and training setup, proof ingredients for Theorem 1, and a pessimistic offline algorithm based on an uncertainty bonus.

B.2 Details of Experiments

  • Reward modeling initializes from an SFT model with a linear scalar-score head and MLE loss; the preferred-response model is trained for one epoch with batch size 256 and learning rate 1e-5.Experiments use 8×A100-40G GPUs with Deepspeed ZeRO-3.Source: §B.2 · p. 17
  • Because human or closed-source preference oracles were unaffordable, simulations use a trained LLaMA3-8B model built from multiple preference datasets and selected comparison pairs.The described processing includes score-ratio filtering or pair sampling for SHP, HelpSteer, UltraFeedback, and UltraInteract.Source: §B.2 · p. 17
  • The LLaMA3-8B-it preference model achieves state-of-the-art test accuracy on RewardBench and serves as a stable preference oracle for simulation.The training stage uses about 700K preference pairs.Source: §B.2 · p. 17

C.1 Proof for Theorem 1

  • Theorem 1's proof bounds preference-estimation error using log-moment and Hellinger arguments, then invokes Nash-equilibrium optimality in the policy comparison.Source: §C.1 · pp. 18–19

C.2 Learning with Pessimism via Uncertainty Bonus

  • PELHF estimates preferences by MLE, subtracts a confidence bonus for the max-player's lower-confidence estimate, and solves the resulting minimax problem.The bonus is constructed from out-of-sample and in-sample errors to control a pointwise confidence interval.Source: §C.2 · pp. 20–21

C.2.1 Analysis for Algorithm 3

Theorem 3 gives a high-probability suboptimality bound for Algorithm 3 under specified choices of λ and β. The accompanying comparison explains how the bound and applicability differ between version-space and pointwise pessimism.

C.2.1 Analysis for Algorithm 3

  • Theorem 3 sets λ = log(|P|/δ) and β² = 2 log(|P|/δ), and states that Algorithm 3’s output policy satisfies its stated condition with probability at least 1 − δ.Source: §C.2.1 · p. 21
  • The suboptimality derivation carries the βΓ confidence penalty and KL-regularization terms through a decomposition of the policy gap.Source: §C.2.1 · pp. 21–22
  • Both bounds depend on distribution shift, but the version-space bound has a sharper term C; version-space pessimism avoids a pointwise uncertainty estimator, while pointwise pessimism may be easier to approximate heuristically.Theorem 3 also has an additional negative KL-divergence term between the Nash policy and the estimated policy.Source: §C.2.1 · p. 23

C.3 Analysis for Refined Coverage Condition

An improved analysis gives Algorithm 1 a refined coverage condition. Unlike Theorem 1’s worst-pair dependence, Theorem 4 adapts to an alternate π2 and trades coverage against suboptimality.

C.3 Analysis for Refined Coverage Condition

  • Theorem 4 is presented as an improved analysis of Algorithm 1 with a refined coverage condition.The section states that its analysis improves Algorithm 1’s coverage condition.Source: §C.3 · p. 23
  • Algorithm 3 is also stated to enjoy a similar bound and coverage condition.Source: §C.3 · p. 24
  • Theorem 4’s bound adapts to the best π2, trading dataset coverage against the suboptimality of that choice.When coverage is poor, the bound can use an alternate policy with a better coverage–suboptimality trade-off.Source: §C.3 · p. 24
  • Theorem 1’s bound depends on coverage of the worst unilateral policy pair, whereas Theorem 4 adapts to a better-covered choice.The earlier condition requires coverage of all unilateral pairs.Source: §C.3 · p. 24

D Proofs for the Online Setting

The online-setting proofs bound estimation error across rounds and connect the empirical bonus to an information-ratio bound. They also relate the Bradley–Terry uncertainty bonus to reward-based RLHF and describe an enhancer algorithm.

D Proofs for the Online Setting

  • With probability at least 1 − δ/2, the in-sample error estimate holds for every online round after a union bound over t.The proof begins its online analysis with this simultaneous in-sample guarantee.Source: D Proofs for the Online Setting · p. 24
  • Nonnegative error terms ensure at least one round has out-of-sample error no greater than the average across rounds.The proof selects such a round t0.Source: D Proofs for the Online Setting · p. 24

D.1 Uncertainty for the Bradley-Terry Model

  • The Bradley–Terry uncertainty bonus is consistent with the reward-based RLHF bonus up to factors involving regularization and the boundedness parameter.Source: §D.1 · p. 26

D.2 Guarantee for Enhancer

  • Algorithm 4 alternates main-agent estimation with enhancer exploration, then outputs the best main-agent policy selected by a validation set.The main agent computes an MLE and calls an oracle for a Nash equilibrium; the enhancer constructs a bonus and version space, then maximizes uncertainty.Source: §D.2 · p. 26
  • Lemma 3 assumes a uniform bound on each policy’s log probability ratio to the initial policy.Source: §D.2 · p. 27

E.1 Auxiliary Lemmas and Proofs

This section presents a unique-equilibrium lemma and supporting inequalities and optimization results, alongside checklist responses on proofs, reproducibility, limitations, and the applicability of IRB review.

E.1 Auxiliary Lemmas and Proofs

  • Lemma 4 asserts a unique Nash equilibrium for the max–min game; its proof invokes Proposition 1 in [46] and uses symmetry in a contradiction argument.Source: §E.1 · p. 28

E.2 Other Lemmas

  • The passage identifies Lemma 6 as martingale exponential inequalities for random functions ξ1(Z1), · · · , ξt(Zt) with filtration {Ft}, and Lemma 8 as multiplicative Chernoff bounds assuming X ∈ [0, 1] and EX = µ.Source: §E.2 · p. 29
  • It also states Sion’s minimax conditions and a policy-optimization error lemma for policies sharing reference support with finite, well-defined KL divergences, citing Xiong et al..Source: §E.2 · p. 29

NeurIPS Paper Checklist

  • The checklist reports that assumptions are stated for Theorems 1 and 2, complete proofs appear in Appendices C and D, and limitations are discussed.It points to lines 168–179 and 243–245 for the limitations discussion.Source: NeurIPS Paper Checklist · pp. 30–31
  • The authors report that experimental-reproduction details are in Appendix B.2, data details are online, and code has been uploaded.Source: NeurIPS Paper Checklist · pp. 31–32
  • The checklist reports no error bars, citing computational costs and carbon emissions, and marks societal impact NA because the theory paper has no visible short-term impact.Source: NeurIPS Paper Checklist · pp. 32–33
  • The paper marks IRB review as not applicable because it involved no crowdsourcing or research with human subjects.Source: NeurIPS Paper Checklist · p. 35
Loading 2402.07314v3…