PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

Shengyang Sun, Yian Zhang, Alexander Bukharin, David Mosallanezhad, Jiaqi Zeng, Soumye Singhal, Gerald Shen, Adithya Renduchintala, Tugrul Konuk, Yi Dong, Zhilin Wang, Dmitry Chichkov, Olivier Delalleau, Oleksii Kuchaiev

arXiv:2502.00203v2cs.LGcs.CL

TL;DR

Preference-optimization algorithms differ in objectives, response collection, and reward-model choices, making their relative effectiveness difficult to assess. RPO unifies these choices for ablation, and the reported outcomes distinguish offline and online methods by tuning and reward-model availability and quality.

  • Problem

    Preference-optimization methods vary across objectives, online or offline response collection, reward-model type, response count, and iteration, leaving their relative effectiveness and best combinations unclear.

    Source: §1 · p. 1

  • Method

    RPO unifies preference-optimization methods by matching implicit and target reward differences and varying design choices, while the paper proposes a testbed for clean ablations.

    Source: §3.1 · p. 2, §3.3 · p. 3, §7 · p. 8

  • Results

    Offline DPO, SimPO, and RPO-bwd perform similarly when well tuned and better than KTO and RPO-sqloo; online RPO-bwd improves over RLOO, while online-versus-offline performance depends on reward-model availability and quality.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The framework supports controlled comparisons of alignment choices and yields competitive new methods such as online RPO-bwd.

    Source: §7 · p. 8

    The authors call for evaluation in a wider variety of setups, including math and coding domains and alternative Ground-Truth Judges such as reward models or verifiers.

    Source: §7 · p. 8

Abstract

from arXiv · show

The rapid development of large language model (LLM) alignment algorithms has resulted in a complex and fragmented landscape, with limited clarity on the effectiveness of different methods and their inter-connections. This paper introduces Reward-Aware Preference Optimization (RPO), a mathematical framework that unifies popular preference optimization techniques in LLM alignment, including DPO, IPO, SimPO, and REINFORCE (LOO), among others. RPO provides a structured approach to disentangle and systematically study the impact of various design choices, such as the optimization objective, the number of responses per prompt, and the use of implicit versus explicit reward models, on LLM preference optimization. We additionally propose a new experimental setup that enables the clean and direct ablation of such design choices. Through an extensive series of ablation studies within the RPO framework, we gain insights into the critical factors shaping model alignment, offering practical guidance on the most effective strategies for improving LLM alignment.

1. Introduction

Preference-optimization methods vary in objectives, response collection, and reward-model design, leaving their relative effectiveness and best combinations unclear. The paper introduces RPO for controlled study of these choices and a synthetic evaluation setup that measures optimization against a Ground-Truth Judge.

1. Introduction

  • The paper’s questions include which offline objective performs best, whether responses should be collected offline or online, and which other design choices matter most.The introduction highlights implicit versus explicit reward models, responses per prompt, and multi-iteration alignment.Source: §1 · p. 1
  • RPO unifies popular preference-optimization methods and lets controlled experiments vary objectives, response counts, and online versus offline responses.The framework also supports combining effective design choices into new algorithms such as online RPO-bwd.Source: §1 · p. 1
  • Existing comparisons can obscure what alignment algorithms learn because training datasets and evaluation benchmarks may be only indirectly related.The paper motivates a synthetic setup where performance is evaluated against the Ground-Truth Judge’s preferences.Source: §1 · pp. 1–2
  • The authors conduct ablations across objectives, response collection, response counts, iterative alignment, and reward-model quality, then summarize findings as a cookbook.The ablations are conducted within the RPO framework.Source: §1 · p. 2

2. Background

The background describes preference optimization as the stage following supervised fine-tuning, in which models learn from human or AI feedback. It contrasts explicit reward-model training and policy optimization in RLHF with DPO’s implicit reward representation and differentiable objective.

2. Background

  • The notation defines x as a prompt, y as a response, πθ as the policy, and πref as the reference model, usually an SFT model and preference-stage initialization.The preference dataset uses chosen and rejected responses unless otherwise specified, and can extend to K responses per prompt.Source: §2 · p. 2
  • Preference optimization follows an SFT checkpoint and trains models toward responses preferred by humans, using approaches such as RLHF or DPO.The broader alignment pipeline pairs expert-demonstration SFT with preference optimization from human or AI feedback.Source: §2 · p. 2
  • RLHF first fits a reward model to preference data with the Bradley-Terry model, then optimizes the policy against its ratings with KL regularization.The cited policy-optimization methods include PPO, RLOO, and GRPO.Source: §2 · p. 2
  • DPO represents a reward model through the policy’s log-probability relative to a reference policy, up to a response-independent partition function.Optimizing this implicit reward with the Bradley-Terry model yields a differentiable objective.Source: §2 · p. 2

3. Reward-aware Preference Optimization Unpacks Preference Optimization Factors

RPO unifies alignment algorithms by varying reward targets, distance metrics, response counts, and online or offline training. These choices recover existing methods and extend the framework to multi-response and online settings.

3. Reward-aware Preference Optimization Unpacks Preference Optimization Factors

  • RPO is a flexible framework for disentangling the effects of implicit versus explicit reward models, online versus offline training, and paired versus multiple responses.Its central objective is to align implicit-reward predictions with a target explicit reward model using a chosen distance metric.Source: §3 · p. 2, §3.1 · p. 2

3.1. Reward-aware Preference Optimization

  • RPO minimizes the distance between implicit and target reward predictions rather than maximizing a preferred-versus-rejected reward difference.The preference pair is the input; η controls reward scale and β controls regularization, under the assumption that the target reward model reflects human preference.Source: §3.1 · p. 2

3.2. Distance metric

3.3. Number of Responses Per Prompt

  • For K responses per prompt, RPO extends the reward-distance objective to K-dimensional rewards, with squared-distance and backward-KL multi-response variants.The log partition function cancels, making the multi-response objective computable; the SimPO-associated πref is not a mathematically valid probability distribution.Source: §3.3 · p. 3

3.4. Online Responses

  • Online RPO samples K responses at each training step and optimizes a differentiable objective whose gradient resembles REINFORCE.With squared distance and Leave-One-Out, RPO recovers RLOO; backward categorical KL instead uses a softmax-probability difference as the score-function scale, normalizing over rewards.Source: §3.4 · pp. 3–4

4. The Experiment Setup

The experiment setup isolates the path from a formalized goal to aligned models, using a Ground-Truth Judge to compare algorithms’ optimization of its preferences. In the prompt-distribution experiment, reward and win-rate trends were clearer than MT-Bench, and in-distribution prompts performed better than synthetic prompts.

4. The Experiment Setup

  • The setup isolates alignment algorithms’ effects by comparing how well they optimize a Formalized Goal, represented by a synthetic Ground-Truth Judge.The judge is Nemotron-4-340B-RM; the experiment starts from an SFT checkpoint and uses RM-annotated preference pairs from 120k LMSYS prompts.Source: §4 · p. 4, §4.1 · pp. 4–5

4.1. Experiment Setup

  • Evaluation uses the Ground-Truth Judge’s predicted reward on validation, in-distribution test, and out-of-distribution prompts, reporting average reward and win-rate against the SFT checkpoint.The prompt-distribution experiment compares preference datasets generated from LMSYS and synthetic prompts using the same Llama 3 8B SFT model.Source: §4.1 · p. 5, §4.2 · p. 5

4.2. The impact of prompt distributions

  • Average reward and win-rate rise during training while MT-Bench fluctuates; training on in-distribution LMSYS prompts achieves higher reward and win-rate than synthetic prompts.Figure 1 reports 95% confidence intervals over three independent runs; its MT-Bench measure has large variance and hardly shows learning.Source: Figure 1 · p. 5, §4.2 · p. 5

5. Related Works

Prior alignment work spans online RLHF and offline preference objectives, while studies often evaluate methods through datasets and benchmarks with an indirect relationship. This paper extends RPO as a framework connecting online and offline methods and their design choices.

5. Related Works

  • Prior alignment methods span PPO-based RLHF, critic-free RLOO and GRPO, and offline DPO-family objectives, alongside many later objective variants.The cited work describes RLOO and GRPO as forgoing PPO's critic, and DPO as directly optimizing the policy from offline data.Source: §5 · p. 5
  • This work extends NVIDIA et al. (2024)'s RPO, which focused on backward KL, paired responses, and offline training, into a roadmap connecting online and offline preference optimization algorithms.The paper describes its extension as covering different design choices.Source: §5 · p. 5
  • Studies often train on existing datasets and evaluate on existing benchmarks, whose indirect relationship can generate misleading conclusions; other work examines online–offline gaps and implicit-reward-model generalization.The passage notes that UNA approximates an explicit reward model with an implicit one but does not disentangle design factors.Source: §5 · pp. 5–6
  • Table 2 covers two model sizes, multiple objectives and response counts, online and offline training, and Ground-Truth or Learnt reward models; 70b results are single-run due to compute limits.The table caption also identifies online RPO-sqloo as equivalent to REINFORCE Leave-One-Out.Source: Table 2 · p. 6

6. Ablation Results

The ablations compare objectives, response counts, training modes, iterative alignment, and reward-model quality. Iterative training improves results, while online RPO-bwd performs strongly against RLOO and online training depends on reward-model quality.

Objectives

  • With K = 2, DPO, RPO-bwd, and SimPO perform better for both model sizes, while KTO and RPO-sqloo have consistently worse rewards and win-rates.Source: §6 · pp. 6–7

Number of Responses

  • Increasing the response count from K = 2 to K = 4 brings no significant improvement for either distance metric in this setup.The comparison keeps model initialization, prompts, and evaluation the same.Source: §6 · p. 7

Comparing Online Algorithms

  • 5.916 vs 5.796 lmsys(test) average reward and 85.7% vs 78.5% win rate favor online RPO-bwd over online RPO-sqloo for 70b.The lmsys(valid) curves show faster reward increases and slower KL growth for RPO-bwd; RPO-sqloo crashes mid-training while RPO-bwd remains stable in all runs.Source: §6 · p. 7, Figure 2 · p. 7

Online vs Offline Training

  • 85.5% vs 81.0% alpacaeval win rate favors online over offline 70b RPO-bwd, and online training outperforms offline training across the tested settings.The online setting assumes access to the Ground-Truth Judge; with only human-annotated preference data, online training is not necessarily better because a reward model must be trained.Source: §6 · p. 7

Iterative (Online / Offline) Alignment

  • 6.157 average OOD reward and 93.8% win rate for online iter3 beat every Table 2 row, while performance improves with more iterations for both online and offline training.The strongest model is online iter3 with the 70b model.Source: §6 · p. 7, Figure 3 · p. 7

The Importance of RM Quality

  • With a Learnt RM, online methods barely improve over the SFT model, and its rewards rise while Ground-Truth RM rewards fall, indicating reward hacking.The authors conclude that a strong reward model is critical to online methods and that naively training one on the preference dataset is likely insufficient.Source: §6 · p. 8, Figure 4 · p. 8

7. Conclusion

The conclusion presents RPO as a framework for unifying alignment algorithms and a synthetic testbed for comparing them. Its findings distinguish effective strategies by training setup and data source, while identifying areas for further study.

7. Conclusion

  • RPO unifies methods including DPO, IPO, and RLOO, and the proposed experimental setup offers a better testbed for ablating alignment algorithms than popular benchmarks such as MT-bench.Source: §7 · p. 8
  • Offline DPO, SimPO, and RPO-bwd perform similarly when well tuned and better than KTO and RPO-sqloo.The conclusion also recommends offline RPO-bwd when sufficient human labeling resources are available.Source: §7 · p. 8
  • The number of responses in offline preference optimization does not significantly affect model performance.Source: §7 · p. 8
  • Online RPO-bwd improves over online RPO-sqloo (RLOO) with more stable training, better KL regularization, and higher rewards.The conclusion recommends online RPO-bwd when a strong reward model or ground-truth verifier is available.Source: §7 · p. 8
  • Iterative alignment consistently improves performance for both offline and online methods.Source: §7 · p. 8
  • When a reward model or verifier annotates preferences, online methods outperform offline methods; without one, online performance depends critically on learned reward-model quality and offline methods can be competitive.Source: §7 · p. 8
  • RPO is sequence-level and lacks token-level credit assignment like PPO; the best recipe for training strong reward models remains an open question.Source: §7 · p. 8
  • The authors call for further study of preference-data generation recipes and experiments across more domains and ground-truth judges.The current setup uses random prompts and responses generated by the current policy, with preferences built using the Ground-Truth Judge.Source: §7 · p. 8

Impact Statement

The impact statement describes the paper as advancing understanding of preference optimization algorithms for model alignment. It links algorithm development and alignment with potential benefits and emphasizes caution about misalignment risks.

Impact Statement

  • The paper presents its work as advancing understanding of preference optimization algorithms in model alignment.Source: Impact Statement · p. 9
  • The statement connects improved algorithms and alignment with human values to potential benefits for society and AI safety.Source: Impact Statement · p. 9
  • It also urges caution in adopting algorithms to mitigate the risk of misalignment in strong AI models.Source: Impact Statement · p. 9

A. Derivation of the distance functions for the multi-response scenario

The appendix describes distance functions for multi-response RPO, including a leave-one-out squared distance and categorical KL metrics. It notes that leave-one-out subtraction or softmax normalization cancels the log partition function for the first two metrics.

A. Derivation of the distance functions for the multi-response scenario

B. Gradient derivation of online RPO with the Backward KL Divergence (rpo-bwd) distance.

The section derives how backward-KL RPO changes the online REINFORCE gradient scale and establishes a Bernoulli backward-KL connection to DPO. It also describes offline, online, and iterative RPO procedures and reports iterative-alignment comparisons.

B. Gradient derivation of online RPO with the Backward KL Divergence (rpo-bwd) distance.

  • With backward KL, online RPO replaces the REINFORCE gradient scale with the difference between softmax probabilities of ground-truth and predicted rewards.The softmax logits are based respectively on ground-truth rewards and β log policy-to-reference likelihood ratios.Source: B. Gradient derivation of online RPO… · p. 11

C. Connection of Bernoulli Backward KL Divergence to DPO

D. Offline, Online, and Iterative Algorithms.

  • Offline RPO optimizes preferences and precomputed rewards, whereas online RPO samples responses from the current policy and computes rewards online.The online procedure is described as more like RLHF, except it does not train the policy with reinforcement learning.Source: D. Offline, Online, and Iterative… · p. 12
  • Iterative offline RPO resamples responses from the current policy between offline rounds, reducing computational cost and making reward hacking less likely than online RPO.The passage says iterations are usually fewer than 10 and responses can be generated offline in parallel.Source: D. Offline, Online, and Iterative… · p. 13

E.1. The impact of prompt distributions (Section 4.2)

  • Table 3 reports average reward and win rate across LMSYS validation, LMSYS test, and AlpacaEval prompts for training on LMSYS versus synthetic prompts.The passage identifies the evaluations and metrics, but supplies no table values here.Source: §E.1 · p. 13

E.2. Iterative (Offline / Online) Alignment (Section 6)

  • The 8b iterative DPO setting shows similar learnings to the iterative-alignment results.The supplied passage does not specify which learnings are similar.Source: §E.2 · p. 14

F. Experimental Details

The experimental details specify checkpoint selection and training configurations for offline and online methods, as well as reward-model training and the iterative-alignment data construction.

Evaluations

  • The best checkpoint is selected by validation average reward, and repeated 8b runs report average validation and test metrics with 95% confidence intervals.Training is repeated with randomly shuffled data; this repetition is limited to the 8b setting for computational-cost reasons.Source: F. Experimental Details · p. 14

Offline preference optimization methods

  • Offline RPO uses Adam at a constant 5e-7 learning rate, batch size 256, and one epoch; checkpoints are selected by validation average reward.This configuration applies to offline RPO jobs across 8b or 70b models and K=2 or K=4.Source: F. Experimental Details · p. 14

Online preference optimization methods

  • Online 70b RPO uses Adam at 5e-7 with global batch size 64 and runs for up to 250 iterations because of computational limitations.The passage reports multiple RPO-sqloo crashes, while all RPO-bwd runs stably increased.Source: F. Experimental Details · p. 14

Reported evaluation tables

  • Table 3 reports average reward and win rate across three prompt evaluations, while Table 4 reports those metrics for iterative DPO and offline and online RPO-bwd.The captions identify LMSYS validation, LMSYS test, and AlpacaEval as evaluation prompt sets.Source: Table 3 · p. 15, Table 4 · p. 15

Training the Learnt RM

  • Reward models are trained from SFT checkpoints for one epoch, using Bradley-Terry loss for 8b and regression for 70b.The 8b reward model uses learning rate 3e-6 and batch size 512; the 70b model uses 1e-6 and batch size 256.Source: F. Experimental Details · pp. 14–15

Iterative alignment

  • Iterative alignment uses 120k prompts in iteration 1, then builds later datasets from newly collected LMSYS prompts and samples from earlier preference data.Iteration 2 draws 100k of 150k new prompts; iteration 3 mixes the remaining 50k with 50k prompts from iteration 1.Source: F. Experimental Details · p. 15
Loading 2502.00203v2…