A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization

1 Introduction

• Traditional RLHF converts pairwise preferences into scalar rewards, creating calibration challenges across diverse prompts and responses.Reward-model initialization from generative models also mismatches their discriminative ranking task, potentially propagating errors into reinforcement learning.Source: §1 · pp. 1–2
• Pairwise-RL unifies reward-model training and reinforcement learning within a consistent pairwise paradigm.It uses generative modeling to improve reward-model calibration and performance.Source: §1 · p. 2
• Pairwise-RL introduces a pairwise PPO algorithm that operates directly on pairwise comparisons instead of scalar reward approximations.This design targets the mismatch between pairwise human-feedback data and scalar reinforcement-learning signals.Source: §1 · pp. 1–2
• Pairwise-RL uses generative modeling techniques to enhance reward-model calibration and performance during RLHF training.The approach is intended to improve consistency when scoring responses across contexts.Source: §1 · p. 2
• Pairwise-RL outperforms traditional RLHF methods on internal datasets and public benchmarks, improving alignment with human preferences and model behavior across tasks.The reported evaluation supports gains beyond a single dataset or task.Source: §1 · p. 2

2 Preliminary

• RLHF maximizes expected reward from a reward model, with the reward model commonly fit to pairwise human-preference comparisons.The framework typically uses Bradley-Terry modeling to assign scalar rewards to responses.Source: §2 · p. 2
• The RLHF objective includes KL regularization against an initialization policy, controlled by coefficient β.The total decision horizon is T, and token-level rewards are supplied by the reward function.Source: §2 · p. 2
• PPO stabilizes RLHF policy optimization by clipping updates relative to the previous policy.Its surrogate objective uses a probability ratio, an estimated advantage, and a clipping-range hyperparameter ϵ.Source: §2 · p. 2
• GAE computes advantage estimates through multi-step bootstrapping to reduce variance.The formulation uses the discount factor γ, GAE parameter λ, temporal-difference errors, and a value function; RLHF commonly sets γ = 1.0.Source: §2 · p. 3

3.1 Pairwise reward models

• Pairwise generative reward models evaluate response pairs in shared context, either producing direct preference decisions or using intermediate reasoning.Chain-of-thought methods can improve human-like reasoning but introduce stochastic variability, motivating voting-based aggregation.Source: §3.1 · p. 3
• CoT-based reward-model methods generate rationales before final judgments, while voting aggregates multiple judgments to stabilize pairwise comparisons.The aggregation is intended to account for uncertainty in intermediate reasoning steps.Source: §3.1 · p. 3

3.2 Pairwise reinforcement learning methods

• DPO and P3O apply pairwise ideas to policy optimization, but their trajectory-wise or contextual-bandit formulations produce sparse rewards and weaker token-level credit assignment.The related methods also have relatively low sample efficiency compared with approaches treating individual tokens as actors.Source: §3.2 · p. 3
• Preference-based reward shaping and robust RLHF introduce centered or contrastive rewards using sigmoid transformations or offline baseline responses.These methods use pairwise rewards or baselines to mitigate reward-hacking and reward-noise concerns.Source: §3.2 · p. 3

4.1 Pair-wise Reward Model

• The pairwise reward model jointly evaluates a generated response and its anchor, reducing dependence on absolute reward calibration under distribution shifts.Its relative judgment is intended to reduce sensitivity to score magnitudes that may fail across changing data distributions.Source: §4.1 · pp. 3–4
• Comparison instances are formatted as natural-language yes/no questions, allowing the generative model to perform discriminative preference judgments using pretrained contextual representations.The model is trained with vocabulary-level softmax cross-entropy.Source: §4.1 · p. 4
• Swapped-response augmentation and an MSE consistency constraint reduce positional bias by encouraging symmetric judgments when comparison order is reversed.The total objective combines classification accuracy with positional consistency, with ζ controlling their trade-off.Source: §4.1 · p. 4

4.2 Pairwise Proximal Policy Optimization

• The reward model averages two opposite-direction yes/no probabilities, reducing yes/no bias and variance while yielding a 0.5 win rate for identical responses.The two evaluations act like an ensemble and cancel positional artifacts when the responses match.Source: §4.2 · pp. 4–5
• Pairwise-RL trains the policy to maximize a generated response’s win probability over a ground-truth anchor instead of an absolute scalar reward.This reframes the RL objective around pairwise preference comparisons.Source: §4.2 · p. 5
• Sigmoid saturation compresses large reward differences, downweights already-superior responses, and keeps learning sensitive when generated and anchor responses are closely matched.Clipping large advantages also controls the policy update’s KL-divergence upper bound on these samples.Source: §4.2 · p. 5
• Including the ground-truth anchor in value estimation aligns the value model with the reward model but makes its evaluated state inconsistent with the policy’s state.Excluding the anchor preserves the policy state but creates information asymmetry, motivating pointwise reward distillation.Source: §4.2 · pp. 5–6
• Pairwise PPO uses the generative reward for policy advantages but distills a pointwise reward model for value returns, addressing the mismatch between anchor-aware rewards and anchor-free policy states.The distillation target matches the generative pairwise score to the difference between pointwise scores for the sampled response and anchor.Source: §4.2 · p. 6

5.1 Setup

• The evaluation compares pairwise and pointwise reward models, Pairwise PPO and standard PPO, and uses smaller models for ablations before larger end-to-end comparisons.Datasets cover seven internal capabilities and external benchmarks including MMLU pro, CEval, KORBench, BBH, MATH, LiveCodeBench, IFEval, and GPQA.Source: §5.1 · p. 6

5.2 Overall Experimental Results

• Pairwise reward modeling outperforms traditional pointwise reward modeling on both in-domain IID and out-of-distribution datasets, while remaining comparable on Reward Bench.Average front-and-back accuracy also exceeds the individual front and back accuracies, suggesting reduced positional bias and more robust relative judgments.Source: §5.2 · p. 7, Table 1 · p. 7
• Pairwise PPO improves mainly on reasoning, instruction following, NLP, and long-text tasks internally, with larger external gains on KOR, IFEval, and GPQA.Reasoning gains are attributed primarily to reduced reward hacking, while instruction-following and NLP gains stem mainly from more accurate pairwise reward-model judgments.Source: §5.2 · p. 7, Table 3 · p. 7

5.3.1 The influence of the use of ground truth on value estimation

• Value w/o GT performs significantly worse than Value w/ GT in pairwise PPO value estimation.The paper attributes this gap to imperfect distillation: a pairwise reward model directly compares both responses, whereas a pointwise value model scores them separately.Source: §5.3.1 · p. 8, Table 4 · p. 8

5.4 The influence of the quality and stability of the ground truth

• Ground-truth responses are obtained with best-of-n sampling from the supervised-fine-tuned model, avoiding annotation intervention while matching the online response distribution.Higher-quality ground truth provides more optimization space, and matching distributions is intended to improve pairwise reward-model scoring robustness.Source: §5.4 · p. 8
• Final-model performance rises from worst of 5 through best of 30, then declines for best of 40 and best of 50 as pointwise reward-model selection becomes less reliable.Best of 1 can underperform worst of 5 because its ground-truth quality varies more across prompts, destabilizing the pairwise reward-model scoring meaning.Source: §5.4 · p. 8, Table 5 · p. 9, Figure 1 · p. 9

• Pairwise-RL outperforms conventional RLHF methods on internal and external benchmarks, improving alignment and model behavior across reasoning, instruction-following, and long-text tasks.Source: §6 · p. 9
• The framework combines a pairwise reward model with pairwise PPO, maximizing generated responses’ win probability over ground-truth anchors.Source: §6 · p. 9
• Pairwise-RL addresses scalar-reward calibration challenges and the mismatch between generative initialization and discriminative reward-modeling tasks.Source: §6 · p. 9

• The derivation uses a second-order KL expansion around θold and identifies the Fisher matrix as the covariance of the policy gradient.Source: §7 · p. 11
• The policy gradient is expressed through the advantage function and bounded using the Cauchy–Schwarz inequality.Source: §7 · p. 11
• Applying positive-semidefinite matrix properties bounds the quadratic gradient form by the Fisher trace times the gradient norm squared.Source: §7 · p. 11
• Reducing excessively large advantages lowers the upper bound on policy-update KL divergence by decreasing the expected squared advantage term.Source: §7 · p. 11
• The KL-divergence bound depends on the step size, expected squared advantage, and the trace of the Fisher information matrix.Source: §7 · p. 11