Source-linked AI summary
Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, Jiantao Jiao
TL;DR
PPO-based LLM alignment faces reward-equivalence and token-wise optimization challenges. The paper proposes relative-feedback learning with P3O, a pairwise policy-gradient method, and reports stronger KL-Reward and GPT-4 results than prior methods. Its conclusion is that P3O combines policy-gradient benefits with simpler algorithm design and function approximation.
Problem
PPO can produce different outcomes for equivalent reward functions carrying identical preference information, while token-wise updates add function-approximation and algorithmic complexity.
Source: §1 · p. 2Method
The paper introduces Reinforcement Learning with Relative Feedback and P3O, which uses pairwise comparative rewards for trajectory-wise policy-gradient updates.
Source: §1 · p. 2Results
Takeaways & Limitations
P3O retains policy-gradient benefits while maintaining simplicity in algorithm design and function approximation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior involves Reinforcement Learning with Human Feedback (RLHF), with Proximal Policy Optimization (PPO) serving as the default RL optimizer. Despite its effectiveness, PPO has limitations when optimizing rewards trained from comparison-based loss. Primarily, PPO is not invariant to equivalent reward functions containing identical preference information due to the need to calibrate the reward scale. Additionally, PPO's necessity for token-wise updates introduces complexity in both function approximation and algorithm design compared to trajectory-wise optimization. This paper proposes a new framework, reinforcement learning with relative feedback, and a novel trajectory-wise policy gradient algorithm, Pairwise Proximal Policy Optimization (P3O) that operates directly on comparative rewards. We show theoretically that P3O is invariant to equivalent rewards and avoids the complexity of PPO. Empirical evaluations demonstrate that P3O outperforms PPO in the KL-Reward trade-off and can align with human preferences as well as or better than prior methods. In summary, this work introduces a simpler yet effective approach for aligning LLMs to human preferences through relative feedback.
1 INTRODUCTION
The introduction frames two challenges in PPO-based alignment: preference rewards can be equivalent despite different scales, and token-wise updates add complexity. It presents relative feedback and P3O as an approach to these issues, with reported gains over PPO and DPO.
1 INTRODUCTION
- PPO is not invariant to equivalent rewards, so reward functions with identical preference information can produce different optimization outcomes.The introduction notes that Bradley-Terry loss is invariant to constant shifts, whereas PPO is not.Source: §1 · pp. 1–2
- PPO’s token-wise updates add function-approximation and algorithmic complexity because they require a value function and GAE to redistribute sparse trajectory rewards.Human preference labels commonly compare whole responses, while the reward model returns a scalar after the response ends.Source: §1 · p. 2
- The paper defines reward equivalence and reports that Bradley-Terry loss is invariant to it while PPO is not.It also reports that online-DPO remains invariant, achieving higher reward but less KL efficiency than PPO.Source: §1 · p. 2
- P3O uses pairwise relative feedback for policy-gradient updates, avoiding a value function, GAE, and normalization techniques.The framework is designed to match reward models trained with comparative loss.Source: §1 · p. 2, Figure 1 · p. 2
- P3O is reported to outperform PPO and DPO on the KL-Reward trade-off and GPT-4 evaluation.The introduction presents P3O as a policy-gradient algorithm under Reinforcement Learning with Relative Feedback.Source: §1 · p. 2
2 RELATED WORK
Related work distinguishes offline and online approaches to LLM alignment, alongside preference-driven policy learning beyond language. The paper places its approach in online training and relates it to existing preference-based methods.
2 RELATED WORK
- Offline approaches include instruction fine-tuning, offline Q-learning, DPO, and reward-ranking methods such as PRO and RRHF.These methods use static data or preference rankings to update or align a language model.Source: §2 · pp. 2–3
- Online alignment methods generate responses from an updated policy, evaluate them with a reward model, and update the policy; RLHF commonly uses PPO or related methods.The passage also describes ReST, RAFT, and AI-feedback approaches as alternatives or neighboring paradigms.Source: §2 · p. 3
- Preference-driven policy learning outside language includes contextual dueling bandits and preference-based RL; P3O resembles pairwise policy gradients while adding clipping and KL control for language generation.The paper identifies Xu et al. (2020) as a related pairwise policy-gradient approach.Source: §2 · p. 3
3 PRELIMINARIES
The preliminaries describe the RLHF stages and the reward-model objective based on human comparisons. They contrast token-level MDP treatment of language generation with a contextual-bandit view that treats each full response as an action.
3 PRELIMINARIES
- RLHF comprises supervised fine-tuning, reward learning from ranked response pairs, and RL fine-tuning; the reward model estimates preferences using Bradley-Terry comparisons.The reward learner is fit from human choices between responses to the same prompt.Source: §3 · p. 3
- In RL fine-tuning, a KL penalty constrains deviation from the SFT model, and PPO optimizes the reward model score minus that penalty.The passage says the constraint is intended to prevent forgetting world knowledge acquired during pre-training.Source: §3 · p. 3
3.1 MDP VS CB
- In the MDP view, generated tokens are actions and the full response receives a scalar reward at episode end; PPO uses GAE to redistribute that sparse reward across tokens.An episode ends at the token limit or when the model produces <eos>.Source: §3.1 · p. 4
- In the contextual-bandit view, a prompt is context and the entire response is an action scored once, with no transitions; the paper adopts this formulation.Source: §3.1 · p. 4
4 ALGORITHM
The section derives P3O as a clipped, pairwise policy-gradient method and contrasts its value estimation with PPO and DPO. It then formalizes reward equivalence and shows that P3O and DPO, unlike PPO, are invariant to equivalent rewards.
4.1 PROXIMAL PAIRWISE POLICY OPTIMIZATION (P3O)
- P3O derives a symmetric Pairwise Policy Gradient from importance-sampled relative reward differences, then generalizes it to contextual bandits.Source: §4.1 · pp. 4–5
4.2 COMBINE WITH CLIPPING
- P3O adapts PPO clipping by constraining policy-ratio updates according to advantage and considers separate and joint clipping variants.Source: §4.2 · p. 5
4.3 RELATIONSHIP WITH PPO AND DPO
- Unlike PPO, P3O uses another sample to form an unbiased value estimator, avoiding a learned value function; the authors hypothesize DPO’s higher reward but weaker KL control reflects its different alignment objective.Source: §4.3 · pp. 5–6
4.4 REWARD EQUIVALENCE & INSTABILITY OF PPO
- Reward equivalence permits prompt-dependent reward offsets, while Bradley-Terry and Plackett-Luce losses remain identical across equivalent rewards.Source: §4.4 · p. 6
- Preference data identify only same-prompt reward differences; P3O and DPO updates are invariant to equivalent rewards, whereas PPO is not given the same value-function initialization.Source: §4.4 · p. 6
5 EXPERIMENTS
The experiments evaluate P3O on summarization and question-answering using KL–reward frontiers and head-to-head comparisons. P3O shows stronger KL–reward trade-offs and GPT-4 win rates than prior baselines in the reported comparisons.
5 EXPERIMENTS
- On TL;DR summarization and Anthropic HH question-answering, P3O-V1 and P3O-V2 are compared with PPO, DPO, and SFT; P3O achieves better KL–reward trade-off and generation quality, assessed by frontiers and head-to-head comparisons.Source: §5 · p. 7
5.1 KL-REWARD FRONTIER
- P3O-V2 maintains nearly the same highest TL;DR reward with better KL efficiency, while P3O variants dominate PPO and DPO frontiers, respectively, on HH at both model sizes.At the same reward on TL;DR, DPO has 25% higher KL-divergence than P3O-V2; P3O-V1 reaches slightly higher reward than P3O-V2 but has a worse trade-off.Source: Figure 2 · p. 7, §5.1 · p. 8, Figure 3 · p. 8
5.2 HEAD-TO-HEAD COMPARISONS
- In pairwise comparisons, P3O achieves GPT-4 win rates of 57.0% against PPO and 69.3% against SFT; DPO’s proxy-reward win rate over P3O is 49.5%, versus 45.4% under GPT-4.Source: §5.2 · p. 9
- DPO marginally exceeds P3O in average reward but has considerably higher KL-divergence, and its GPT-4 win rate against P3O is lower than its proxy-reward win rate.The reported rankings for average KL-divergence and reward are DPO > P3O > PPO > SFT.Source: §5.2 · p. 9
6 CONCLUSION & FUTURE WORKS
The conclusion presents relative-feedback reinforcement learning and P3O as a simpler policy-gradient approach for preference alignment. The pseudocode contrasts P3O's pairwise, trajectory-wise updates with the value- and advantage-estimation steps used by vanilla policy gradient.
6 CONCLUSION & FUTURE WORKS
- The paper introduces reinforcement learning with relative feedback and P3O, reporting better KL–reward frontiers and GPT-4 win rates than prior methods.It describes P3O as retaining policy-gradient advantages while simplifying algorithm design and function approximation.Source: §6 · p. 9
- Future work includes studying reward over-optimization, extending policy gradients to more than two ranked responses, and applying P3O beyond language-model alignment.Source: §6 · p. 9
A.1 PSEUDOCODES
- P3O collects pairwise trajectories and computes trajectory-wise rewards, avoiding value-function and GAE-based advantage estimation and introducing less policy-gradient estimation bias.The paper characterizes this approach as simpler to implement than vanilla policy gradient.Source: §A.1 · p. 12
- P3O collects two trajectories per prompt, computes trajectory-wise KL and preference rewards, and updates policy parameters using a clipped pairwise policy gradient.The pseudocode initializes policy parameters without initializing a value function.Source: §A.1 · p. 13
A.2 DERIVATION OF DPO
The appendix describes an online variant of DPO that generates preference labels from proxy rewards, then outlines a variance-reduction adjustment. It also derives a contextual-bandit policy-gradient expression by converting expectations to sums and using importance sampling.
A.2 DERIVATION OF DPO
- Online-DPO labels newly generated response pairs according to their proxy-reward difference and optimizes a DPO loss using responses stored in a replay buffer.The Bradley–Terry probabilities for the two possible preferences are σ(r1 − r2) and σ(r2 − r1).Source: §A.2 · p. 13
- The derivation reduces loss variance by incorporating the known preference-labeling probability instead of sampling labels.Source: §A.2 · p. 13
B.1 PROOF OF THEOREM 1
- The contextual-bandit VPG gradient derivation replaces prompt expectations with sums, applies importance sampling, and averages after swapping the two responses.The proof also notes that a subtracted term is zero because response probabilities sum to one.Source: §B.1 · p. 14
B.2 PROOF OF LEMMA 1
The proof shows that adding a prompt-dependent constant to a reward leaves the Bradley–Terry loss unchanged. It states that the same argument applies to the Plackett–Luce loss.
B.2 PROOF OF LEMMA 1
- A prompt-dependent reward shift cancels in the preferred-minus-less-preferred difference, leaving the Bradley–Terry loss unchanged.The proof states that the same argument applies to Plackett–Luce loss.Source: §B.2 · p. 14
B.3 PROOF OF LEMMA 2
The proof establishes that P3O and DPO updates are invariant to reward-equivalent functions, while PPO updates need not be. The setup also specifies how learning rates and response exposure were matched across methods.
B.3 PROOF OF LEMMA 2
- P3O updates depend on same-prompt reward differences, so adding a prompt-only offset leaves the update unchanged.Reward-equivalent functions differ by an offset δ(x) that depends only on the prompt.Source: §B.3 · p. 15
- DPO is invariant under reward equivalence because the equivalent rewards preserve the Bernoulli sampling distribution used in its gradient.Source: §B.3 · p. 15
- PPO is not invariant: its GAE-estimated advantage can change under equivalent rewards, producing different policy gradients.The proof notes that the advantage can be changed by choosing δ.Source: §B.3 · pp. 15–16
C.1 SETUP
- PPO’s tuned learning rate was reused for P3O and online-DPO, while PPO’s batch size was doubled to equalize the number of responses seen.P3O and online-DPO therefore saw half as many prompts.Source: §C.1 · p. 16
C.2 ABLATION STUDY
The ablations examine clipping and KL-control on HH, alongside additional checkpoint statistics and details of the GPT-4 evaluation prompt. Clipping improves the early KL-Reward frontier, while larger KL coefficients trade a slight frontier gain for a larger asymptotic reward decrease.
2 What is the effect of varying the KL-control coefficient on the KL-Reward frontier?
- Clipping improves the KL-Reward frontier, especially early in training and at larger learning rates, but slightly lowers asymptotic reward.Source: Figure 5 · p. 16, §2 · p. 17
- Larger KL coefficients slightly improve the KL-Reward frontier but more substantially decrease asymptotic reward; the authors suggest smaller coefficients may trade off better.Source: Figure 5 · p. 16, §2 · p. 17
C.3 ADDITIONAL STATISTICS
- P3O achieves nearly the same reward as DPO while incurring much less KL, and the methods differ in response length.PPO and SFT tend to generate longer responses, while P3O and DPO generate shorter ones.Source: §C.3 · p. 17
C.4 PROMPT
- GPT-4 response order was randomized to reduce order bias, and in about 3% of evaluations it judged neither response adequate.Source: §C.4 · p. 17
C.5 GPT-4 EVALUATION RESULTS
The GPT-4 evaluation examples favor P3O’s substantive responses over competitors’ less specific or less helpful replies. The examples span Bitcoin, city recommendations, and homemade soap.
C.5 GPT-4 EVALUATION RESULTS
- For the Bitcoin query, P3O provides an explanation, while the other methods’ responses are less substantive; the table describes P3O as outperforming them.Source: Table 2 · p. 18, §C.5 · p. 18
- For the city query, P3O suggests Austin, Dallas, or Phoenix in each comparison, while DPO is vague and PPO and SFT ask for more information.Source: Table 3 · p. 18
- For homemade soap, GPT-4 assesses P3O as most helpful; the other methods offer discouragement, no specific steps, or unclear guidance.Source: Table 4 · p. 19