PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Trust Region in LLM Reinforcement Learning

Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee

arXiv:2602.04879v3cs.LGcs.AIcs.CL

TL;DR

PPO’s sampled-token ratio clipping can over-constrain low-probability updates while under-constraining shifts in high-probability tokens. DPPO directly constrains estimated policy divergence using efficient Binary and Top-K approximations, achieving superior stability and efficiency across five large-scale experiments.

  • Problem

    PPO’s single-sample ratio clipping is poorly matched to LLMs’ large, long-tailed vocabularies, distorting trust-region control across tokens.

  • Method

    DPPO replaces ratio clipping with direct policy-divergence constraints, using Binary and Top-K approximations to reduce the overhead of distribution-wide estimation.

  • Results

    DPPO consistently achieves superior stability and efficiency across five large-scale experiments, optimizing rewards faster and reaching better converged performance than GRPO-ClipHigher.

  • Takeaways & Limitations

    DPPO provides a more robust trust-region framework for reinforcement-learning-based LLM fine-tuning.

  • Takeaways & Limitations

    PPO can heavily clip low-probability-token updates despite their small total-variation contribution while allowing substantial shifts in dominant tokens within the clipping range.

Abstract

from arXiv · show

Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inherent to LLMs. PPO constrains policy updates based on the probability ratio of sampled tokens, which serves as a noisy single-sample Monte Carlo estimate of the true policy divergence. This creates a sub-optimal learning dynamic: updates to low-probability tokens are aggressively over-penalized, while potentially catastrophic shifts in high-probability tokens are under-constrained, leading to training inefficiency and instability. To address this, we propose Divergence Proximal Policy Optimization (DPPO), which substitutes heuristic clipping with a more principled constraint based on a direct estimate of policy divergence (e.g., Total Variation or KL). To avoid huge memory footprint, we introduce the efficient Binary and Top-K approximations to capture the essential divergence with negligible overhead. Extensive empirical evaluations demonstrate that DPPO achieves superior training stability and efficiency compared to existing methods, offering a more robust foundation for RL-based LLM fine-tuning. Our code is available at https://github.com/sail-sg/Stable-RL.

1. Introduction

The introduction argues that PPO’s sampled-token ratio clipping is poorly matched to LLM vocabularies, causing inefficient and unstable updates. It proposes DPPO, which constrains directly estimated policy divergence with efficient approximations and develops a domain-specific theoretical foundation.

  • Motivation: LLM reinforcement learning often involves training-inference distribution mismatch and multiple minibatch updates, making control of policy movement from the data-generating policy essential.Rollouts may be sampled by an inference engine while gradients are computed by a trainer engine.
  • PPO limitation: PPO is widely used because it is simple and scalable, but its ratio clipping constrains only the sampled token’s probability ratio rather than the full policy divergence.The ratio is intended to keep updates inside a trust region associated with theoretical monotonic improvement guarantees.
  • DPPO: DPPO replaces ratio clipping with a constraint based on directly estimated policy divergence, including Total Variation or KL divergence.This avoids relying on noisy single-sample ratios as proxies for the true divergence.
  • PPO limitation: PPO over-penalizes low-probability-token updates and under-penalizes high-probability-token updates, respectively slowing learning and risking instability.The probability ratio is highly volatile for low-probability tokens, whereas token-level TV divergence is more stable.
  • Contributions: Binary and Top-K divergence approximations capture essential distributional shifts with negligible overhead, while the paper derives policy improvement bounds for finite-horizon, undiscounted LLM generation.The contributions also analyze primary sources of training instability to provide practical stabilization guidance.

2. Background

The paper grounds its analysis in standard reinforcement-learning theory: policies generate trajectories in an MDP and seek to maximize discounted return. Policy-performance bounds then motivate trust-region optimization, where divergence-constrained surrogate updates provide stable, monotonic improvement guarantees.

  • MDP formulation: An MDP is defined by state and action spaces, transition dynamics, rewards, an initial-state distribution, and discount factor γ.A stochastic policy samples actions and generates state-reward trajectories through the transition dynamics.
  • RL objective: The central RL objective is to find a policy maximizing expected discounted return.The policy induces trajectories by sampling actions and transitioning between states.
  • Policy evaluation: Value and advantage functions characterize policy outcomes, while the performance difference theorem relates the returns of target policy π and behavior policy µ.The advantage is defined as A^π(s, a) = Q^π(s, a) − V^π(s).
  • Policy improvement: A policy update with non-negative expected advantage guarantees monotonic performance improvement, η(π) ≥ η(µ).This result uses the normalized discounted state-visitation distribution induced by the target policy.
  • Trust-region optimization: Trust-region methods replace the intractable new-policy visitation distribution with a behavior-policy surrogate penalized by policy divergence.The surrogate is a tight lower bound at π = µ, so iteratively maximizing it follows Minorize-Maximization and guarantees monotonic improvement.
  • Trust-region optimization: The trust-region constraint may use KL divergence, whose relationship to Total Variation is justified by Pinsker’s inequality.Maximizing the surrogate within a small trust region provides stable and meaningful policy improvement.

3. Trust Region Under LLM Regime

This section adapts trust-region analysis to undiscounted, finite-horizon LLM fine-tuning with sequence-level advantages and derives performance-difference and policy-improvement results. The resulting bounds use maximum or average token-level TV divergence, with the average form tighter for long responses and aligned with per-token divergence control.

  • LLM Regime: LLM fine-tuning is modeled as an undiscounted (γ = 1), finite-horizon T episodic task with sparse rewards and often sequence-level advantage estimates.The classical discounted bound becomes ill-defined because its 1/(1−γ) term diverges to infinity.
  • LLM Regime: A response y is generated token by token from conditional distributions π(yt|st), receives a scalar reward R(y, x), and defines the objective through the complete-response probability.The response probability is the product of the conditional token probabilities.
  • Performance Difference Identity: Theorem 3.1 decomposes the performance difference between policies π and µ into a surrogate objective and an error term.The surrogate is a first-order approximation, while the error captures higher-order effects of the policy change.
  • Policy Improvement Bound: Theorem 3.2 lower-bounds policy improvement using either maximum statewise TV divergence or average token-level TV divergence.The bounds apply in finite horizon T with γ = 1 and define ξ as the maximum absolute reward.
  • Policy Improvement Bound: The average-divergence bound is tighter for long LLM responses and more directly matches the per-token divergence control used in PPO and DPPO.The constraint can also be applied using KL divergence.

4. Methodology

The methodology identifies PPO’s single-sample ratio clipping as a poor proxy for distributional trust-region constraints in LLMs and introduces DPPO, which directly constrains policy divergence. Binary and Top-K approximations make divergence-based updates practical for large-vocabulary policies.

  • PPO Limitation: PPO constrains a noisy single-sample estimate rather than the true TV divergence, causing unreliable trust-region enforcement for LLM vocabulary distributions.The clipping rule stops incentives outside [1 − ϵ, 1 + ϵ], but its estimate depends on the sampled token.
  • PPO Limitation: Low-probability tokens can be aggressively clipped despite negligible mass changes, while high-probability tokens can remain unclipped despite substantial TV-divergence contributions.The examples use rlow = 100 for a low-probability token and rhigh = 0.80/0.99 ≈ 0.808 for a dominant token losing 0.19 probability mass.
  • DPPO: DPPO replaces ratio clipping with a divergence-based mask that blocks updates when the full policy shift exceeds threshold δ and the update moves farther from the trusted region.The divergence D may be TV or KL, and replacing D with |rt − 1| recovers PPO.
  • Divergence Approximations: The binary approximation collapses the sampled token and all other tokens into a Bernoulli distribution, computing divergence with negligible overhead while tracking absolute probability-mass shifts.It is designed to distinguish large from small shifts in absolute probability mass, addressing PPO’s primary failure mode.
  • Divergence Approximations: The Top-K approximation retains the K highest-probability behavior-policy tokens plus the sampled token, aggregating the remainder into an “other” category.This reduced distribution better captures head changes that typically dominate divergence while keeping overhead minimal.

5. Analysis on Training Stability

The analysis shows that trust regions remain essential for stable LLM RL training, must be anchored to the original rollout policy, and should target rare, high-divergence updates on negative samples. It also finds that truncated importance sampling can worsen stability by biasing away low-probability-token gradients.

  • Trust-region anchoring: Anchoring the trust region to the original behavior policy µθ′ is necessary, while using recomputed πθ′ causes instability.Avoiding recomputation also reduces training costs by approximately 25%.
  • Instability sources: A minimal mask with δ = 0.5 stabilizes training by blocking large-divergence updates on negative samples, whereas δ = 0.8 and recomputed anchoring fail.The masked updates are defined as bad when divergence exceeds 0.5.
  • Instability sources: Only ≤0.5% of updates are bad, yet this small subset of negative-sample updates that push the policy far outside the trust region primarily drives instability.Aggressively penalizing tokens the model considers probable may corrupt internal knowledge and destabilize learning.
  • Truncated importance sampling: Truncated importance sampling worsens stability: PG-TIS and MiniRL-TIS collapse prematurely and underperform their untruncated counterparts.The experiments reveal an unexpected drawback despite TIS being intended to reduce policy-gradient variance.
  • Truncated importance sampling: TIS may harm training because truncating high-variance ratios for low-probability tokens down-weights their gradients, introducing harmful bias like naive clipping.The analysis suggests naive truncation can be as damaging as naive clipping.

6. Analysis on Training Efficiency

The analysis examines how low-probability tokens and directional trust-region relaxation affect training efficiency. Experiments show that relaxing clipping can accelerate learning, but the relaxation direction creates a stability–efficiency tradeoff tied to entropy behavior.

  • Low-probability tokens: PPO ratio clipping may over-constrain low-probability-token updates that a divergence-based trust region would permit.The section analyzes low-probability tokens to motivate DPPO’s divergence-based trust region.
  • Experimental setting: Experiments fine-tune Qwen3-1.7B-Base on DAPO with GRPO and Clip-Higher, relaxing clipping for tokens with µ(y_t|s_t) < α by setting ϵ=∞.This isolates the effect of low-probability tokens.
  • Directional relaxation: Directional relaxation uses α=0.1 with asymmetric thresholds ϵ_low=0.2 and ϵ_high=0.28, relaxing the high end, low end, or both.Relax-high sets ϵ_high=∞ for low-probability tokens while retaining ϵ_low=0.2.
  • Directional relaxation: Relax-high maintains high entropy but produces no significant efficiency gains, whereas Relax-low learns substantially faster initially before eventually dropping due to entropy collapse.The results establish that relaxation direction critically affects both efficiency and stability.

7. Broader Evaluation

Large-scale evaluations across five model configurations show that DPPO improves training stability and efficiency, outperforming baselines in verifiable-reward and noisy learned-reward settings. Its advantages extend across MoE, dense, Llama-family, and general reasoning experiments.

  • Large-scale evaluation: Experiments train on approximately 13k filtered DAPO-Math samples across five model configurations, spanning MoE, dense, router-replay, and LoRA settings.The configurations include Qwen3-30B-A3B-Base, its R3 variant, Qwen3-30B-A3B, Qwen3-8B-Base, and Qwen3-30B-A3B with LoRA.
  • Large-scale evaluation: DPPO consistently delivers superior stability and efficiency across all five large-scale experiments, optimizing rewards faster and achieving better converged performance than GRPO-ClipHigher.Baseline methods frequently show training instability or catastrophic collapse in the reported experiments.
  • Rollout router replay: Without R3, DPPO variants consistently outperform R3-enhanced baselines, while adding R3 provides further gains for DPPO.These results indicate that DPPO supplies inherent stability while benefiting additionally from rollout router replay.
  • RLHF evaluation: In RLHF with Skywork-Reward-Llama-3.1-8B, DPPO-Binary-TV improves learned rewards, demonstrating effectiveness for noisy learned-reward optimization.The comparison uses Gemma-2-9B-It with UltraFeedback and Qwen3-4B-Instruct-2507 with HH-RLHF.
  • Broader applicability: Across Llama-family models and general reasoning tasks, DPPO outperforms the baseline across most settings, highlighting broad applicability.These results are presented in Appendix F.5.

8. Related Work

Prior work links PPO’s ratio clipping to asymmetric treatment of token probabilities in LLMs, motivating DPPO’s direct divergence constraint and binary approximation. Related research instead addresses training–inference mismatch through importance sampling, higher precision, or engineering alignment, each with efficiency or applicability limitations.

  • Divergence-based trust regions: PPO’s ratio clipping over-penalizes low-probability tokens and under-penalizes high-probability tokens, impairing LLM training efficiency and stability.DPPO addresses this structural flaw by directly constraining policy divergence.
  • Divergence-based trust regions: Wang et al. proposed adaptive KL-divergence clipping for similar exploration issues in traditional reinforcement learning.DPPO follows the related-work insight while targeting the LLM setting.
  • Divergence-based trust regions: Because exact divergence is prohibitive for LLMs’ huge memory footprint, DPPO introduces a binary approximation that empirically captures most benefits.The paper refers to Figure 11 for this empirical finding.
  • Training–inference mismatch: Training–inference mismatch arises when πθ differs from µθ because of numerical precision errors and implementation differences.Recent work identifies this mismatch as a key source of training instability.
  • Training–inference mismatch: Importance-sampling methods such as TIS and Masked Importance Sampling correct biased gradients at token and sequence levels but may not balance training efficiency satisfactorily.The cited methods include Yao et al. (2025), Zheng et al. (2025), Liu et al. (2025b), and Team et al. (2025b).
  • Training–inference mismatch: Higher precision and rigorous engineering alignment address mismatch but have limited applicability, requiring specialized training engines or model architectures in some cases.These approaches are also constrained in low-precision, high-speed training settings.

9. Conclusion … A.2. Proof of Policy Improvement Bound: Max-Divergence Part

The paper develops DPPO to replace PPO’s structurally biased ratio clipping with direct policy-divergence constraints, supported by finite-horizon policy-improvement theory. The appendix proves the performance-difference identity and derives max-divergence bounds from sequence-level to single-step TV divergence.

  • 9. Conclusion: The conclusion frames the work as a rethinking of trust-region methods for finite-horizon, undiscounted LLM fine-tuning.It reports policy-improvement bounds and empirical analysis of training stability versus efficiency.
  • 9. Conclusion: DPPO addresses PPO’s structural bias by replacing heuristic ratio clipping with a more principled policy-divergence constraint.PPO over-constrains low-probability-token updates while under-constraining potentially catastrophic high-probability-token shifts.
  • A.1. Proof of Performance Difference Identity: The performance-difference proof rewrites expected-return differences using an identity for sequence-generation probability differences.The derivation uses a telescoping sum and then matches the resulting terms to the theorem definitions.
  • A.2. Proof of Policy Improvement Bound: Max-Divergence Part: Sequence-level TV divergence is bounded by the sum of expected single-step TV divergences under the state distribution induced by µ.The proof applies a product-difference identity, the triangle inequality, marginalization over future tokens, and the definition of single-step TV divergence.
  • A.2. Proof of Policy Improvement Bound: Max-Divergence Part: The max-divergence proof bounds the performance-difference error using the maximum absolute reward and future-trajectory TV divergence.The reward is bounded by ξ = maxy |R(y)|, while sequence-level TV is controlled through the policy-improvement identity.
  • A.2. Proof of Policy Improvement Bound: Max-Divergence Part: The derivation upper-bounds each step by Dmax_TV(µ ∥π) = maxs DTV(µ(·|s)∥π(·|s)) and obtains the max-divergence bound in Equation (7).The final substitution into the performance-difference identity completes the proof.

A.3. Proof of Policy Improvement Bound: Average-Divergence Part … D. Characterizing Clipped Tokens

The paper establishes divergence-based guarantees and principled approximations for LLM policy updates, connects its sequence-level surrogate to classical policy gradients, and details stability-analysis procedures. It further shows that PPO-style clipping frequently targets mathematically and reasoning-critical tokens, disrupting useful learning signals.

  • A.3. Proof of Policy Improvement Bound: Average-Divergence Part: The average-divergence policy-improvement bound is linear rather than quadratic in the response horizon, making it more suitable for long LLM responses.The proof uses DTV(P∥Q) ≤1 instead of bounding future-trajectory divergence by (T −t)Dmax; together with the quadratic bound, this yields a composite guarantee.
  • A.4. Comparing Surrogate Objectives with Classical RL: The LLM trajectory-level surrogate has a gradient directly analogous to the classical policy-gradient surrogate, validating its trust-region formulation for undiscounted sequence rewards.Defining Aµ(st, yt) = R(y) −V (x) converts the sequence-level gradient into a timestep sum analogous to the classical formulation.
  • B. Approximations as Lower Bounds of True Divergence: Binary and Top-K divergence approximations are principled lower bounds on the true divergence, with exactness characterized by within-partition alignment conditions.The approximations arise by computing divergence after partitioning the vocabulary; the paper explicitly states conditions under which the lower bounds become exact.
  • B.1. Total Variation Divergence: For Total Variation, both Binary and Top-K partitions provide lower bounds, while Top-K’s gap is controlled by the probability mass of its tail category.The tail is typically small, making Top-K TV a high-fidelity approximation; equality holds when token-level differences share a sign within each partition.
  • B.2. KL Divergence: For KL divergence, the partitioned approximation becomes exact when the probability ratio µ(a|st)/π(a|st) is constant within every partition.For Binary and Top-K, this requires probabilities in the “other” category to be scaled by a uniform factor.
  • C. More Details for Stability Analysis: The stability analysis samples 64 questions per policy iteration, generates 8 responses per question, performs 4 gradient steps, and trains with VeRL, ODC, and BFloat16.The setup uses a maximum context length of 8,000 and follows the sanity test established in Qi et al. (2025b).
  • C.1. Algorithmic Details for Stability Analysis: All evaluated algorithms fit a unified policy-gradient framework, while DPPO uniquely conditions its update mask on policy divergence with C = ∞.The experiments use δ = 0.15 for TV divergence and δ = 0.05 for KL divergence; other methods differ through their masks and clipping bounds.
  • D. Characterizing Clipped Tokens: PPO-style clipping frequently targets high-utility numerical, mathematical, reasoning, and structural tokens, blocking beneficial updates and necessary suppression during incorrect reasoning paths.The observed clipped sets include tokens such as ‘1’, ‘4’, ‘+’, ‘=’, ‘Wait’, ‘Next’, ‘Thus’, and ‘Since’ in both positive- and negative-reward samples.

E. More Details for Broader Evaluation · F. More Empirical Results · F.1. AlpacaEval 2.0 Evaluation for RLHF

The paper details scaling-experiment settings and evaluates DPPO on AlpacaEval 2.0, where it achieves stronger rewards and win rates than GRPO while later checkpoints exhibit reward hacking.

  • E. More Details for Broader Evaluation: Scaling experiments use VeRL on NVIDIA H Series GPUs, with rollout router replay only for MoE Base w/ R3 to mitigate training-inference mismatch.All methods follow the hyperparameter configurations in Table 1.
  • E. More Details for Broader Evaluation: All methods construct the trust region from the behavior policy µθ′, using unified policy-gradient hyperparameters with C = 5 by default.GRPO-ClipHigher uses ϵhigh = 0.27 and ϵlow = 0.2; CISPO uses Mt = 1.
  • E. More Details for Broader Evaluation: DPPO-Binary-KL and DPPO-Binary-TV mask updates using binary KL or TV divergence estimates, with δ values varying by method and scaling experiment.DPPO-Binary-KL uses δ = 0.05 throughout; DPPO-Binary-TV uses δ = 0.15 for MoE Base w/ LoRA and δ = 0.2 otherwise.
  • E. More Details for Broader Evaluation: Online evaluation monitors AIME24 and AIME25 throughout training, every 5 steps for most MoE configurations and every 10 steps for Dense Base and MoE Base w/ LoRA.Sampling uses temperature=0.7, top p=0.95, and n=32, reporting average scores across 32 samples per question.
  • F. More Empirical Results: DPPO obtains higher learned reward than GRPO at step 150 (51.32 vs. 36.70) and step 450 (70.27 vs. 45.24).The AlpacaEval checkpoint is taken at step 150 because later checkpoints score lower despite higher reward, indicating reward hacking.
  • F.1. AlpacaEval 2.0 Evaluation for RLHF: On AlpacaEval 2.0, DPPO achieves the highest length-controlled and raw win rates among compared checkpoints and establishes a new SOTA on the community leaderboard.This shows DPPO's reward improvement transfers to an external open-ended alignment benchmark rather than only increasing the reward-model score.

F.2. Extended Main Results … F.5. Extended Results for Different Model × Task Combinations

Across extended scaling experiments and additional model–task settings, DPPO shows more stable, efficient, and often stronger training than ratio-clipping baselines. Binary divergence approximations remain sufficient, while DPPO is robust to tested thresholds and benefits further from rollout router replay.

  • F.2. Extended Main Results: Across five scaling experiments, DPPO consistently improves training rewards, AIME 2024 Avg@32, and AIME 2025 Avg@32 with stable learning curves and strong final performance.The experiments track rewards, benchmark scores, mean of |π −µ|, policy entropy, and response length.
  • F.2. Extended Main Results: DPPO maintains stable training-inference mismatch and policy entropy while increasing response length in all scaling experiments except MoE Thinking.Qwen3-30B-A3B already produces extremely long responses, and training enforces a maximum length of approximately 16k tokens.
  • F.2. Extended Main Results: GRPO-ClipHigher has lower stability and inferior final performance than DPPO across all five large-scale experiments, including collapse in MoE Thinking.CISPO is also less stable, with severe collapse in MoE Base w/o R3 and degenerative or fluctuating behavior in other settings.
  • F.2. Extended Main Results: DPPO variants without rollout router replay outperform baselines using R3, while adding R3 provides further gains, indicating largely orthogonal benefits.These results underscore the importance of proper masking and position DPPO as a robust foundation for LLM RL fine-tuning.
  • F.3. Ablation on Divergence Approximation: With K = 20, DPPO-TopK-TV and DPPO-TopK-KL do not significantly improve performance over binary approximations, supporting binary TV/KL as an efficient trust-region proxy.The K = 20 choice is limited by vLLM, which returns log probabilities for at most 20 candidate tokens per step.
  • F.4. Hyperparameter Sensitivity: DPPO-Binary-TV performs comparably for δ ∈[0.10, 0.20], DPPO-Binary-KL remains strong for δ ∈[0.05, 0.15], and all tested configurations outperform GRPO baselines.GRPO curves vary more noticeably with the upper clipping parameter.
  • F.5. Extended Results for Different Model × Task Combinations: Extended evaluations cover OctoThinker-3B-Hybrid-Base on math reasoning, Qwen3-1.7B-Base on Arc1D and Acre, and Qwen3-1.7B-Base on Sudoku-v0easy.These settings test different model families, abstract reasoning and induction, and multi-turn reasoning.
  • F.5. Extended Results for Different Model × Task Combinations: Using δ = 0.2, DPPO-Binary-TV improves efficiency and sometimes asymptotic performance over PPO-Ratio under the same GRPO framework.The comparison changes only the trust-region masking strategy and uses standard Oat example-script hyperparameters.
Loading 2602.04879v3…