PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence

Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

arXiv:2610.13015v1cs.LGcs.AI

TL;DR

PPO’s clipping proxy and subsequent heuristic variants leave a gap in principled trust-region design for LLM reinforcement learning. GPPO unifies divergence and estimator choices under a generalized f-divergence trust-region theory and introduces estimator topologies. Experiments find both design dimensions matter, with KL and Atomic estimation producing the strongest reported performance.

  • Problem

    PPO’s TV-based clipping proxy has fixed, symmetric geometry, while later variants make fragmented heuristic changes without a unified account of trust-region design.

    Source: §1 · pp. 1–2

  • Method

    GPPO unifies proximal optimization through divergence and estimator choices, with a generalized f-divergence trust-region theory and novel estimator topologies.

    Source: §1 · p. 2

  • Results

    KL with Atomic estimation improves average score by 50% on math and 14% on coding relative to standard TV-MC, and achieves the strongest performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GPPO provides a systematic foundation for understanding and designing future LLM reinforcement-learning algorithms.

    Source: §5 · p. 10

    Single-sample Monte Carlo estimates make proximal clipping decisions susceptible to sampling variance.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized $f$-divergence trust-region theory with a monotonic performance improvement guarantee, and show that different $f$-divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of {Divergence $\times$ Estimator}. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance.

1 INTRODUCTION

PPO’s clipping proxy and fragmented variants leave LLM policy optimization without a unified account of trust-region design. GPPO unifies divergence and estimator choices, establishes a generalized trust-region guarantee, and reports gains from both design dimensions.

1 INTRODUCTION

  • PPO’s zero-curvature, symmetric TV proxy cannot adapt resistance to shift magnitude, while later variants make fragmented heuristic changes without a unified design account.These limitations can manifest as premature clipping, distorted trust-region scaling, and exploration collapse.Source: §1 · pp. 1–2
  • GPPO frames proximal optimization as a two-dimensional Divergence × Estimator design space that unifies existing variants and supports systematic algorithm design.PPO, DAPO, DPPO, and RIPO emerge as special cases under this framework.Source: §1 · p. 2
  • An f-divergence level-set trust region guarantees monotonic performance improvement, while divergence-specific geometry explains distinct policy update dynamics.Source: §1 · p. 2
  • Novel estimator topologies reduce single-sample Monte Carlo variance and provide more faithful divergence estimates.Source: §1 · p. 2
  • KL with Atomic estimation improves average score by 50% on math and 14% on coding versus standard TV-MC, achieving the strongest performance.Atomic estimation outperforms single-sample Monte Carlo across all evaluated divergences; divergences with non-zero curvature generally outperform linear, symmetric TV geometry.Source: §1 · p. 2

2 PRELIMINARY

Trust-region constraints bound policy changes while preserving a performance-improvement objective. PPO replaces computationally costly constrained optimization with clipping, which acts on a noisy single-sample proxy for TV divergence.

2.1 MONOTONIC IMPROVEMENT UNDER TRUST REGION CONSTRAINT

  • The trust-region performance bound lower-bounds improvement using the behavior-policy surrogate return, with a penalty tied to maximum state-wise TV divergence.Source: §2.1 · p. 2
  • TRPO maximizes the surrogate objective while constraining policy distance; Pinsker’s inequality also motivates a KL-based trust-region constraint.Source: §2.1 · pp. 2–3

2.2 PROXIMAL POLICY OPTIMIZATION AS A DIVERGENCE PROXY

  • PPO avoids TRPO’s costly second-order optimization with a first-order clipped surrogate; advantages use GAE in PPO or group-relative estimates in modern LLM RL.Source: §2.2 · p. 3
  • PPO clipping bounds a noisy single-sample estimate of TV divergence, whose symmetric, zero-curvature geometry lacks adaptive resistance and is sensitive to sampling noise.The limitations can manifest as premature clipping, distorted trust-region scaling, and exploration collapse.Source: §2.2 · p. 3

3 GENERALIZED PROXIMAL POLICY OPTIMIZATION

GPPO generalizes proximal policy optimization along divergence and estimator dimensions, providing a framework for interpreting existing methods and designing alternatives.

3.1 GENERALIZED f-DIVERGENCE TRUST REGION

  • Constraining a sufficiently small level set of any f-divergence guarantees monotonic performance improvement; GPPO implements these constraints with divergence-induced clipped proximal regions.Source: §3.1 · p. 4

3.2 DIVERGENCE GEOMETRY AND POLICY UPDATE DYNAMICS

  • Smooth divergences share local Fisher–Rao geometry but differ at finite shifts, while TV's zero curvature yields magnitude-independent resistance.Source: §3.2 · p. 5
  • KL's asymmetric geometry increasingly penalizes contraction but offers diminishing resistance to expansion; Hellinger induces spherical geometry with stronger curvature near r → 0.Source: §3.2 · p. 5

3.3 DIVERGENCE ESTIMATOR TOPOLOGIES

  • Atomic estimation weights each action's divergence contribution by reference probability, while Binary includes complementary mass; single-sample MC remains sensitive to sampling variance.Source: §3.3 · p. 6

3.4 UNIFIED VIEW OF EXISTING PROXIMAL POLICY OPTIMIZATION METHODS

  • PPO and GRPO map to TV–MC, DAPO to KL–MC, DPPO to TV–Binary, and RIPO to χ2–Atomic; f sets resistance while the estimator governs enforcement.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Across eight competition-level math and coding benchmarks, GPPO's best configuration leads existing RL algorithms, while results vary with both divergence and estimator choice.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans eight competition-level math and code benchmarks, testing a 4 × 3 divergence–estimator grid that recovers several existing methods.Source: §4 · p. 7, §4.1 · pp. 7, 9

4.2 MAIN RESULTS

  • 41.5 on math and 59.7 on code; KL–Atomic improves over standard TV–MC (GRPO) by +49.8% and +13.9%, respectively.Source: Table 1 · p. 8, §4.2 · p. 9
  • 24.9% on math and 3.6% on code: under single-sample MC, KL beats TV; under Atomic, KL exceeds TV by +14.3% and +8.7%, respectively.Source: §4.2 · p. 9
  • Binary reverses the KL-over-TV math result seen under Atomic, showing that divergence rankings depend on estimator topology.Source: §4.2 · p. 9
  • 21.3% on math and 3.2% on code for Binary over MC; Atomic gains are 28.3% and 7.7%, respectively, averaged across four divergences.Source: Figure 2 · p. 9, §4.2 · p. 10

5 CONCLUSION

GPPO unifies proximal policy optimization through divergence geometry and estimator topology, supported by a generalized f-divergence trust-region theorem. Across eight math and coding benchmarks, it substantially outperforms standard PPO and identifies both design dimensions as critical.

5 CONCLUSION

  • GPPO unifies proximal policy optimization through a two-dimensional design space of divergence geometry and estimator topology, supported by a generalized f-divergence trust-region theorem.Source: §5 · p. 10
  • Across eight mathematical reasoning and code generation benchmarks, GPPO substantially outperforms standard PPO.Source: §5 · p. 10
  • The results demonstrate that both divergence geometry and estimator topology are critical for effective LLM reinforcement learning.Source: §5 · p. 10
  • GPPO provides a systematic foundation for understanding and designing future LLM RL algorithms.Source: §5 · p. 10

A DERIVATION FOR THEOREM 2

The derivation reduces the f-divergence lower-bound problem to a binary partition determined by total variation and establishes the bound's tightness. It also characterizes the optimal universal quadratic constant and distinguishes local from global behavior.

A DERIVATION FOR THEOREM 2

  • Reducing the sample space to the binary partition induced by total variation yields a tight lower bound, so no larger universal bound depending only on total variation holds.Source: A DERIVATION FOR THEOREM 2 · pp. 13–14
  • The auxiliary parameter β lies in [0, 1 − V], with boundary cases interpreted by continuity and extended-valued f-divergences when a denominator vanishes.Source: A DERIVATION FOR THEOREM 2 · p. 14
  • C_f is the optimal universal quadratic constant: no larger value satisfies the quadratic lower bound uniformly over all probability distributions.Source: A DERIVATION FOR THEOREM 2 · p. 15
  • The local quadratic coefficient around V = 0 is attained at β = 1/2 and differs from the global Vajda constant, which controls all V ∈ (0, 1].Equality between the local and global constants holds when the global minimum of L_f(V)/V^2 is attained asymptotically at V = 0.Source: A DERIVATION FOR THEOREM 2 · p. 16

B VARIANCE REDUCTION OF ATOMIC ESTIMATION

Atomic Estimation probability-weights action contributions, reducing the influence of rare actions on second-moment variance. This helps control long-tail sensitivity while distinguishing a ratio deviation's magnitude from its contribution to the divergence.

B VARIANCE REDUCTION OF ATOMIC ESTIMATION

  • The second moment captures large-magnitude samples and is a dominant source of variance in long-tailed importance-weighted estimation.Source: B VARIANCE REDUCTION OF ATOMIC… · p. 17
  • For rare actions with µ(a | s) ≪ 1, Atomic Estimation reduces the second-moment contribution by an additional factor µ(a | s)^2, keeping large ratio deviations controlled.Source: B VARIANCE REDUCTION OF ATOMIC… · p. 17
  • Atomic Estimation mitigates long-tail sensitivity in MC-based constraints through probability-aware weighting, improving proximal-estimation stability while separating ratio magnitude from divergence contribution.Source: B VARIANCE REDUCTION OF ATOMIC… · p. 17
Loading 2610.13015v1…