PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization

Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Mingxuan Wang, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

arXiv:2607.10169v1cs.LGcs.AI

TL;DR

PPO-Clip’s Euclidean policy discrepancy measure mismatches the policy manifold’s intrinsic geometry, contributing to exploration collapse and limited theoretical understanding. RIPO enforces isometric manifold updates and consistently substantially outperforms representative RL baselines across competition-level benchmarks, with up to 60% relative improvement over GRPO on AIME24.

  • Problem

    PPO-Clip suffers exploration collapse, while subsequent fixes remain heuristic and do not identify its geometric root cause.

  • Method

    RIPO dynamically adjusts clipping to enforce isometric updates on the policy Riemannian manifold, with a favorable bias–variance trade-off.

  • Results

    Up to 60% relative improvement over GRPO on AIME24 accompanies RIPO’s consistent, substantial gains across competition-level benchmarks and representative RL baselines.

  • Takeaways & Limitations

    RIPO provides a principled pathway for more effective and stable reinforcement learning algorithms for large language models.

  • Takeaways & Limitations

    Trust-region policy optimization incurs significant computational overhead, making it difficult to scale in practice.

Abstract

from arXiv · show

Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).

1. Introduction

The introduction identifies exploration collapse as PPO-Clip’s fundamental failure, attributing it to a mismatch between Euclidean clipping and the policy manifold’s intrinsic Riemannian geometry. It presents RIPO, which enforces geometry-aware isometric updates and achieves up to 60% relative improvement over GRPO on AIME24.

  • Problem: PPO-Clip concentrates policies on narrow sets of high-probability actions, leaving rare but vital actions unexplored and suppressing performance in long-horizon reasoning.The vast state-action spaces of long-horizon tasks make this exploration collapse especially damaging.
  • Motivation: Existing fixes such as DAPO raise the clipping boundary, but remain heuristic within PPO-Clip and do not resolve its underlying exploration-collapse problem.These methods provide symptomatic fixes without identifying the root cause of PPO-Clip’s failure.
  • Geometric diagnosis: PPO-Clip’s fundamental flaw is measuring policy discrepancy with a Euclidean importance-ratio metric incompatible with the KL-induced Riemannian geometry of the policy manifold.Equal ratio deviations therefore do not represent equal intrinsic policy changes.
  • Method: RIPO dynamically adapts clipping to local Riemannian geometry, permitting larger low-probability-action updates and constraining high-probability-action updates to balance exploration and exploitation.The proposed isometric updates are theoretically grounded and induce a favorable bias–variance trade-off.
  • Results: 60% relative improvement over GRPO on AIME24 is achieved by RIPO across extensive experiments on various LLMs and competition-level benchmarks.The experiments report significant outperformance over existing RL algorithms.

2. Preliminary

This section traces the progression from TRPO’s KL-constrained policy optimization to PPO’s clipped first-order surrogate and GRPO’s value-model-free adaptation for LLM RL. It also highlights the computational motivation for these simplifications and the unresolved theoretical limitations of PPO-Clip variants.

  • TRPO: TRPO constrains the KL divergence between target and behavior policies, theoretically guaranteeing monotonic policy improvement.Despite these guarantees and empirical performance, constrained optimization incurs significant computational overhead and is difficult to scale.
  • PPO: PPO approximates TRPO’s second-order optimization with a first-order clipped surrogate objective, constraining updates within a proximal region.The importance ratio is clipped using a fixed boundary ϵ, with advantages estimated by GAE; this stabilizes training and improves policy performance.
  • GRPO: GRPO retains PPO-Clip while eliminating the value model and estimating advantages in a group-relative manner for computational and memory efficiency.Given a query q ∼D, the behavior policy samples a group of G individual responses whose advantages are normalized within the group.
  • PPO-Clip variants: GRPO variants target PPO-Clip’s exploration collapse and gradient instability through asymmetric, dynamic, or sequential-ratio clipping.DAPO uses ϵhigh > ϵlow, DCPO adapts thresholds dynamically, and GSPO and GMPO clip sequential importance ratios to reduce gradient variance.
  • Open limitation: Although these variants yield empirical gains, they remain largely heuristic and do not identify or address PPO-Clip’s fundamental flaw.The unresolved flaw is examined in the following section.

3. The Geometric Flaw of PPO Clip

PPO-Clip’s exploration collapse stems from using a Euclidean ratio constraint that mismatches the non-uniform Riemannian geometry induced by KL divergence. This mismatch makes updates overly conservative for low-probability actions and aggressive for high-probability actions.

  • Geometric Consequence: This geometric mismatch causes overly conservative updates in low-probability regions and aggressive updates in high-probability regions, ultimately producing exploration collapse.PPO-Clip therefore fails to align its trust-region approximation with the intrinsic geometry of policy distributions.
  • Exploration Collapse: PPO-Clip’s ratio clipping suppresses exploration by favoring already-preferred exploitation tokens over low-likelihood exploration tokens.With ϵ=0.2, a high-probability action at 0.8 can rise to 0.96, whereas a low-probability action at 0.01 can rise only to 0.012.
  • The Euclidean Assumption: PPO-Clip treats equal ratio deviations as equal policy discrepancies, implicitly assuming uniform policy change across the statistical manifold.Its constraint |r(θ) −1| < ϵ induces an implicit Euclidean distance measure independent of the underlying probability distribution.
  • Riemannian Geometry: KL divergence and the Fisher Information Matrix instead induce a Riemannian policy manifold whose geometric distances depend on the underlying probability simplex.Distances expand in high-probability regions and shrink in low-probability regions, making the geometry non-uniform.

4. Riemannian Isometric Policy Optimization

RIPO replaces PPO-Clip’s Euclidean ratio clipping with Riemannian Isometric Clip, which equalizes geometric update distances across state-action samples. This distribution-dependent mechanism expands updates for rare actions, reduces aggressive updates for common actions, and improves the bias–variance trade-off while supporting LLM RL objectives.

  • Riemannian Isometric Clip: RIC dynamically adjusts clipping boundaries using each local probability simplex so every state-action update consumes the same Riemannian trust-region budget.The geometric constraint is 2πθold(a|s)(rs,a(θ) −1)2 ≤δ, with δ representing the maximum geometric distance.
  • Riemannian Isometric Clip: RIC allows larger updates for low-probability actions and less aggressive updates for high-probability actions, balancing exploration and exploitation.This design directly addresses PPO-Clip’s overly conservative behavior in low-probability regions and aggressive behavior in high-probability regions.
  • Illustrative Example: With δ = 0.02, RIC raises a probability-0.01 action to 0.024 versus 0.012 under PPO-Clip, while limiting a probability-0.8 action to 0.92 versus 0.96.The example illustrates RIC’s larger rare-action update and reduced common-action update relative to PPO-Clip.
  • Bias–Variance Trade-off: RIC produces density-independent, constant-order variance that is lower than standard importance sampling and substantially less biased than PPO-Clip.PPO-Clip reduces variance by discarding clipped rare samples, whereas RIC considers more rare samples while retaining a controlled variance contribution.
  • RIPO Objective: RIPO combines RIC with GRPO’s group-relative advantage estimator and DAPO’s token-level policy-gradient loss to balance gradient influence across trajectory lengths.The complete algorithm is designed for LLM RL tasks and uses dynamic RIC clipping boundaries.

5. Experiments · 5.1. Experimental Setup

RIPO is evaluated across four language models, seven challenging mathematical-reasoning benchmarks, and comparisons with six representative RL algorithms. The setup uses DAPO-Math-17k training, standardized optimization settings, and contamination-free competition-level evaluation.

  • 5. Experiments: The study spans seven challenging benchmarks and evaluates mathematical reasoning as a strong indicator of general reasoning capability.Training uses DAPO-Math-17k with 17,917 questions, eight rollouts per question, and a maximum response length of 16,384 tokens.
  • 5.1. Experimental Setup: RIPO is tested on four large language models spanning different scales and architectures.The models are Llama3.2-3B-Instruct, Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen3-8B-Base.
  • 5.1. Experimental Setup: The experiments compare RIPO with six representative RL algorithms using distinct clipping mechanisms and default hyper-parameter settings.The baselines include GRPO, DAPO, GSPO, GMPO, and DCPO; RIPO uses δ = 0.05 by default.
  • 5.1. Experimental Setup: The experiments report algorithm comparisons using Avg@8 across mathematical-reasoning benchmarks.Table 1 is designated for comparisons of different RL algorithms across these benchmarks.
  • 5.1. Experimental Setup: All seven RL algorithms use group-relative advantage estimation, while KL penalties are removed and clipping bounds are set to 0.5 and 10.These settings follow prior work and apply across the compared methods.
  • 5.1. Experimental Setup: Models are optimized for 300 steps with AdamW, token-mean loss aggregation, and a constant learning rate of 1 × 10^-6.Experiments run in the VeRL framework on 8×A100 GPUs.
  • 5.1. Experimental Setup: The seven contamination-free evaluation benchmarks are AMC23, AIME24, AIME25, HMMT25, BRUMO25, CMIMC25, and Stanford Math Tournament 2025.These are competition-level mathematical-reasoning benchmarks.

5.2. Main Results · 5.3. Training Dynamics and Analysis

RIPO consistently delivers the strongest average performance across diverse models and seven competition-level mathematical reasoning benchmarks. Its training dynamics show faster, more stable improvement while preserving exploration through balanced entropy, gradients, and clipping.

  • 5.2. Main Results: RIPO achieves the best average performance across seven competition-level mathematical reasoning benchmarks and four base models with different scales and architectures.The experiments evaluate diverse model scales and architectures across seven benchmarks.
  • 5.2. Main Results: RIPO outperforms GRPO by up to 37.2% for Qwen3-1.7B-Base, 34.4% for Llama3.2-3B-Instruct, 17.1% for Qwen3-4B-Base, and 35.1% for Qwen3-8B-Base.These average improvements span all four evaluated base models.
  • 5.2. Main Results: RIPO delivers consistent and scalable improvements across diverse models and benchmarks, validating its theoretical foundation and practical effectiveness.The reported gains generalize across the evaluated model and benchmark settings.
  • 5.3. Training Dynamics and Analysis: RIPO improves AIME24 substantially faster than other methods, surpassing GRPO’s 200-step performance within 40 steps and achieving five times the token-efficiency.Its evaluation curve also rises smoothly with minimal oscillation and shows no training collapse.
  • 5.3. Training Dynamics and Analysis: GRPO’s policy entropy rapidly collapses to near zero, while DAPO exhibits uncontrolled entropy growth, corresponding to severe exploration collapse and excessive exploration, respectively.The entropy trajectories indicate sharply different exploration failures.
  • 5.3. Training Dynamics and Analysis: RIPO’s entropy decreases initially and then stabilizes within a moderate range, balancing exploration and exploitation while maintaining sustained exploration without destabilizing training.This trajectory contrasts with both GRPO’s collapse and DAPO’s uncontrolled growth.
  • 5.3. Training Dynamics and Analysis: RIPO maintains an almost fluctuation-free gradient norm, unlike other algorithms’ oscillations and spikes, consistent with stabilization through balancing bias and variance.The competing methods exhibit unstable updates and high-variance gradients during optimization.
  • 5.3. Training Dynamics and Analysis: RIPO achieves a balanced clipping ratio between rarely clipped DCPO and GMPO and much more frequently clipped GSPO and DAPO, indicating a calibrated trust region.This clipping behavior balances update flexibility and stability.

5.4. Ablation Study

The ablation study evaluates symmetric and decoupled clipping budgets for RIPO on AIME24, finding stable performance across δ values from 0.02 to 0.08 while identifying a δhigh setting that causes training collapse.

  • Ablation setup: The study varies RIPO’s symmetric clipping parameter δ and a decoupled variant with separate lower and upper budgets, δlow and δhigh, on AIME24.Experiments use Qwen3-8B-Base.
  • Ablation findings: RIPO maintains stable performance across a broad δ range of 0.02 to 0.08, demonstrating hyperparameter robustness.The reported range concerns the symmetric setting.
  • Ablation findings: The ablation also reports that training collapse occurs in an examined setting where δhigh is substantially adjusted.The supplied passage truncates the exact δhigh value or adjustment.

5.5. Comparison with Other Clippings · 5.6. Transfer to PPO Objective

Section 5.5 compares RIPO with alternative clipping strategies, while Section 5.6 tests RIPO-Clip beyond GRPO-style objectives under PPO. Across PPO experiments, RIPO-Clip consistently outperforms other clipping mechanisms and better preserves exploration with smoother optimization.

  • 5.5. Comparison with Other Clippings: RIPO is compared with GPPO, which preserves clipped-token gradients, and Clip-Cov, which clips high-covariance tokens to regulate entropy and encourage exploration.Experiments use Qwen3-8B-Base under the prior experimental settings.
  • 5.6. Transfer to PPO Objective: RIPO-Clip is extended beyond GRPO-style objectives to PPO, which uses a learned value function and generalized advantage estimation.This evaluates whether RIPO-Clip transfers across policy-optimization frameworks with different objective and advantage-estimation formulations.
  • 5.6. Transfer to PPO Objective: PPO experiments use GSM8K, with 7K training problems, 1K held-out test problems, and Qwen2.5-Instruct models at different scales.Responses are limited to 8,192 tokens; each iteration generates 256 rollouts with a train batch size of 512.
  • 5.6. Transfer to PPO Objective: RIPO-Clip consistently outperforms the other three clipping mechanisms on PPO, measured by Avg@1 on GSM8K.The comparison is reported in Table 4 despite differences in objective formulation and advantage estimation.
  • 5.6. Transfer to PPO Objective: PPO-Clip suffers severe exploration collapse as entropy rapidly approaches zero, whereas PPO with RIPO-Clip maintains a certain entropy level and promotes consistent exploration.These training dynamics are visualized for Qwen2.5-1.5B-Instruct on GSM8K.
  • 5.6. Transfer to PPO Objective: Other clipping methods show substantial gradient fluctuations and sharp spikes, while RIPO-Clip produces smoother gradients that indicate more stable optimization.The comparison concerns the training dynamics of Qwen2.5-1.5B-Instruct under PPO.

5.7. RIPO Breaks through the Capacity Boundaries · 5.8. Generalization to Coding and Search Tasks

RIPO extends reasoning capacity beyond the base model’s early plateau, reaching higher Pass@k performance on challenging benchmarks while preserving exploration. It also generalizes effectively to long-horizon coding and multi-hop search tasks, consistently outperforming GRPO.

  • 5.7. RIPO Breaks through the Capacity Boundaries: Deep-dive Pass@k analysis up to k=128 evaluates Qwen3-8B-Base on AIME-25 and HMMT-25, the two most challenging complex-reasoning benchmarks.The analysis studies capacity boundaries under different RL algorithms.
  • 5.7. RIPO Breaks through the Capacity Boundaries: The base model plateaus prematurely around k = 16, indicating limited intrinsic capacity.This plateau contrasts with the continued scaling achieved by RIPO.
  • 5.7. RIPO Breaks through the Capacity Boundaries: 60.0% on AIME-25 and 45.3% on HMMT-25 at k = 128 are RIPO’s peak results, exceeding other RL algorithms while scaling continuously.The results empirically indicate that RIPO mitigates exploration collapse, maintains policy diversity, and breaks through capacity boundaries in long-horizon reasoning.
  • 5.8. Generalization to Coding and Search Tasks: For coding, RIPO is evaluated on Codeforces, CodeContest, TACO, and APPS after training on the Eurus-Code dataset.TACO denotes Text-Assisted Coding with Objectives, and APPS denotes Automated Programming Progress Standard.
  • 5.8. Generalization to Coding and Search Tasks: For search, RIPO is evaluated on TriviaQA, PopQA, HotpotQA, and WikiMultiHopQA after training on the Search-R1 dataset using Qwen3-8B-Base recipes.The experiments follow previously stated training recipes.
  • 5.8. Generalization to Coding and Search Tasks: RIPO consistently outperforms GRPO with significant gains across coding and search benchmarks, validating effectiveness across different long-horizon reasoning tasks.The findings support that the addressed geometric mismatch is intrinsic and critical.

6. Conclusion

The paper identifies PPO-Clip’s fundamental limitation as a mismatch between the Euclidean metric and the policy manifold’s intrinsic geometry. It introduces RIPO, which enforces isometric policy updates to balance exploration and exploitation while enabling a favorable bias-variance trade-off.

  • 6. Conclusion: PPO-Clip’s fundamental limitation arises from a mismatch between the Euclidean metric and the policy manifold’s intrinsic geometry.This limitation is identified theoretically.
  • 6. Conclusion: RIPO guarantees isometric policy updates on the policy manifold to balance exploration and exploitation.The method is grounded in the paper’s geometric derivation.
  • 6. Conclusion: Geometric isometry induces statistical homoscedasticity, enabling a favorable bias-variance trade-off.The conclusion links the geometric property directly to optimization statistics.

Impact Statement

The paper introduces RIPO as a theoretically grounded RL algorithm that addresses a fundamental flaw in clipping-based policy optimization. By enforcing geometry-isometric updates, RIPO provides a principled framework for stable, scalable RL and improves LLM reasoning capabilities.

  • Impact Statement: RIPO resolves a fundamental flaw underlying clipping-based policy optimization through a theoretically grounded RL algorithm.The paper presents RIPO as its central methodological contribution.
  • Impact Statement: Geometry-isometric policy updates establish a principled framework for stable and scalable reinforcement learning.The framework is grounded in enforcing isometry during policy updates.
  • Impact Statement: The work provides theoretical insights for designing RL algorithms and enhances large language models’ reasoning capabilities.These broader contributions extend beyond the specific RIPO algorithm.
Loading 2607.10169v1…