PaperlayerView exact revision on arXiv

Source-linked AI summary

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

Chongyu Fan, Pengfei Liu, Jingjia Huang, Sijia Liu, Yi Lin

arXiv:2607.06987v1cs.LG

TL;DR

Importance-sampling RL faces a dilemma between exploration and training stability because clipping can restrict promising reasoning updates. UP uses a self-anchored, asymmetric objective to remove clipping for positive advantages while retaining safeguards for negative ones, improving exploration and reasoning accuracy across diverse settings.

  • Problem

    Importance-sampling RL must balance exploration against stability, but conservative clipping can restrict updates for promising low-confidence reasoning trajectories.

  • Method

    UP anchors the policy with stop-gradient, applying unbounded updates to positive advantages and standard clipping safeguards to negative advantages across token- and sequence-level optimization.

  • Results

    UP consistently improves exploration capacity and reasoning accuracy across RL algorithms, architectures, and language and multimodal training settings, achieving the best average Pass@1 among eleven baselines.

  • Takeaways & Limitations

    UP is supported as a universal plug-and-play enhancement for RL-based reasoning training across token-level and sequence-level frameworks.

  • Takeaways & Limitations

    Unbounded updates must be prohibited for negative advantages because they can aggressively ascend gradients and destroy the original representation.

Abstract

from arXiv · show

Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs). To achieve sample efficiency, modern RL frameworks rely on importance sampling (IS). However, these algorithms suffer from an exploration-stability dilemma. Pure IS often leads to catastrophic training instability, while standard clipping mechanisms used to mitigate this instability strictly constrain the policy update budget. By formalizing the concept of Probability Capacity (Cap), we reveal that conservative clipping structurally stifles exploration by prematurely truncating the update budget for correct but low-confidence reasoning paths. To break free from these constraints, we propose Unbounded Positive Asymmetric Optimization (UP), a universal and plug-and-play objective. UP theoretically restructures the optimization process by anchoring the policy to its current state via the stop-gradient operator. This asymmetric design unleashes unclipped, stable gradients for positive advantages to maximize exploration, while maintaining standard clipping safeguards for negative advantages to prevent training instability. Furthermore, our formulation readily extends across different optimization granularities, including token-level (GRPO, DAPO) and sequence-level (GSPO) frameworks. Extensive experiments demonstrate that UP enhances exploration capacity and achieves superior reasoning accuracy across diverse RL algorithms (DAPO, GSPO, and GRPO), model architectures (Dense, MoE, and vision-language), and training modalities (language and multimodal), validating UP as a truly universal plug-and-play enhancement for RL-based training.

1 Introduction

The introduction identifies an exploration–stability dilemma in importance-sampling-based RL: unclipped updates can become unstable, while conservative clipping limits exploration. It presents UP, which asymmetrically removes clipping for positive advantages while preserving safeguards for negative advantages, and reports broad empirical gains.

  • Problem: Importance sampling improves sample efficiency in multi-step RL, but unclipped ratios can explode on rare, long-tail reasoning paths and cause catastrophic training instability.GRPO and DAPO use importance sampling to estimate target distributions from historical policies.
  • Problem: Standard clipping limits allowable token-probability changes, structurally stifling exploration for positive-advantage reasoning paths.Probability Capacity (Cap) formalizes the maximum allowable probability increase or decrease before optimization gradients are truncated.
  • Method: UP uses stop-gradient anchoring and asymmetric optimization to provide unclipped, stable gradients for positive advantages while safeguarding against instability for negative advantages.The formulation is designed to maximize exploration capacity without sacrificing optimization stability.
  • Method: UP is a universal, plug-and-play objective that extends across token-level GRPO and DAPO and sequence-level GSPO optimization.The introduction explicitly describes UP as applicable across different optimization granularities.
  • Results: UP enhances exploration capacity and accuracy across DAPO, GSPO, and GRPO, Dense, MoE, and vision-language models, and language and multimodal training.The reported experiments also show prevention of training instability across these settings.

2 Related Work

Prior work has advanced RL for LLM reasoning from REINFORCE and PPO to critic-free methods such as GRPO and DAPO, while importance-sampling methods use clipping to control instability. However, existing clipping refinements retain the historical policy in the denominator, which can bottleneck exploration when successful trajectories are sparse.

  • Reinforcement Learning for LLM Reasoning: PPO became the standard RLHF/RLAIF paradigm by improving sample efficiency, but its auxiliary critic limits scalability for long-context reasoning.This motivates critic-free alternatives for reasoning-focused RL.
  • Reinforcement Learning for LLM Reasoning: GRPO removes the critic through group-level relative scoring and has driven breakthroughs in models including DeepSeek-R1 and Qwen2.5-Math.GRPO extends the policy-gradient lineage toward more scalable reasoning optimization.
  • Reinforcement Learning for LLM Reasoning: DAPO builds on GRPO by introducing decoupled clipping.The supplied passage identifies DAPO as a GRPO-based development but does not specify the clipping components further.
  • Importance Sampling and Asymmetric Optimization: TRPO and PPO introduced heuristic clipping to prevent instability from exploding importance-sampling ratios.Later methods refine these clipping bounds, but the passage frames clipping as a response to instability rather than a solution to exploration constraints.
  • Importance Sampling and Asymmetric Optimization: BandPO, SAPO, GMPO, and M2PO refine clipping bounds but retain the historical policy πold in the denominator, bottlenecking exploration when golden trajectories are sparse.The passage presents this denominator assumption as a shared limitation across these asymmetric optimization efforts.

3 The Exploration-Stability Dilemma in Importance Sampling and Clipping

Importance sampling improves sample efficiency by reusing sampled data, but unclipped ratios can cause pathological gradient explosions. Clipping stabilizes optimization while imposing a probability-capacity bottleneck that prematurely blocks exploration of promising low-likelihood reasoning paths.

  • GRPO and DAPO: Importance Sampling enables multi-step optimization over reused samples, addressing REINFORCE’s severe sample inefficiency.REINFORCE discards trajectories after one gradient step, whereas GRPO introduces importance sampling to reuse sampled data.
  • Clipping safeguards: Clipping constrains importance ratios within a trust region to mitigate instability, but DAPO relies exclusively on clipping after removing the KL penalty.GRPO uses [1 − ϵ, 1 + ϵ], while DAPO introduces decoupled bounds (ϵlow, ϵhigh) with β = 0.
  • Pure IS instability: Unclipped importance sampling becomes unstable because rare, high-reward paths with tiny behavior probabilities produce exploding ratios and pathological gradients.As the active policy marginally improves, the ratio can explode, destroying stable representations and causing severe training instability.
  • Probability Capacity: Probability Capacity measures the maximum absolute probability increase for positive advantages or decrease for non-positive advantages before policy updates are blocked.It represents the effective exploration budget relative to the reference policy πold.
  • Probability Capacity: 0.0028 is the maximum absolute probability increase when πold = 0.01 and ϵhigh = 0.28; capacity then reaches exactly zero at πθ = 0.0128.The gradient is nullified in the Upper Clip region, even when the action has an exceptionally high advantage, structurally stifling exploration.

4 Breaking the Dilemma: Unbounded Positive Asymmetric Optimization

UP breaks the exploration–stability dilemma by anchoring optimization to the current policy with stop-gradient and routing updates asymmetrically by rollout correctness. It removes clipping for positive advantages to expand exploration while retaining clipping-based safeguards for negative advantages, and extends across token- and sequence-level GxPO algorithms.

  • Core formulation: UP replaces the historical policy with the current policy through stop-gradient, creating a self-anchored ratio for correct rollouts with positive advantages.The stop-gradient denominator equals the current policy in value while acting as a constant during backpropagation.
  • Core formulation: The self-anchored unclipped ratio is mathematically equivalent to the REINFORCE objective and enables unbounded reinforcement without the analyzed importance-sampling gradient explosion.Anchoring to πθ rather than πold removes the stated root cause of IS-induced instability.
  • Asymmetric routing: UP applies unbounded reinforcement exclusively to correct rollouts, maximizing exploration for rare, low-confidence tokens while prohibiting unbounded updates for wrong rollouts.Unbounded updates in the negative-advantage regime would reverse gradient direction and damage the original representation.
  • Asymmetric routing: UP dynamically routes gradients by rollout correctness, combining an unbounded positive branch with baseline clipping mechanisms for negative advantages across GxPO algorithms.UP-DAPO uses DAPO’s decoupled clipping for wrong rollouts, while UP-GRPO retains GRPO clipping and its KL penalty on the negative branch.
  • Universality: UP extends across optimization granularities through UP-DAPO and UP-GRPO at token level and UP-GSPO at sequence level.For UP-GSPO, the positive branch is analytically shown to equal an unclipped, length-normalized REINFORCE gradient.

5 Experiments

Experiments evaluate UP across multiple optimization variants, model architectures, benchmarks, and modalities. Across these settings, UP improves accuracy and exploration while preserving training stability, and ablations support its self-anchored asymmetric design.

  • Cross-Setting Generalization: Additional evaluations test UP-GSPO on a Qwen3-30B-A3B-Base MoE model and UP-GRPO on Qwen3-VL-8B-Instruct with Geometry3K, extending validation across architectures and modalities.The paper also describes evaluations across dense, MoE, and vision-language model families.
  • Accuracy and Exploration: UP-DAPO achieves peak Best@32 81.79 versus DAPO’s 80.49, with higher generation-probability entropy indicating greater exploration capacity.The increased entropy is reported on DAPO-17K-MATH, while Best@32 is evaluated on AIME24.
  • Training Stability: Despite removing importance-sampling and clipping mechanisms for correct rollouts, UP-DAPO maintains gradient norms and KL divergence comparable to or slightly lower than DAPO.These metrics assess training stability on DAPO-17K-MATH.
  • Ablation Studies: Ablations show that relaxing the original DAPO ratio with ϵ_high = ∞ and applying unbounded updates to negative advantages fail to provide the stable exploration achieved by UP-DAPO.The first modification initially remains stable, while the symmetric off-policy REINFORCE-style variant causes immediate catastrophic instability.
  • Cross-Setting Generalization: UP-GRPO attains 61.31% average Pass@1 across five benchmarks, surpassing GSPO’s 60.15% by 1.16 percentage points and supporting broad gains across benchmark difficulties.The evaluation uses Qwen3-8B trained on MATH Levels 3–5.

6 Conclusion

The conclusion presents UP as a solution to the exploration–stability dilemma in importance-sampling-based reinforcement learning, maximizing exploration capacity while preventing training instability. It emphasizes UP’s universal applicability across optimization algorithms, model architectures, and training modalities.

  • 6 Conclusion: UP addresses the exploration–stability dilemma by unleashing exploration while strictly preventing training instability.Its asymmetric design is presented as the mechanism enabling both objectives.
  • 6 Conclusion: UP demonstrates universal applicability across DAPO, GSPO, and GRPO.The conclusion identifies these as the covered optimization algorithms.
  • 6 Conclusion: UP generalizes across Dense, MoE, and vision-language architectures and language and multimodal training modalities.This breadth supports the conclusion’s characterization of UP as a universal paradigm.

Appendix · A Training and Evaluation Details

The appendix specifies fair training and evaluation protocols by holding foundational setups constant between each baseline and its UP variant, changing only the optimization objective. It documents asymmetric clipping, decoding configurations, and the corresponding token-level, sequence-level, language, and multimodal experiment settings.

  • A Training and Evaluation Details: Baseline and UP comparisons use identical foundational training setups, hardware allocations, and decoding strategies within each model class, differing only in the mathematical optimization objective.This design enforces a strictly fair empirical comparison.
  • A Training and Evaluation Details: UP applies clipping bounds only to negative advantages, leaving positive advantages unconstrained and eliminating the upper clip hyperparameter ϵhigh.Table A1 covers DAPO and UP-DAPO at token level and GSPO and UP-GSPO at sequence level.
  • A Training and Evaluation Details: Table A2 specifies the decoding and rollout-generation configurations for the token-level DAPO and UP-DAPO and sequence-level GSPO and UP-GSPO experiments.The appendix separates core optimization settings from rollout decoding settings.
  • A Training and Evaluation Details: GRPO and UP-GRPO on Qwen3-8B trained on MATH (level 3-5) share identical configurations, with UP-GRPO removing the positive-advantage upper clipping bound and ϵhigh.Tables A3 and A4 provide their training and decoding configurations, while remaining baselines retain the stated training and evaluation settings.
  • A Training and Evaluation Details: The comprehensive baseline comparison retains the specified backbone model, training corpus, and evaluation settings for remaining baselines.This preserves comparability beyond the direct GRPO and UP-GRPO pairing.
  • A Training and Evaluation Details: For multimodal reasoning, GRPO and UP-GRPO on Qwen3-VL-8B-Instruct trained on Geometry3K share identical configurations, with UP-GRPO removing the positive-advantage upper clipping bound and ϵhigh.Tables A5 and A6 report the training, algorithmic, decoding, and rollout configurations.

B Derivation of Probability Capacity for Negative Advantages

For non-positive advantages, DAPO and UP-DAPO use decoupled lower and dual clipping to bound probability decreases and prevent excessive penalization or instability. The resulting Probability Capacity is the allowable decrease within the effective window, maximized near the dual-clip boundary and vanishing at either clipping limit.

  • Negative-advantage clipping: For non-positive advantages, DAPO and UP-DAPO employ decoupled clipping to prevent excessive penalization and training instability.The clipped objective is denoted Jneg.
  • Negative-advantage clipping: The lower boundary is πlower = (1 −ϵlow)πold, below which the gradient is nullified to prevent over-penalization.The dual boundary is πupper = c · πold, above which the gradient vanishes to keep the policy from moving farther from the reference.
  • Capacity definition: In the negative regime, Probability Capacity represents the maximum allowable probability decrease before πθ reaches the lower clip boundary.Optimization decreases the token probability, so Capacity is defined relative to the lower trust-region limit.
  • Capacity definition: The piecewise Capacity is πθ −(1 −ϵlow)πold inside the clipping window and 0 below the lower clip or above the dual clip.The effective window is (1 −ϵlow)πold ≤πθ ≤c · πold.
  • Capacity behavior: Capacity is maximized near the dual-clip boundary and gradually decreases to zero as πθ approaches the lower trust-region limit.This behavior corresponds to the blue regions in Fig. 1(c).

C Derivation of the UP-GSPO Gradient

The UP-GSPO derivation shows that stop-gradient anchoring converts positive-advantage updates into a length-normalized REINFORCE gradient without an importance-sampling clipping upper bound. The unified objective routes positive advantages through this unbounded formulation while retaining standard sequence-level clipping for negative advantages.

  • Gradient derivation: Stop-gradient anchoring treats the denominator as constant during backpropagation while preserving its forward value as the policy probability.This enables direct application of the chain rule and substitution back into the gradient expression.
  • Gradient derivation: Length-normalization exponents cancel, yielding a token-level sum representation of the sequence-level log-probability gradient.The sequence probability expands into the sum of token-level log probabilities.
  • Positive-advantage gradient: For positive advantages, UP-GSPO equals a length-normalized REINFORCE gradient that removes the importance-sampling clipping upper bound.The formulation preserves the variance-reducing effect of 1 |oi| normalization while increasing exploration capacity.
  • Unified objective: UP-GSPO uses asymmetric routing: unbounded positive optimization handles positive advantages, while standard sequence-level clipping safeguards negative advantages.This unified objective reflects the divergent optimization dynamics of correct and incorrect rollouts.
Loading 2607.06987v1…