PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Framework for Rethinking Policy Divergence Measures in GRPO

Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan, Yanning Dai, Shilong Deng, Sarra Habchi, Qi Zhu, Matthias Gallé, Chao Huang

arXiv:2602.05494v2cs.LGcs.AI

TL;DR

Existing RLVR methods commonly stabilize LLM policy updates with ratio-based clipping, but the effects of alternative divergence constraints on exploration and performance remain insufficiently principled. The paper introduces a unified framework, identifies KL3, and develops ATR-GRPO; experiments report improved stability and final performance on mathematical reasoning benchmarks. ATR-GRPO also retains GRPO’s computational efficiency while reallocating probability mass toward promising actions.

  • Problem

    Existing RLVR methods rely mainly on ratio-based clipping, while the effects of different policy-divergence constraints on exploration, stability, and performance remain insufficiently understood.

  • Method

    The paper unifies ratio- and KL-based clipping constraints, identifies KL3, and develops ATR-GRPO using its asymmetric ratio-clipping equivalence.

  • Results

    Experiments on mathematical reasoning benchmarks show that KL3-based ATR-GRPO improves training stability and final performance compared with existing state-of-the-art baselines.

  • Takeaways & Limitations

    KL3-based constraints provide a principled approximate trust-region mechanism that promotes stronger exploration while retaining GRPO-style computational efficiency.

  • Takeaways & Limitations

    The method primarily uses a static trust-region threshold and focuses on token-level objectives despite sequence-level rewards potentially producing high-variance gradients and unstable training.

Abstract

from arXiv · show

Reinforcement Learning with Verified Reward (RLVR) has emerged as a critical paradigm for advancing the reasoning capabilities of Large Language Models (LLMs). Most existing RLVR methods, such as GRPO and its variants, ensure stable updates by constraining policy divergence through clipping likelihood ratios. This paper introduces a unified clipping framework that characterizes existing methods via a general notion of policy divergence, encompassing both likelihood ratios and Kullback-Leibler (KL) divergences and extending to alternative measures. The framework provides a principled foundation for systematically analyzing how different policy divergence measures affect exploration and performance. We further identify the KL3 estimator, a variance-reduced Monte Carlo estimator of the KL divergence, as a key policy divergence constraint. We theoretically demonstrate that the KL3-based constraint is mathematically equivalent to an asymmetric ratio-based clipping that reallocates probability mass toward high-confidence actions, promoting stronger exploration while retaining the simplicity of GRPO-style methods. Empirical results on mathematical reasoning benchmarks demonstrate that incorporating the KL3 estimator into GRPO improves both training stability and final performance, highlighting the importance of principled policy divergence constraints in policy optimization.

1. Introduction

RL provides a general framework for optimizing complex, nondifferentiable LLM objectives, but policy-divergence constraints create an unresolved exploration–stability design problem. This paper unifies such constraints, identifies KL3 as a principled basis for ATR-GRPO, and reports improved stability and competitive performance.

  • RL enables LLM optimization over complex, nondifferentiable objectives beyond direct supervised learning, including mathematical reasoning and code generation.
  • GRPO offers a memory-efficient alternative to PPO by using group-normalized returns as advantage baselines.
  • Existing ratio-based clipping stabilizes updates but leaves the exploration–stability effects of alternative policy-divergence definitions insufficiently understood.
  • The unified clipping framework characterizes likelihood-ratio and KL-divergence constraints under a general policy-divergence notion that extends to alternative measures.
  • KL3 yields an asymmetric ratio-based clipping equivalent that reallocates probability mass toward high-confidence actions, promoting stronger exploration while retaining GRPO efficiency.
  • ATR-GRPO enhances learning stability and achieves performance competitive with various state-of-the-art baselines.

2. Related Work

Prior work develops increasingly scalable and exploratory policy-optimization methods, but existing clipping approaches differ in heuristic asymmetry, KL computation, and computational cost. ATR-GRPO is positioned as a principled approximate trust-region alternative within this landscape.

  • GRPO reduces memory demands relative to PPO by avoiding a separate critic, while RLOO reduces variance through a Leave-One-Out reward baseline.
  • PPO approximates trust-region optimization with first-order ratio-based clipping, while TRPO uses an explicit KL constraint but requires computationally expensive second-order optimization.
  • Recent methods address exploration and credit assignment through asymmetric clipping, dynamic clipping, KL-based clipping, or semantic-entropy reward shaping.
  • ATR-GRPO is presented as a principled approximate trust-region constraint with low computational cost, contrasting with heuristic ratio clipping and full-expectation KL methods.

3. Preliminaries

The preliminaries formulate language generation as an MDP and describe clipped policy optimization through token-level likelihood ratios and advantage estimates. They distinguish ratio-based and KL-based constraints while emphasizing clipping’s role in limiting policy updates.

  • Language generation is modeled as an MDP whose states combine the query with the partial response, whose actions are vocabulary tokens, and whose reward is defined over query–response concatenations.
  • The policy maximizes an objective over the query distribution and generated responses, with a reference policy and KL regularization controlling policy drift.
  • The clipped surrogate objective uses a token-level likelihood ratio and an advantage estimate to constrain policy optimization.
  • The likelihood ratio compares current and old-policy token probabilities, while the advantage is group-normalized in GRPO and generalized in PPO.
  • Clipping prevents excessively large policy updates and supports training stability; PPO and GRPO use symmetric ranges, whereas DAPO uses asymmetric ranges.
  • Ratio-based and KL-based clipping induce distinct objectives, denoted Lratio and LKL, because KL-based constraints involve an expectation over the full action space.

4. Method

The method unifies ratio-based and KL-based clipping through a general policy-divergence constraint, then uses the KL3 estimator to derive ATR-GRPO. KL3 yields an asymmetric, theoretically grounded clipping rule that avoids full KL expectations while supporting stable updates.

  • Unified Clipping Framework: The unified clipping operator represents policy-divergence constraints through a flexible sample-level constraint function.It can accommodate likelihood ratios, KL divergences, and arbitrary alternative measures.
  • Unified Clipping Framework: Ratio-based and KL-based objectives are recovered as special cases of the unified formulation.KL constraints correspond to explicit trust-region objectives, whereas ratio constraints reproduce the standard ratio-clipping gradient and implicitly restrict updates.
  • KL3 Estimator: Full-expectation KL constraints are often intractable for LLMs because the action space is extremely large.The method therefore turns to Monte Carlo estimators of KL divergence.
  • KL3 Estimator: KL3 is a non-negative, lower-variance, sample-level estimator that locally approximates KL divergence near the identity ratio.These properties make it a lightweight trust-region surrogate for large-scale policies.
  • ATR-GRPO: KL3-based constraints are equivalent to ratio constraints with specific hyperparameters and always induce asymmetric clipping with greater upper-range deviation.This asymmetry is characterized exactly rather than introduced as a heuristic.
  • ATR-GRPO: ATR-GRPO integrates KL3 clipping into GRPO to enable stable updates without explicitly computing full KL expectations.It uses the KL3-induced likelihood-ratio range within the unified framework.

5. Theoretical Analysis

The theoretical analysis compares ATR-based and symmetric ratio-based clipping under events where the approximate trust-region constraint is satisfied or violated. It links entropy differences to advantage–log-likelihood covariance and describes regime-dependent exploration behavior.

  • Setup and Assumptions: The analysis models GRPO and ATR-GRPO with full-batch policy gradients and a softmax policy.It examines representative events corresponding to satisfying or unsatisfying the KL3 approximate trust-region constraint.
  • Policy Logits Difference Analysis: Theorem 5.1 characterizes policy-logit differences between ATR-based and ratio-based clipping under the two constraint events.The comparison is defined using the updated policy logits produced by each clipping method.
  • Entropy Difference Analysis: The entropy difference between ATR-based and ratio-based clipping is related to covariance between the advantage and log-likelihood.This covariance determines how the methods differ in exploration behavior.
  • Entropy Difference Analysis: When the approximate trust-region constraint is violated, ATR-based clipping maintains higher entropy for advantageous high-probability actions.This preserves entropy in high-confidence regions and avoids prematurely amplifying low-confidence tails.
  • Entropy Difference Analysis: When the approximate trust-region constraint is satisfied, ATR-based clipping increases entropy for advantageous but low-probability actions.The resulting behavior reallocates probability mass toward promising underexplored actions while retaining the approximate constraint.

6. Experiments

Experiments evaluate ATR-GRPO against clipping baselines across Qwen3 models and mathematical reasoning benchmarks, then examine stability, threshold sensitivity, estimator choice, and test-time sampling. ATR-GRPO generally achieves strong performance and stable learning, while results identify δ = 0.07 as a favorable threshold and KL3 as the best tested estimator.

  • Experiment Setting: ATR-GRPO is compared with Clip-Higher, Dynamic Clipping, Clip-Cov, and Soft Gate using a standardized Dr.GRPO backbone.Experiments use Qwen3-1.7B and Qwen3-8B, DAPO-Math-17k, sparse binary rewards, and LoRA; evaluations cover AMC2023, AIME2024, and AIME2025.
  • Performance Comparison: 22.93% Mean@8 and 42.18% Pass@8 are achieved by ATR-GRPO on Qwen3-1.7B, exceeding the best baseline on final average performance.On AIME2025, final Mean@8 is 13.75% versus 13.33% for Soft Gate.
  • Performance Comparison: 56.02% Mean@8 and 80.72% Pass@8 are achieved on AMC2023 with Qwen3-8B, while AIME2024 reaches 25.42% Mean@8 and 50.00% Pass@8.On AIME2025, ATR-GRPO obtains the highest final average Mean@8 at 33.67%.
  • Learning Stability and Efficiency: 0.36 return is reached by ATR-based clipping after surpassing baselines around 400 steps, while baselines plateau near 0.28.Evaluation curves likewise consistently favor ATR-based clipping, indicating more efficient performance improvement in this comparison.
  • Learning Stability and Efficiency: ATR-based clipping maintains moderate entropy and steadily decreases completion length, unlike baselines with oscillatory or potentially premature behavior.These curves are presented alongside return and evaluation performance in Figure 2, whose training curves use a 100-step moving average.
  • Ablation Studies: δ = 0.07 maximizes Mean@8 and Pass@8, whereas larger thresholds sharply degrade performance and smaller thresholds can hinder improvement.The experiments attribute excessive degradation at large δ to training instability or policy collapse.
  • Ablation Studies: KL3 consistently provides the performance gains among the evaluated KL estimators, and ATR-GRPO remains strongest against finely tuned Clip and Clip-Higher baselines.Table 3 reports final-checkpoint means and standard deviations from three independent runs per method.
  • Test-time Sample Budget: 24.36% Pass@1 increases to 56.29% Pass@128, while Mean@K remains stable around 0.23 as the test-time sample budget grows.These results are reported for ATR-GRPO on Qwen3-1.7B.

7. Conclusion

The paper proposes a unified policy-divergence clipping framework and ATR-GRPO, which uses KL3 to support more directed exploration while retaining GRPO’s computational efficiency. Experiments on verifiable mathematical reasoning benchmarks report improved training stability and final performance over existing baselines.

  • ATR-GRPO generalizes policy divergence constraints across ratio-based and KL-based clipping measures.The framework can also extend to arbitrary alternative measures.
  • KL3-based clipping reallocates probability mass toward promising actions instead of passively truncating updates through symmetric ratio clipping.The paper’s theoretical analysis characterizes this as more principled and effective exploration.
  • ATR-GRPO preserves the computational efficiency of GRPO while targeting exploration through the policy-divergence constraint.
  • ATR-GRPO improves training stability and final performance across multiple verifiable mathematical reasoning benchmarks compared with existing baselines.

A.1. Equivalence and Asymmetry Analysis

This analysis characterizes KL3-based clipping through asymmetric trust-region boundaries. Because the divergence grows faster below a likelihood ratio of one, the upper boundary lies farther from one than the lower boundary at the same divergence level.

  • Equivalence and Asymmetry: Theorem A.1 defines lower and upper clipping ranges using a positive trust-region threshold δ.
  • Equivalence and Asymmetry: The KL3 clipping-boundary equation is solved using the Lambert W function with z = −e^−1−δ.The two real-valued Lambert branches W0 and W−1 correspond to the two real solutions.
  • Equivalence and Asymmetry: KL3 divergence increases more rapidly for wt(θ) < 1 than for wt(θ) > 1.Its derivative is positive and uniformly bounded by 1 in the analyzed expression.
  • Equivalence and Asymmetry: At the same divergence level δ, the solution above one deviates farther from one than the solution below one.This establishes the asymmetry of the KL3-induced clipping range.

A.2. Policy Logits Difference Analysis

The analysis compares ATR-based and ratio-based clipping through policy-logit updates, clipped surrogate objectives, and entropy differences. The supplied passages identify these comparison targets but do not state the resulting theorem conclusions.

  • Policy Logits Difference Analysis: Theorem A.2 defines Δθs,a as the policy-logit difference between ATR-based and ratio-based clipping.The comparison is framed for a policy-gradient algorithm with learning rate η and state visitation distribution dπθold(s).
  • Policy Logits Difference Analysis: The proof compares the clipped surrogate objectives for ratio-based clipping and ATR-based clipping.
  • Policy Logits Difference Analysis: The ratio-based comparison uses the symmetric interval (1 − ϵ, 1 + ϵ).
  • Policy Logits Difference Analysis: Theorem A.3 defines ΔH as the entropy difference between ATR-based and ratio-based clipping after the update.The proof begins with a first-order Taylor expansion of policy entropy.

B. Implementation Details

The implementation uses standard software implementations, formatting-oriented supervised fine-tuning, and single-GPU training runs for two Qwen3 model sizes. Hyperparameters are documented in Table 4.

  • Implementation Details: All methods use supervised fine-tuning on OpenMathReasoning for formatting before the RLVR process.
  • Implementation Details: The implementation adopts official Unsloth and TRL implementations.
  • Implementation Details: ATR-GRPO averages approximately 7 hours per run for Qwen3-1.7B and 16 hours for Qwen3-8B on one NVIDIA A100 GPU.
  • Implementation Details: Table 4 reports the hyper-parameter setting used in the experiments.
Loading 2602.05494v2…