CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning

1 Introduction

• PA&HP and NA&LP updates reinforce high-probability choices and can accelerate entropy collapse, while PA&LP and NA&HP updates encourage exploration and help preserve response diversity.The paper groups updates by advantage sign and token probability to explain their differing effects on entropy.Source: §1 · p. 1
• PPO clipping mainly optimizes unclipped high-probability tokens, while clipped low-probability PA&LP and NA&LP tokens are omitted from updates.The introduction reports that high-probability tokens typically lie within the clipping interval and out-of-clip tokens are predominantly low-probability.Source: §1 · p. 2, Figure 1 · p. 2
• Omitting PA&LP tokens can restrict exploration and cause entropy collapse; DAPO's higher upper bound does not address tokens beyond that bound.PA&LP tokens beyond the importance-sampling threshold 1 + ϵ are described as often including high-entropy exploratory behaviors.Source: §1 · p. 2
• Omitting NA&LP tokens removes exploitation-guiding gradients, forcing excessive exploration and potentially causing entropy explosion.Source: §1 · p. 2
• The proposed control strategy combines gradients outside the clipping range, weighting PA&LP and NA&LP gradients differently to balance exploration and exploitation.The introduction says greater weight on PA&LP and less on NA&LP helps maintain exploration and improve performance over strong baselines.Source: §1 · p. 2
• CE-GPPO regulates out-of-clip gradients to maintain high, stable policy entropy, with the introduction reporting empirical entropy coordination and hyperparameter robustness.The paper also claims theoretical and empirical evidence that using clipped-token gradients need not cause excessive deviation from the old policy or unstable training.Source: §1 · pp. 2–3

2.1 Policy Optimization Algorithms

• PPO constrains policy-update magnitude through clipping to balance training stability and sample efficiency.Its objective uses importance ratios and an advantage estimate, with ϵ setting the clipping range.Source: §2.1 · p. 3

Group Relative Policy Optimization (GRPO)

• GRPO estimates advantages by normalizing rewards across sampled responses and uses a clipped policy-gradient objective.The approach is described as preserving gradient reliability in sparse-reward settings while avoiding critic approximation errors.Source: Group Relative Policy Optimization… · p. 3
• DAPO combines asymmetric clipping ranges with dynamic sample filtering and token-level loss aggregation with reward shaping.Its filtering discards batches in which all responses share identical correctness, while aggregation addresses response-length variation.Source: Group Relative Policy Optimization… · p. 3

2.2 Policy Entropy in Reinforcement Learning

• Policy entropy reflects exploration versus exploitation, and entropy regularization weights that objective through α.A larger α encourages exploration by maintaining higher entropy; a smaller α focuses more on exploitation, while clip-higher is also described as preventing premature entropy collapse.Source: §2.2 · pp. 3–4

3.1 Impact of Clipped-Token Gradients on Entropy Dynamics

• DAPO’s clip-higher strategy extends the upper clipping bound to include some higher-entropy tokens, but out-of-clip gradients remain relevant to entropy dynamics.The discussion identifies entropy collapse and explosion as forms of instability associated with policy clipping.Source: §3.1 · p. 4
• Out-of-clip gradients influence entropy in opposite directions: PA&LP gradients slow entropy decline, while NA&LP gradients accelerate convergence and reduce entropy.The analysis identifies out-of-clip gradients as consequential for dynamically balancing exploration and exploitation.Source: §3.1 · p. 4

3.2 Gradient-Preserving Clipping Policy Optimization

• CE-GPPO uses stop-gradient weighting to preserve out-of-clip gradients, with β1 scaling PA&LP gradients and β2 scaling NA&LP gradients.Larger β1 slows entropy decline and promotes exploration; larger β2 accelerates convergence and reduces entropy to facilitate exploitation. Setting both coefficients to zero recovers PPO.Source: §3.2 · pp. 4–5

3.3 Ensuring Stable Optimization in CE-GPPO

• Outside the clipping interval, CE-GPPO bounds gradient contributions by β1 · (1 −ϵ) or β2 · (1 +ϵ), supporting optimization stability comparable to PPO.The method also retains PPO’s computational and memory complexity through element-wise loss reweighting without additional forward passes or auxiliary components.Source: §3.3 · p. 5

4.1 Experimental Setup

• Training used KlearReasoner-MathSub-30K, two model sizes, up to 1,000 steps, and evaluations across math, code, and instruction-following benchmarks.The dataset contains approximately 30k samples; training used 8 rollouts per prompt and a learning rate of 1 × 10−6.Source: §4.1 · p. 5

4.2 Main Results

• CE-GPPO outperforms baselines across benchmarks, improving over the best baseline by 2.5 points at 1.5B and 3 points at 7B.The comparison covers multiple benchmarks and model sizes.Source: §4.2 · pp. 5–6, Table 1 · p. 6
• Native GRPO suffers entropy collapse, while DAPO and CE-GPPO mitigate it; CE-GPPO maintains stable entropy and improves over DAPO.CE-GPPO propagates clipped PA&LP-token gradients in a bounded, moderate manner.Source: §4.2 · p. 6, Figure 2 · p. 6
• The analysis also identifies backpropagating clipped NA&LP-token gradients as resolving entropy explosion, while CE-GPPO coordinates exploration and exploitation.The comparison figure tracks entropy dynamics and AIME25 accuracy for GRPO, DAPO, and GPPO.Source: §4.2 · p. 6

5.1 Impact of Different β Hyperparameters on Entropy Dynamics

• Entropy falls faster with larger β1 or smaller β2: β1 amplifies mainly NA&LP gradients, while β2 amplifies mainly PA&LP gradients.The former strengthens exploitation and can hasten entropy collapse; the latter encourages exploration and slows entropy reduction.Source: §5.1 · pp. 6–7

5.2 Entropy-Guided Training Dynamics

• Relatively high, stable entropy supports sustained improvement, but excessively high entropy need not improve performance; β2-weighted PA&LP gradients and smaller β1 weights are beneficial.The reported settings β1 = 0.5, β2 = 1 and β1 = 0.75, β2 = 1 improve performance across model sizes.Source: §5.2 · p. 7

5.3 Training Stability Analysis of CE-GPPO

• CE-GPPO maintains stable KL divergence and gradient-norm trends while adding bounded gradients from outside PPO’s trust region.The authors report no abrupt fluctuations or abnormal values beyond reasonable ranges.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4 Comparison with Other RL Algorithms

• CE-GPPO achieves the best performance on 4 of 5 mathematical reasoning datasets in comparisons with CISPO and GSPO.It also shows advantages over GSPO on AIME2025, HMMT25, and MATH500, with a higher average score.Source: §5.4 · pp. 7–8, Table 2 · p. 8
• CISPO’s performance and entropy decline sharply later in training, whereas CE-GPPO improves steadily; GSPO clips nearly 15% of tokens from gradient updates.The paper attributes CE-GPPO’s comparison with GSPO partly to preserving gradients across more tokens.Source: §5.4 · p. 8

5.5 Comparison with Existing Entropy Collapse Mitigation Methods

• Entropy regularization is coefficient-sensitive: α = 0.001 still leads to entropy near 0.06 and a benchmark performance drop, while α = 0.003 triggers entropy explosion and substantial degradation.CE-GPPO instead shows a slow, stable entropy decline and superior AIME25 performance among the compared mitigation strategies.Source: Figure 5 · p. 8, §5.5 · p. 9

5.6 The Role of Entropy at Different Stages of Training

• Maintaining high, stable entropy early and gradually reducing it later supports exploration first and later stabilizes performance.Switching from β1 = 0, β2 = 1 to β1 = 0.5, β2 = 1 at step 585 stabilizes and reduces later entropy, with further performance improvement.Source: §5.6 · p. 9

6 Conclusion

• CE-GPPO incorporates out-of-clip gradient signals with stop-gradient weighting and tunable scaling to control policy entropy while preserving training stability.The conclusion reports experiments comparing it with GRPO, DAPO, CISPO, and GSPO.Source: §6 · p. 9
• The β1 = 0, β2 = 1 to β1 = 0.5, β2 = 1 configuration transition occurs at step 585; CE-GPPO preserves training stability, controls policy entropy, and achieves superior performance to strong baselines.The cited passage also says CE-GPPO prevents entropy collapse and avoids excessive exploration.Source: Figure 6 · p. 9, §6 · p. 9

Limitations

• Model-specific entropy patterns mean that β1 = 0.5, β2 = 1 is generally effective but optimal performance still requires some tuning.Some models begin RL training with high entropy, while others begin with low entropy.Source: Limitations · p. 9

• GRPO uses ϵ = 0.2 for both upper and lower clipping bounds.Source: §A.1 · p. 12
• DAPO sets its lower and upper clipping thresholds to ϵl = 0.2 and ϵh = 0.28, respectively.Source: §A.1 · p. 12
• CISPO clips importance-sampling weights directly rather than clipping the final policy update, with ϵ = 0.2 for both bounds.Source: §A.1 · p. 12
• GSPO uses a sequence-level importance ratio and sets ϵl = 0.0003 and ϵh = 0.0004.Source: §A.1 · p. 12

A.2.1 Problem Setup and Assumptions

• The setup defines policy entropy at state s for a stochastic policy πθ(a | s), where s is a state and a an action.Source: §A.2.1 · p. 12
• The appendix frames these conditions as standard assumptions for the entropy-change analysis.Source: §A.2.1 · p. 12
• The analysis assumes a tabular softmax policy parameterized by logits zs,a for state-action pairs.Source: §A.2.1 · p. 12
• Policy updates use the policy gradient theorem with a sufficiently small learning rate η > 0 for first-order Taylor approximations.Source: §A.2.1 · p. 12
• The advantage function is assumed to be baseline-centered, consistent with computing advantages as Q(s, a) − V(s).Source: §A.2.1 · p. 12
• The stated goal is to approximate the entropy change between successive policy updates.Source: §A.2.1 · p. 12

A.2.2 Derivation

• A first-order Taylor expansion of entropy with respect to logits provides the approximation for entropy change between policy updates.Source: §A.2.2 · p. 12
• The derivation computes entropy’s gradient with respect to logits under the softmax parameterization.Source: §A.2.2 · p. 12
• With the policy gradient theorem and baseline-centered advantages, the logit update is substituted into the entropy expansion to express the change as a covariance.Source: §A.2.2 · p. 13

A.3 Proof of the Gradient of CE-GPPO Objective

• The gradient is piecewise: it uses sg(δ)δÂi,t for the specified clipped cases and δÂi,t otherwise.Source: §A.3 · p. 13
• The proof defines δ as the importance-sampling ratio between the updated and reference policies.Source: §A.3 · p. 13
• For clipped tokens, stop-gradient keeps the scaling terms from propagating gradients through δ.Source: §A.3 · p. 13
• The proof also gives clipped-case gradient coefficients β2(1 + ϵ)Âi,t and δÂi,t for the displayed cases.Source: §A.3 · pp. 13–14