PaperlayerView exact revision on arXiv

Source-linked AI summary

Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective

Kun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao

arXiv:2602.09782v2cs.LGcs.AIcs.CL

TL;DR

RLVR training can collapse LLM policy entropy, producing premature overconfidence, reduced diversity, and constrained gradients, while existing clipping-based control is largely static. The paper links ratio regions to entropy changes, uses dynamic clipping thresholds, and evaluates ID, DID, and OD strategies. Across multiple benchmarks and settings, the strategies mitigate entropy collapse and improve entropy control and performance.

  • Problem

    Existing entropy-control work largely treats clipping thresholds statically and lacks a theoretical link between Gradient-Preserving Clipping and precise entropy control.

  • Method

    The paper analyzes importance-sampling-ratio regions, dynamically modulates clipping thresholds, and designs ID, DID, and OD entropy-control strategies.

  • Results

    The proposed strategies mitigate entropy collapse and improve entropy control and model performance across multiple benchmarks, non-math domains, and a different model family.

  • Takeaways & Limitations

    Dynamic entropy control supports flexible training schedules, with ID, DID, and OD targeting different model or training regimes.

  • Takeaways & Limitations

    The theoretical region analysis includes separate positive- and negative-advantage assumptions.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a critical method for enhancing the reasoning capabilities of Large Language Models (LLMs). However, continuous training often leads to policy entropy collapse, characterized by a rapid decay in entropy that results in premature overconfidence, reduced output diversity, and vanishing gradient norms that inhibit learning. Gradient-Preserving Clipping is a primary factor influencing these dynamics, but existing mitigation strategies are largely static and lack a framework connecting clipping mechanisms to precise entropy control. This paper proposes reshaping entropy control in RL from the perspective of Gradient-Preserving Clipping. We first theoretically and empirically verify the contributions of specific importance sampling ratio regions to entropy growth and reduction. Leveraging these findings, we introduce a novel regulation mechanism using dynamic clipping thresholds to precisely manage entropy. Furthermore, we design and evaluate dynamic entropy control strategies, including increase-then-decrease, decrease-increase-decrease, and oscillatory decay. Experimental results demonstrate that these strategies effectively mitigate entropy collapse and achieve superior performance across multiple benchmarks.

1 Introduction

RLVR improves LLM reasoning but uncontrolled training can cause policy entropy collapse, while existing entropy-control work lacks a theoretical clipping-based framework and strategy design.

  • Motivation: Uncontrolled continuous RLVR training may drive policy entropy toward near-zero, causing premature overconfidence and reduced output diversity.Entropy collapse can trap the model in a locally optimal solution.
  • Motivation: Policy entropy constrains training-gradient norms, inhibiting continued improvement during later RL training and affecting final performance.
  • Clipping perspective: Gradient-Preserving Clipping significantly influences policy entropy dynamics during RLVR training.
  • Clipping perspective: Harsh clipping can ignore low-probability points, reducing output diversity and contributing to entropy decline; Clip-Higher instead raises exploration-token probabilities.DAPO uses a higher upper clipping threshold to increase policy entropy.
  • Research questions: The paper addresses how to precisely regulate entropy and design entropy-control strategies for effective RL training.
  • Contributions: The proposed framework analyzes ratio-region contributions, dynamically modulates clipping thresholds, and evaluates ID, DID, and OD entropy strategies across benchmarks.The reported goal is mitigating entropy collapse while improving entropy control and model performance.

2 Preliminary

PPO-Clip limits policy-ratio changes to approximate a trust region, while policy entropy measures uncertainty and is linked to output diversity, gradients, and performance.

  • PPO and GRPO: PPO-Clip clips the policy update ratio to approximate TRPO’s trust-region constraint and prevent drastic policy drift during individual updates.
  • PPO and GRPO: The importance sampling ratio r_t(θ) is the current-policy probability divided by the old-policy probability and is constrained by the PPO-Clip objective.
  • PPO and GRPO: PPO commonly uses ε_low = ε_high = 0.2 as its clipping thresholds.
  • PPO and GRPO: GRPO replaces separate Critic-based advantage estimation with a group-based method to improve training efficiency and stability.
  • Policy entropy: Policy entropy characterizes uncertainty at the current decision point, equivalently describing the flatness of the policy.
  • Policy entropy: Prior work reports a negative exponential relationship between policy entropy and performance, while entropy collapse restricts update gradients and degrades final performance.

3 Theoretical and Empirical Investigations

The paper analyzes how token-specific RL updates align with entropy gradients, identifying four ratio regions with distinct entropy effects and validating them experimentally.

  • Theoretical analysis: Theoretical analysis examines the angle between RL-training and entropy gradients to identify how four importance-sampling-ratio regions affect entropy.The effects are also tested empirically.
  • Theoretical analysis: The single-token surrogate objective uses the current policy, sampling policy, and advantage function to characterize an RL update.
  • Theoretical analysis: The policy is parameterized by logits through softmax, with the token gradient using a one-hot token vector and the full vocabulary probability vector.
  • Entropy-gradient criterion: The entropy gradient is ∇_zH = −p ⊙(ln p + H · 1), and its inner product with the objective gradient determines whether an update raises or lowers entropy.
  • Positive-advantage regions: For positive advantage, E1 decreases entropy when pa(ln pa + H) exceeds B(p), whereas E2 increases entropy when it falls below B(p).These conditions are typically associated with lower surprisal than H for E1 and higher surprisal than H for E2.
  • Negative-advantage regions: For negative advantage, E3 and E4 both increase entropy under complementary token-specific conditions relative to B(p).E3 corresponds to pa(ln pa + H) > B(p), while E4 corresponds to pa(ln pa + H) < B(p).
  • Empirical validation: Experiments apply Gradient-Preserving Clipping separately within the extended trust region 0.7 < r < 1.3 and compare the resulting entropy dynamics.The regions distinguish high- and low-probability states while unaffected tokens continue training normally.

4 Methodology

The method dynamically adjusts PPO-Clip’s upper and lower thresholds according to token probability and advantage sign, then schedules entropy control across training. It uses these mechanisms to promote or suppress entropy through ID, DID, and oscillatory strategies.

  • Dynamic clipping threshold regulation: Dynamic clipping thresholds selectively encourage or constrain policy updates using current token probability and advantage sign.The thresholds are probability-dependent and use stop-gradient to avoid introducing additional gradients through the clipping mechanism.
  • Dynamic clipping threshold regulation: The upper threshold is increased for low-probability positive tokens and decreased for high-probability tokens, encouraging exploration without over-optimizing dominant tokens.This expands updates in E2 while constraining E1.
  • Dynamic clipping threshold regulation: The lower threshold is reduced for high-probability negative tokens to limit distribution shifts caused by penalizing sampled tokens and unintentionally boosting alternatives.The mechanism expands E4 while constraining E3, thereby targeting entropy decrease without destabilizing high-probability regions.
  • Increase-then-Decrease: The ID strategy transitions from dynamic upper-threshold control to dynamic lower-threshold control at the midpoint of training.During Phase I, the upper threshold anneals toward ϵstd while the lower threshold remains fixed; during Phase II, the lower threshold transitions toward dynamic M(p).
  • Decrease-Increase-Decrease: The DID strategy first decreases entropy, then uses clipping to increase entropy before controlling convergence through lower-threshold scheduling.Its phases transition the upper threshold from ϵstd to dynamic H(p), then the lower threshold from ϵstd to dynamic M(p).
  • Oscillatory Decay: The oscillatory decay strategy switches between entropy-increasing and entropy-decreasing modes using evolving thresholds and a discrete control state.The lower entropy bound is defined as 0.2Hinit; the state triggers boost, suppression, or holding based on the current entropy.

5 Experiments

The experiments evaluate dynamic entropy-control mechanisms and three phase-based training strategies across mathematical, non-mathematical, and model-family settings. The results support effective entropy regulation and consistent performance gains, with strategy suitability depending on the training regime.

  • 5.2 Experimental Results and Analysis: Dynamic upper and lower clipping thresholds effectively control entropy increase and decrease during training.The experiments specifically evaluate dynamic threshold modulation and report effective control over both directions of entropy change.
  • 5.2 Experimental Results and Analysis: Three entropy-control strategies consistently improve performance across multiple mathematical benchmarks.The strategies are evaluated on AIME24, AIME25, GSM8k, AMC, MATH-500, and Olympiad.
  • 5.2 Experimental Results and Analysis: The method achieves consistent gains on non-mathematical benchmarks and outperforms GRPO on most Phi-4-14B benchmarks.The additional evaluation covers GPQA-Diamond, LiveCodeBench, and MMLU-Redux, while Phi-4-14B is trained for 200 steps.
  • 5.2 Experimental Results and Analysis: ID suits SFT-aligned models, DID suits base or high-entropy models, and OD suits long training or potential local minima.ID emphasizes early diversity, DID begins with entropy reduction, and OD periodically maintains exploration.
  • 5.4 Ablation Study: Phase-ratio experiments vary the entropy-control phase among 0.3, 0.4, 0.5, and 0.6 while holding other settings unchanged.Qwen2.5-Math-7B is trained for 200 steps in this analysis of ID and DID.
  • 5.4 Ablation Study: The linear probability-dependent clipping function is selected for monotonicity, numerical stability, and computational simplicity.Additional ablations report that it consistently outperforms or matches the exponential variant.

6 Conclusions

The paper studies dynamic entropy control in RLVR from a gradient-preservation perspective to address entropy collapse. It introduces three training strategies and reports that theoretical analyses, training curves, and extensive evaluations validate their effectiveness.

  • 6 Conclusions: The paper addresses entropy collapse in RLVR through dynamic entropy control viewed from gradient preservation.The stated research issues are precise entropy regulation and entropy-control strategy design.
  • 6 Conclusions: It introduces increase-then-decrease, decrease-increase-decrease, and oscillatory decay strategies for RLVR training.These strategies target different entropy trajectories during training.
  • 6 Conclusions: The authors report validation through theoretical analyses, training-curve analyses, and extensive performance evaluations.The conclusion characterizes these analyses as validating the effectiveness of the proposed method.
  • 6 Conclusions: Prior approaches include entropy regularization and clipping modifications such as Clip-Higher to mitigate entropy decline.Clip-Higher uses a larger upper clipping threshold to avoid clipping low-probability tokens and increase entropy.

B.1 Proof of eq. (5)

The proof analyzes how the policy-gradient update aligns with the entropy gradient by expressing both gradients with respect to the logits. Their inner product serves as a diagnostic for the direction of entropy change.

  • B.1 Proof of eq. (5): The objective-gradient and entropy-gradient inner product measures alignment between the learning signal and entropy growth.The proof formulates this quantity as proportional to ˆA(e_a − p)^T[−p ⊙(ln p + H·1)].
  • B.1 Proof of eq. (5): The entropy-gradient expression combines token-specific and global-baseline terms involving p_x, ln p_x, and H.The proof expands the entropy derivative through the softmax Jacobian and chain rule.
  • B.1 Proof of eq. (5): The policy-gradient loss for a sampled action has logit gradient ∇_zL = ˆA(e_a − p).Here e_a is the one-hot vector for the sampled action and p is the policy distribution.
  • B.1 Proof of eq. (5): The token-specific approximation agrees in sign with the full expression for 91.7% of sampled updates on average.This supports using the approximation as an empirical diagnostic criterion.

C.1 Models and Datasets

The evaluation uses Qwen and Phi models with public mathematical and non-mathematical benchmarks, GRPO-based training, and standardized implementation settings. Computational cost is also measured for 400-step runs on eight H100 GPUs.

  • C.1 Models and Datasets: The benchmark evaluation uses Qwen2.5-Math-7B and Qwen2.5-7B trained on DAPO-Math data.The primary benchmarks are AIME24, AIME25, AMC, MATH-500, GSM8k, and Olympiad.
  • C.1 Models and Datasets: Additional evaluations use GPQA-Diamond, LiveCodeBench, MMLU-Redux, and Phi-4-14B to test broader generality.Phi-4-14B experiments are conducted for 200 steps.
  • C.1 Models and Datasets: Training uses GRPO advantages, 400 steps, eight sampled responses per prompt, global batch size 512, and learning rate 1 × 10^-6.AdamW, weight decay 0.1, gradient clipping 1.0, and zero KL penalty are also specified.
  • C.1 Models and Datasets: Dynamic clipping parameters are calibrated differently across low- and high-probability regimes.The upper-threshold slope and intercept are −0.25 and 0.5; the lower-threshold values are −0.13 and 0.3.
  • C.1 Models and Datasets: Experiments run on one node with 8 × H100 GPUs using hybrid parallelism through verl, with vLLM inference and sharded training.The evaluation pipeline uses EvalScope and lmdeploy across eight GPUs.
  • C.4 Computational Cost Analysis: Training time is compared across methods for 400 steps on 8× 80GB H100 GPUs.Reported times use Hours:Minutes or Days:Hours:Minutes formatting.

D.2 Benchmarks and Metrics

The evaluation uses mathematical reasoning benchmarks with mean_and_pass_at_k, while sample counts vary by dataset. Training curves compare entropy and average clipping thresholds across dynamic-threshold and entropy-control methods.

  • Evaluation covers a suite of mathematical reasoning benchmarks using mean_and_pass_at_k.
  • Sampling uses 32 outputs for AMC and AIME, 16 for GPQA-Diamond, 4 for MATH-500 and OlympiadBench, 2 for GSM8K, and 1 for LiveCodeBench and MMLU-Redux.
  • All mathematical reasoning tasks use a chain-of-thought prompt requiring step-by-step reasoning and a boxed final answer.
  • Training curves are recorded for dynamic lower and upper clipping thresholds and the Ours-ID, Ours-DID, and Ours-OD methods.
  • Larger average upper clipping thresholds coincide with increasing entropy, while decreasing upper and lower thresholds coincide with decreasing entropy.

E.2 Analysis of Clipping Function

The paper evaluates a probability-dependent linear clipping schedule against an exponential alternative. The linear schedule matches or outperforms the nonlinear variant while offering monotonicity, numerical stability, and computational simplicity, though fixed schedules limit flexibility.

  • The linear clipping schedule makes the threshold decrease monotonically with reference probability, supporting theoretical consistency and simple calibration.Its probability-dependent form uses the negative correlation suggested by the theoretical analysis.
  • Linear decay produces a constant, predictable contraction of the effective trust region, avoiding overly aggressive shrinkage for medium- and high-probability tokens.
  • The ablation compares linear and exponential schedules under upper-bound-only dynamic clipping after calibrating identical boundary values.
  • The linear schedule consistently outperforms the nonlinear variant across all Pass@K metrics in the upper-bound-only setting.
  • Across Ours-ID and Ours-OD on Qwen2.5-Math-7B, the linear schedule achieves stronger or comparable performance on most benchmarks.
  • Fixed functional forms and hyperparameters may not provide the flexibility intended by probability-dependent clipping, motivating adaptive or learnable schedules.

E.3 Analysis of Clipping Probability Curve

The clipping probability tracks the paper’s entropy-control strategy during training. Ours-ID links low clipping probability to entropy increase early and high clipping probability to entropy decrease later, while Ours-OD produces entropy fluctuations.

  • Clipping probability denotes the proportion of training tokens affected by the clipping mechanism.
  • In Ours-ID, low token clipping probability during early training coincides with an entropy-increase phase.
  • In Ours-ID, high token clipping probability during later training coincides with an entropy-decrease phase.
  • Ours-OD exhibits entropy fluctuations throughout training, consistent with its oscillatory control behavior.

E.4 Experiment on Replacing Dynamic Clipping Threshold with Clip-Higher and Clip-Lower

The experiment replaces dynamic upper and lower thresholds with Clip-Higher and Clip-Lower. Although these methods influence entropy direction, their one-size-fits-all treatment does not precisely realize the intended entropy trajectories and can leave entropy collapse unresolved.

  • The experiment applies Clip-Higher and Clip-Lower instead of dynamic thresholds within the paper’s entropy-control strategies.
  • Clip-Higher and Clip-Lower may affect model performance because they treat tokens across different regions uniformly without dynamic adjustment.
  • The fixed alternatives fail to precisely produce the intended increase-then-decrease or decrease-increase-decrease entropy patterns.
  • Entropy collapse remains a problem when dynamic clipping thresholds are replaced by Clip-Higher and Clip-Lower.

F Implementation Code Example

The implementation replaces fixed GRPO clipping with probability-dependent, annealed bounds or entropy-state-dependent bounds while preserving the clipped objective. It also documents broader-impact risks and checklist-supported theoretical and reproducibility considerations.

  • Ours-ID: Ours-ID replaces fixed scalar clipping with annealed, token-dependent bounds computed from detached current-policy probabilities.The upper and lower bounds use linearly annealed coefficients, preventing gradients from flowing through the clipping bounds.
  • Implementation: The Ours-ID code computes a clipped likelihood ratio, linearly interpolates coefficients by training progress, and returns a response-masked mean loss.The likelihood ratio is formed from clipped log-probability differences before dynamic bounds are applied.
  • Ours-OD: Ours-OD uses an entropy-driven state machine that restricts probability increases at high entropy and relaxes the upper bound at low entropy.Its implementation compares mean entropy with upper and lower thresholds before selecting adaptive clipping bounds.
  • Scope and broader impacts: The authors report that their contribution is an entropy-control mechanism and training strategy using existing assets rather than a new dataset, model checkpoint, or software asset.The broader discussion notes that deployment should include evaluation, safety testing, monitoring, and domain-specific risk assessment.
Loading 2602.09782v2…