Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective

1 Introduction

• Entropy collapse rapidly lowers policy entropy, restricting exploration and making group-based advantages less discriminative, which can cause training to stagnate.The paper links homogeneous reasoning rollouts with reduced ability to find informative, potentially correct solutions.Source: §1 · p. 1
• Prior interventions remain heuristic, while the analysis identifies four factors governing token-level entropy change and limitations in methods that adjust only some factors.Such partial adjustments can neglect tokens susceptible to severe collapse and may even accelerate collapse in some cases.Source: §1 · p. 2
• STEER adaptively reweights tokens using theoretically estimated entropy variations, down-weighting tokens susceptible to drastic entropy decay.The method is presented as a principled entropy-modulation approach grounded in the paper’s theoretical analysis.Source: §1 · p. 2
• Across six mathematical reasoning and three coding benchmarks, STEER outperforms strong baselines and generalizes across models and RL algorithms.The reported evaluations span different model scales and families, as well as GRPO, RLOO, and OPO.Source: §1 · p. 2

2.1 RLVR Algorithms

• GRPO avoids a value model by computing relative advantages within a sampled group, with the response-level advantage typically shared across its tokens.The objective is presented in token-level form, and the passage notes that its KL-divergence term is omitted.Source: §2.1 · p. 2

2.2 Policy Entropy of LLMs

• Token-level policy entropy measures generation uncertainty at each step, while global policy entropy averages token-level entropy over sampled responses.In practice, the paper estimates global entropy using the training dataset and reports that this estimate generalizes well to unseen data.Source: §2.2 · p. 3
• Entropy collapse weakens exploration and training effectiveness; in group-based methods, homogeneous rollouts make relative advantages less discriminative and can cause stagnation.Reduced exploration limits discovery of informative and potentially correct solutions.Source: §2.2 · p. 3

3 Analyses on Entropy Dynamics

• For logit-independent policies trained with GRPO, token-level entropy change is approximated with O(η^2) error and depends on clipping, advantage, token probability, and state entropy.The advantage and old-policy term act as weighting factors in the analysis.Source: §3 · pp. 3–4
• MSE is on the order of 1e-4, with lower error and higher PCC and SRCC than Cov for entropy-change estimation.Table 1 compares the Cov baseline with the paper’s estimator; the text reports orders-of-magnitude lower MSE and substantially higher correlations.Source: Table 1 · p. 4, §3 · p. 4
• The joint effect of advantage and token probability yields four entropy-change directions, whose competing updates combine to determine global policy entropy.Entropy collapse corresponds to entropy-decreasing updates in exploitation and suppression quadrants consistently overwhelming the entropy-increasing updates.Source: §3 · p. 4, Figure 2 · p. 4
• Empirical analyses of quadrant-level interventions align closely with the theoretical findings, and Figure 3 validates mechanisms for Clip-Higher and Positive-Reweighting.The paper reports that the global entropy reflects the combined effects of the four quadrant-level dynamics.Source: §3 · p. 4, Figure 3 · p. 5

Entropy Effect of Clip-Higher

• Increasing εhigh can mitigate or reverse collapse by filtering fewer low-probability positive Quadrant II samples; increasing εlow intensifies it by filtering fewer Quadrant III instances.Clip-Higher [Yu et al. (2025)] adjusts the clipping thresholds, but the interventions remain coarse-grained.Source: §4 · p. 5

Entropy Effect of Re-weighting Positives

• Unlikeliness [He et al. (2025a)] strengthens Quadrant II and weakens Quadrant I, while W-REINFORCE [Zhu et al. (2025)] down-weights positives, mainly weakening Quadrant I.In GRPO experiments, positive-only training rapidly collapsed entropy, while negative-only training sustained high entropy.Source: §4 · p. 5

Entropy Effect of Entropy-aware Advantage

• Entropy-aware advantages can amplify entropy changes in high-entropy tokens, and Entro. Adv. and GTPO decline faster than GRPO once policy entropy starts falling.The entropy-change relationship is illustrated in Figure 4; Figure 5 compares entropy dynamics on Qwen2.5-Math-7B. Methods include Entro. Adv. [Cheng et al. (2026)] and GTPO [Tan and Pan (2025)].Source: §4 · pp. 5–6, Figure 5 · p. 6, Figure 4 · p. 6

Limitations of Existing Methods

• Existing methods make qualitative adjustments to only one or two entropy-change factors, leaving other factors and their joint effects unconsidered.The resulting interventions do not capture the entropy evolution observed during training, compromising their effectiveness.Source: §4 · p. 6

5 Stabilizing Token-level Entropy-change via Reweighting

• STEER translates theoretically estimated entropy changes into adaptive token-level weights, attenuating gradient contributions for tokens susceptible to drastic entropy decay.The method aims to modulate token-level entropy changes rather than monitor only global expectations.Source: §5 · p. 6
• STEER's exponential weighting decreases with normalized absolute entropy change, keeps weights strictly positive, and uses batch-maximum normalization for numerical stability.The absolute value also limits large entropy increases, which the authors describe as potentially sub-optimal.Source: §5 · p. 6
• Table 3 reports Qwen2.5-Math-7B benchmark results as percentages, using avg@32 for AIME24, AIME25, and AMC23, and avg@1 for other datasets.Source: Table 3 · p. 7
• STEER adds only the calculation of entropy change, which is described as negligible in complexity, and can be applied to various LLM architectures and RL algorithms.Source: §5 · p. 7

6.1 Experimental Setup

• The evaluation covers six mathematical reasoning and three coding benchmarks, compares against 11 math baselines and GRPO on coding, and averages main results over two training runs.STEER's only introduced hyperparameter, λmin, is set to 0.7.Source: §6 · p. 7, §6.1 · p. 7

6.2 Main Results

• STEER leads math benchmarks, averaging 2.2 points above OPO; it also achieves the best Pass@256/512/1024 on AIME24 and AIME25 and exceeds GRPO by at least 1% on each coding model and test set.The coding comparisons use three Qwen2.5-Coder models.Source: §6.2 · p. 8, Figure 6 · p. 8, Table 5 · p. 8

6.3 Empirical Results for Entropy Modulation

• With εhigh = 5 and εlow = 0.99, STEER stabilizes after an initial entropy decline and achieves the highest accuracy across test sets, while other methods collapse, over-expand entropy, or fail to control it.The setup nearly removes ratio clipping; STEER's token weights remain near 1 for small entropy changes and decrease for large changes.Source: §6.3 · pp. 8–9, Figure 8 · p. 9, Table 6 · p. 9

6.4 Generalization Study

• On RLOO, STEER raises average score from 45.8 to 46.8 and on OPO from 46.4 to 47.5; it also improves performance across model sizes and additional backbones without backbone-specific retuning.The additional backbones are Llama-3.2-3B-Instruct for math and Mistral-7Bv0.3 for code editing.Source: §6.4 · p. 9

7 Conclusion

• The paper derives a tight approximation for token-level entropy change at each update step, identifying four governing factors and unifying explanations of existing strategies.Source: §7 · p. 9
• The analysis combines theoretical and empirical study of entropy dynamics in RLVR.Source: §7 · p. 9
• STEER adaptively reweights tokens based on theoretically estimated entropy variations.Source: §7 · p. 9

8 Limitations

• The study covers RL training with verifiable rewards, not training without verifiable rewards; limited-verifiability settings and agentic or long-horizon RL remain future directions.Source: §8 · p. 9

B.1 Reinforcement Learning with Verifiable Rewards in LLMs

• RLVR derives rewards from rule-based verification, commonly checking whether outputs match reference answers and returning binary success-or-failure signals.Source: §B.1 · p. 15

B.2 Entropy-Oriented RL Methods for LLM Reasoning

• Prior entropy interventions adjust clipping, sample weighting, KL regularization, or entropy-based advantages, but their control is broad or their mechanisms remain insufficiently explained.The cited discussion also notes that entropy-based advantages can magnify entropy change and that prior entropy-change estimation relies on a state-equivalence assumption.Source: §B.2 · pp. 15–16

C.1 A Unified Token-level Gradient Reweighting Framework

• Existing entropy interventions modify token-level gradient weights, but adjust only subsets of the four governing factors, limiting their effectiveness.Table 7 maps the interventions to their modifications of wi,t.Source: §C.1 · p. 16, Table 7 · p. 17

C.2 Theoretical Analyses Extended to Other RL Algorithms

• The policy-gradient derivation extends beyond GRPO: modifying the algorithm-specific advantage formulation yields a unified entropy-change estimator for multiple RL algorithms.The passage gives GRPO, PPO, RLOO, and OPO as examples.Source: §C.2 · p. 16

D Supplementary Empirical Analysis for RLVR Entropy

• Supplementary experiments examine policy-entropy behavior, validate the entropy-change estimator, and test quadrant interventions against GRPO.Source: D Supplementary Empirical Analysis for… · p. 16, §D.1 · p. 16, §D.2 · p. 17

D.1 Empirical Properties of Policy Entropy

• Entropy decays smoothly during standard GRPO training; its initial level and decay speed vary with model and dataset, while same-domain test sets follow similar decreasing trajectories.Source: §D.1 · pp. 16–17, Figure 10 · p. 18, Figure 11 · p. 18

D.2 Entropy Change Estimation Comparison

• On DAPO-Math-17k, our method’s estimated entropy changes show a clear positive correlation with ground-truth entropy change, while the estimation scheme in Cui et al. (2025b) shows no clear correlation.Source: §D.2 · p. 17

D.3 Influencing Entropy Dynamics by Strengthening or Weakening the Quadrants

• Intervening on 10% of tokens in each quadrant—doubling weights in entropy-increasing quadrants and masking updates in entropy-decreasing quadrants—increases policy entropy versus standard GRPO.Source: §D.3 · p. 19, Figure 15 · p. 20
• Across interventions from 5% to 20%, entropy effects align with the analysis and grow more pronounced as the intervention ratio rises.Up-weighting Quadrant II notably increases entropy; the 20% case is excluded because it produces excessively high entropy.Source: §D.3 · pp. 19–20

D.4 Entropy Effect of Clipping Operation

• Larger clip-high thresholds produce a late-stage entropy rebound, whereas clip-low at εlow = 0.1 raises entropy monotonically and εlow ≥ 0.2 drives it toward near zero.Source: §D.4 · p. 20
• Math performance degrades under both entropy collapse and entropy explosion, while stable entropy yields consistently better results.The passage attributes potential harm from excessive entropy to policy stochasticity, fewer informative reasoning rollouts, and near-zero rewards that hinder optimization.Source: §D.4 · p. 20

E.1 Detailed Information for Dataset

• Math training uses DAPO-Math-17k; coding uses ArcherCodeR3’s 6,753 tasks and an internal code-edit corpus of over 50,000 instances.Evaluation includes LiveCodeBench v5, an internal held-out code-edit split, and the Zeta “dpo” subset.Source: §E.1 · pp. 20–21

E.2 Training Details for Our Method and Baselines

• Shared training settings use batch sizes 512 and 32, eight rollouts, no entropy or KL loss, and a learning rate of 1e-6.Models train for at most 200 rollout steps; baseline-specific recipes are also described.Source: §E.2 · pp. 21–22

F.1 Ablation Study

• Binary mapping degrades performance, while linear mapping does not materially harm it, supporting continuous token-level reweighting over truncation.Table 11 reports the mapping ablation; Figure 17 illustrates the mappings.Source: Table 11 · p. 23, §F.1 · p. 23, Figure 17 · p. 23

F.2 Hyperparameter Sensitivity

• STEER performs consistently well for λmin ∈ [0.5, 0.8], while values that are too small may destabilize learning and values too large reduce entropy control.Figure 19 shows test-accuracy dynamics across λmin settings.Source: Figure 18 · p. 23, §F.2 · pp. 23–24, Figure 19 · p. 23

F Supplementary Performance Evaluation

• Supplementary experiments assess STEER’s robustness and generality under varied settings.Source: F Supplementary Performance Evaluation · p. 23

• The additional-backbone experiments retain the main STEER hyperparameter rather than retuning it per backbone.Source: §F.3 · p. 24
• 19.5→21.6 average score on an additional backbone accompanies higher entropy at Step 200 (0.22→0.85).Source: §F.3 · p. 24
• 45.8→46.8 average score on RLOO and 46.4→47.5 on OPO indicate STEER’s effectiveness is not GRPO-specific.Source: Figure 21 · p. 24, §F.3 · p. 24
• The reported metric improves from 7.64 to 8.35, while entropy at Step 200 rises from 0.04 to 0.38; these improvements suggest transfer beyond the Qwen2.5 family.Source: §F.3 · p. 24

• The theorem assumes that a gradient step on the sampled token does not substantially affect other vocabulary-token logits.It then states a decomposition of token-level entropy change under a single GRPO update.Source: G Theorem Proof Details · p. 25
• A first-order Taylor expansion around the current logits supplies the entropy-change estimate.The proof defines the logit vector over all actions under a state and derives the entropy gradient with respect to logits.Source: G Theorem Proof Details · pp. 25–26
• The proof writes the GRPO logit update using the clipping indicator and token-level advantage, while treating non-sampled action logits as unchanged.Source: G Theorem Proof Details · pp. 26–27
• The higher-order remainder is O(η^2) and is further suppressed by GRPO’s per-update normalization factor L^-2.The stated bound assumes uniformly bounded importance ratios and token-level advantages.Source: G Theorem Proof Details · pp. 27–28