The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

Ganqu Cui; Yuchen Zhang; Jiacheng Chen; Lifan Yuan; Zhi Wang; Yuxin Zuo; Haozhan Li; Yuchen Fan; Huayu Chen; Weize Chen; Zhiyuan Liu; Hao Peng; Lei Bai; Wanli Ouyang; Yu Cheng; Bowen Zhou; Ning Ding

Source-linked AI summary

The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, Ning Ding

arXiv:2505.22617v1May 28, 2025cs.LGcs.AIcs.CL

TL;DR

The paper addresses policy-entropy collapse as a barrier to scaling reinforcement learning for reasoning LLMs. It analyzes entropy dynamics and introduces Clip-Cov and KL-Cov to control updates on high-covariance tokens. Experiments report that these methods encourage exploration, help escape entropy collapse, and achieve better downstream performance.

Problem
Policy entropy in RL for reasoning LLMs can collapse early, diminishing exploration and accompanying performance saturation, which challenges continued scaling of RL compute.
Method
The paper theoretically and empirically analyzes entropy dynamics and introduces Clip-Cov and KL-Cov to restrict updates or apply KL penalties to high-covariance tokens.
Results
The proposed methods encourage exploration, help the policy escape entropy collapse, and achieve better downstream performance.
Takeaways & Limitations
Scaling RL for reasoning LLMs requires managing entropy to sustain exploration rather than allowing policy entropy to become exhausted.
Takeaways & Limitations
The optimal entropy value for balancing exploration and training stability remains open, and entropy patterns differ across policy models and off-policy data settings.

Abstract

from arXiv · show

This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy. Such phenomenon is consistently observed across vast RL runs without entropy intervention, where the policy entropy dropped sharply at the early training stage, this diminished exploratory ability is always accompanied with the saturation of policy performance. In practice, we establish a transformation equation R=-a*e^H+b between entropy H and downstream performance R. This empirical law strongly indicates that, the policy performance is traded from policy entropy, thus bottlenecked by its exhaustion, and the ceiling is fully predictable H=0, R=-a+b. Our finding necessitates entropy management for continuous exploration toward scaling compute for RL. To this end, we investigate entropy dynamics both theoretically and empirically. Our derivation highlights that, the change in policy entropy is driven by the covariance between action probability and the change in logits, which is proportional to its advantage when using Policy Gradient-like algorithms. Empirical study shows that, the values of covariance term and entropy differences matched exactly, supporting the theoretical conclusion. Moreover, the covariance term stays mostly positive throughout training, further explaining why policy entropy would decrease monotonically. Through understanding the mechanism behind entropy dynamics, we motivate to control entropy by restricting the update of high-covariance tokens. Specifically, we propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively. Experiments show that these methods encourage exploration, thus helping policy escape entropy collapse and achieve better downstream performance.

1 Introduction

RL for reasoning LLMs faces policy-entropy collapse: entropy falls sharply early, exploration diminishes, and performance reaches a predictable ceiling. The paper analyzes this entropy mechanism and proposes token-level controls to preserve exploration and improve performance.

Challenge: Policy entropy sharply declines toward zero within a few training steps, while validation performance reaches a plateau.This pattern is reported across broad experiments on reasoning LLMs.
Entropy–performance relationship: The policy-performance relationship follows the exponential form R = −a exp H + b, making performance predictable from policy entropy.The relationship is observed without entropy intervention such as entropy loss or KL regularization.
Entropy–performance relationship: When entropy is exhausted at H = 0, the corresponding performance ceiling is R = −a + b, limiting the benefit of additional RL compute.The paper identifies this ceiling as deterministic under the empirical law.
Entropy dynamics: Entropy changes are driven by covariance between action probability and logit change, which is proportional to advantage under Policy Gradient-like algorithms.The theoretical analysis is paired with empirical validation of the covariance mechanism.
Entropy dynamics: Positive covariance persists through training, explaining why policy entropy continues to decrease even as covariance gradually declines.High-probability, high-advantage actions reduce entropy, whereas rare high-advantage actions increase it.
Entropy control: Clip-Cov detaches gradients for selected positive-covariance tokens, while KL-Cov applies KL penalties to tokens with the largest covariances.Both methods restrict updates to high-covariance tokens to control entropy and help the policy escape low-entropy behavior.

2 The Predictable “Collapse” of Policy Entropy

Across RL runs, policy entropy collapses rapidly toward zero while validation performance rises and then saturates. The entropy–performance relationship is empirically predictable, including its coefficients, early-stage forecasts, and performance ceiling.

The predictable collapse: Without entropy or KL intervention, entropy is predictably traded for reward, with the ceiling at H = 0 and R = −a + b.The coefficients a and b characterize the empirical relationship between policy entropy H and validation reward R.
The predictable collapse: Entropy drops sharply at training onset and declines monotonically toward zero, while validation performance rises rapidly before saturating.This pattern is observed consistently across the experiments.
The predictable collapse: 73% of entropy consumption and 76% of performance gain occur in the first 200 of 2400 gradient steps, while the first 800 steps account for over 93% and 94%, respectively.The remaining training steps therefore produce marginal returns relative to the early phase.
Entropy–performance predictability: A two-coefficient exponential function precisely fits entropy–performance relationships across model sizes, model families, and math and coding tasks.The fitted curves describe over 200 data points using only two coefficients.
Discussion: The coefficients are algorithm-irrelevant across GRPO, RLOO, and PRIME, vary smoothly with Qwen2.5 model size at a log-linear rate, and are not universal across all settings.Different policy models and off-policy data have produced distinct entropy patterns.

3 Dynamics Analysis of Policy Entropy

The paper explains policy-entropy changes in softmax policies through covariance between action log-probabilities and logit updates, linking this mechanism to policy-gradient advantages and empirical GRPO dynamics.

Gradient-based updates: Under policy-gradient and natural-policy-gradient updates, logit changes are proportional to action advantages, connecting entropy dynamics to probability–advantage covariance.Positive correlation tends to decrease entropy, while negative correlation tends to increase it.
Theoretical analysis: The analysis derives entropy differences for softmax policies and extends the result to policy-gradient and natural-policy-gradient algorithms before experimental validation.The study uses first-order entropy derivatives and algorithm-specific logit-update derivations.
Softmax entropy dynamics: For softmax policies, step-wise entropy change approximately equals the negative covariance between action log-probability and corresponding logit change.High-probability actions whose logits increase reduce entropy.
Empirical verification: In on-policy GRPO, empirical covariance and negative entropy change exhibit highly similar dynamics, supporting the theoretical proportionality between them.Entropy falls rapidly alongside large positive covariance early in training, then both stabilize at lower levels.
Empirical verification: Covariance is smaller for harder prompts and higher for easier, more accurate prompts, indicating distinct entropy-driving dynamics across example difficulty groups.The paper attributes lower covariance on harder examples to weaker alignment between high-probability actions and higher expected returns.

4 Entropy Control by Covariance Regularization

The paper controls entropy collapse by targeting tokens with unusually high covariance rather than relying on global entropy or reference-KL regularization. Clip-Cov and KL-Cov improve entropy control and downstream performance across evaluated settings.

Covariance-aware control: Clip-Cov and KL-Cov restrict updates to high-covariance tokens, respectively detaching selected gradients or applying a KL penalty.The methods target a small subset of tokens identified through covariance rather than regularizing all tokens uniformly.
Limits of conventional regularization: Naive entropy regularization is coefficient-sensitive, while reference-KL regularization stabilizes entropy but degrades policy performance.Entropy loss can cause entropy explosion at large coefficients, and its stabilizing setting does not outperform other baselines.
Covariance-aware control: High-covariance tokens can exceed average covariance by more than 500×, motivating targeted constraints on their contribution to policy updates.Clip-Cov samples a small fraction within predefined high-covariance bounds; KL-Cov selects a top-k proportion with k ≪1.
Experimental results: Compared to GRPO, the proposed methods improve average performance by 2.0% for the 7B model and 6.4% for the 32B model.The reported improvements span all benchmarks evaluated in Table 2.
Experimental results: KL-Cov can sustain entropy over 10× higher than the baseline after the baseline reaches an entropy plateau, while response length and test performance continue increasing.The paper associates this sustained entropy with freer exploration during training.
Experimental results: On Qwen2.5-32B, improvements over GRPO reach 15.0% on AIME24 and 14.6% on AIME25, exceeding the reported average gain.The authors infer that lifting entropy collapse allows the larger model to explore more diverse and higher-quality policies.
Entropy control: The methods expose entropy to direct control: more clipping raises entropy for Clip-Cov, while a larger KL coefficient raises entropy for KL-Cov.KL-Cov produces stabler entropy curves than Clip-Cov in the comparison described.
Discussion and limitation: Intervening on only 10^-4 to 10^-3 of tokens can substantially change the entropy curve, but the optimal entropy value for balancing exploration and stability remains unknown.The paper also reports no observed relationship between intervened entropy and model performance.

5 Related Work

Related work frames entropy as a central mechanism for balancing exploitation and exploration, situates RL among major approaches to LLM post-training, and connects reasoning progress to verifiable-reward RL.

Policy entropy in reinforcement learning: Entropy-regularized reinforcement learning uses regularization to encourage high-entropy actions and manage the exploitation–exploration trade-off.This approach has been incorporated into multiple reinforcement-learning algorithms.
Predictability of reinforcement learning: Neural scaling laws relate model performance to computational budgets, model size, and training data, motivating predictability studies for RL reasoning.The cited related work describes using smaller-model scaling experiments to estimate larger-model outcomes.
Reinforcement learning for LLMs: Reinforcement learning is a major approach for LLM post-training, with verifiable rewards supporting recent advances in reasoning capability.The related work highlights math and coding-style settings where verifier-based rewards are used.

6 Conclusion

The paper addresses entropy collapse in RL for LLM reasoning by analyzing its dynamics and introducing covariance-aware controls. It reports that these methods counteract collapse and improve downstream performance, while motivating further study of entropy's role in RL scaling.

Conclusion: The study targets policy-entropy collapse, which limits exploratory capacity and creates a foreseeable constraint on RL improvement for reasoning LLMs.The paper frames entropy management as necessary for scaling RL compute beyond entropy minimization.
Conclusion: The paper combines theoretical entropy-dynamics analysis with Clip-Cov and KL-Cov, which directly manage high-covariance tokens.The methods are designed to counteract entropy collapse rather than apply only global regularization.
Conclusion: The reported methods counteract entropy collapse and achieve better downstream performance, supporting entropy as a key consideration for scaling RL reasoning.The conclusion presents these findings as insights into the role of entropy in future RL scaling.

A Training Details for Different Models

Models from different families were trained with data at different difficulty levels to stabilize reinforcement learning, using distinct math datasets for selected families and GSM8K for others.

Different model families were trained on data with different difficulty levels to stabilize the RL process.
Qwen-family and Mistral-24B models used Eurus-2-RL-Math for math tasks.
Other model families were trained on GSM8K for math tasks.

B More Fitting Results

The paper presents additional fitting experiment results in this section.

This section presents more fitting experiment results.
The experiments are described as fitting experiments.
The section contains experiment results related to fitting.

C Fitting Results of Training with Different Dataset.

Figure 13 presents training results for Qwen2.5-7B using different data.

Figure 13 concerns training Qwen2.5-7B with different data.
The figure compares training conditions involving different data.
The presented model is Qwen2.5-7B.

D Fitting Results of Instruct Models

The paper reports that its fitting function remains valid on instruct models and presents fitting results for Qwen2.5 instruct models on a math task.

The fitting function remains valid in experiments on instruct models.
Figure 14 presents training Qwen2.5 instruct models on a math task.
The data were processed into a format handled by unit tests, while part of the online-judge data was removed because it derived from instruct-style data.

E Proof

This proof develops entropy and logit updates for a tabular softmax actor, using first-order Taylor expansion and gradient backtracking. The derivation relates logit changes to action advantages.

Logit update: The logit update derivation reduces to η · πθ(a | s) · A(s, a), linking each action’s logit change to its policy probability and advantage.The intermediate expression uses the softmax gradient and the expectation of the advantage function.
Entropy change: Lemma 1 characterizes the change in information entropy at a state between consecutive policy-update steps.Its derivation uses the entropy definition and the expected policy-gradient expression.
Proof setup: The proof assumes each state-action pair has its own logit parameter, z_s,a = θ_s,a, and updates logits with a small-step gradient rule.The first-order approximation relies on η being relatively small.

E.4 Proof for Theorem 2

This section proves an entropy-difference result for a tabular softmax actor updated by natural policy gradient. It first derives the corresponding logit difference and then substitutes that result into Lemma 1.

Theorem statement: Theorem 2 concerns the change in information entropy at a state between consecutive updates under natural policy gradient.The policy is specified as tabular softmax and updated via natural policy gradient.
Proof derivation: The proof derives the natural-policy-gradient logit difference using Lemma 1 and gradient backtracking.The derivation draws on Agarwal et al. (2021).