Source-linked AI summary
Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
Bolian Li, Yifan Wang, Yi Ding, Anamika Lochab, Ananth Grama, Ruqi Zhang
TL;DR
RL training in LLMs can saturate as entropy collapse reduces exploration, while existing entropy-preserving methods may be unstable over long horizons. Entrocraft uses rejection sampling to bias advantages and precisely control entropy schedules, with experiments showing broader generalization, diversity, and sustained improvement.
Problem
RL performance can saturate as entropy collapse shrinks exploration, while existing methods offer coarse or unstable long-term entropy control.
Method
Entrocraft uses rejection sampling to bias the advantage distribution and realize user-customized entropy schedules without modifying the RL objective.
Results
Entrocraft improves generalization, output diversity, and training duration; a 4B model surpasses an 8B baseline, AIME-25 pass@32 is 50% higher than baseline, and improvement lasts up to 4× longer before plateauing.
Takeaways & Limitations
Entrocraft provides customizable entropy curves and integrates as a drop-in method for policy-gradient RL, enabling continual improvement on more data without saturation.
Takeaways & Limitations
A slightly higher constant entropy target can become unstable and fluctuate over time because scarce negative samples limit entropy-increasing steps.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has enabled complex reasoning abilities in large language models (LLMs). However, most RL algorithms suffer from performance saturation, preventing continued gains as RL training scales. This problem can be characterized by the collapse of entropy, a key diagnostic for exploration in RL. Existing attempts focus on preventing entropy collapse through regularization or clipping. However, their resulting entropy curves often exhibit instability in the long term, which hinders performance gains. In this paper, we introduce Entrocraft, a simple rejection-sampling approach that realizes user-customized entropy schedule by biasing the advantage distributions. Entrocraft requires no objective regularization and is advantage-estimator-agnostic. Theoretically, we relate per-step entropy change to the advantage distribution under minimal assumptions. This explains the behavior of existing RL and entropy-preserving methods. Entrocraft also enables a systematic study of entropy schedules, which reveals that linear annealing, which starts high and decays to a slightly lower target, performs best. Empirically, Entrocraft addresses performance saturation, significantly improving generalization, output diversity, and long-term training. It enables a 4B model to outperform an 8B baseline, sustains improvement for up to 4x longer before plateauing, and raises pass@K by 50% over the baseline.
1 Introduction
RL performance can saturate as entropy collapse narrows exploration, while existing entropy-preserving methods may remain unstable or difficult to schedule precisely. Entrocraft controls entropy through rejection sampling, and experiments identify linear annealing as the best schedule with gains in generalization, diversity, and training duration.
- Entropy collapse narrows exploration and contributes to early performance saturation as RL training scales.
- Existing entropy-preserving methods can increase entropy but may drift too high, destabilize RL, and provide only coarse long-term schedule control.
- Linear annealing performs best among the compared entropy schedules.
- Theoretical analysis relates entropy changes negatively to advantages and shows that high model confidence amplifies those changes.
- Entrocraft uses rejection sampling to filter rollout samples and bias the advantage distribution toward a user-specified entropy schedule without modifying the RL objective.
- A 4B model surpasses an 8B baseline, AIME-25 pass@32 is 50% higher than baseline, and improvement lasts up to 4× longer before plateauing.
2 Preliminaries
LLM policy-gradient RL generates rollouts, estimates advantages, and updates the policy, while predictive entropy measures uncertainty, diversity, and exploration during training.
- The actor is a θ-parameterized policy that produces next-token probabilities over the vocabulary.
- Policy-gradient RL alternates rollout generation, advantage estimation, and policy updates to learn from environment feedback.
- Many recent RL algorithms use a PPO-style objective with an importance-sampling ratio multiplied by an estimated advantage.
- The simplified theoretical update is Δθ = η · E[Â · ∇θ log πθ], where η is the learning rate.
- Predictive entropy measures model uncertainty and serves as an indicator of response diversity and exploration capability.
3 Theoretical Analysis: How Entropy Evolves during LLM RL
Theoretical analysis links entropy change in LLM RL to advantage and model confidence, explaining why advantage-weighted updates often cause entropy collapse. The results also clarify how existing RL and entropy-preserving methods affect entropy dynamics.
- Assumptions and Guarantees: Under a small-learning-rate assumption, the analysis derives token- and sequence-level guarantees for entropy changes in realistic LLM policies.The theory avoids requiring a tabular softmax policy and uses minimal assumptions about the actor policy and advantage distribution.
- Entropy–Advantage Relationship: Entropy change is negatively related to advantage when rollout probability exceeds an output-space baseline.The paper summarizes this relationship as entropy change being proportional to negative advantage times the rollout log likelihood above the baseline.
- Entropy–Advantage Relationship: Positive-advantage samples tend to reduce entropy, whereas negative-advantage samples tend to increase it when model confidence is sufficiently high.Empirical evidence shows the confidence condition holds across positive, zero-mean, and negative advantage estimators.
- Existing RL Dynamics: Entropy collapse becomes the default when training is dominated by positive advantages, while selective advantage-region updates can maintain entropy during reward improvement.The theory also explains the accuracy–entropy tradeoff in standard RL without asserting that entropy directly determines performance.
- Existing RL Dynamics: Clipping alleviates entropy drop by removing high-advantage or high-confidence tokens and reducing expected absolute entropy change.Positive-negative decoupling methods are likewise interpreted as explicitly controlling the sign of advantages.
4 Methodology
Entrocraft controls RL entropy by filtering rollout samples according to current entropy and advantage, directly shaping the advantage distribution. This enables target entropy curves, including annealing schedules designed for more stable long-term training.
- Rejection Sampling Controller: Entrocraft uses rejection sampling during rollout generation to select which samples contribute to the policy-gradient update.The approach controls entropy without introducing a new RL objective or auxiliary entropy loss.
- Precise Curve Control: Entrocraft can move entropy toward target values within a few steps by directly modifying the advantage distribution.This makes the method a responsive, objective-agnostic control mechanism that can be added to existing RL algorithms.
- Precise Curve Control: Stable training requires entropy to remain high enough for exploration but low enough to avoid numerical overflow and performance saturation.The framework treats entropy dynamics as a controllable training hyperparameter applicable to policy-gradient methods.
- Rejection Sampling Controller: The filter dynamically biases the advantage distribution according to whether current entropy falls below or exceeds a target range.When entropy is too low, it rejects most high-advantage rollouts while retaining lower- and negative-advantage samples.
- Annealing Schedules: A slightly higher constant entropy target can eventually become unstable because long-term rollout imbalance leaves too few negative samples for entropy-increasing updates.This motivates annealing the entropy target rather than holding it constant.
- Annealing Schedules: A linear-decaying schedule starts near 0.6 and gradually approaches 0.2, reducing unstable entropy-increasing steps later in training.Among the compared annealing schemes, the simple linear-decaying curve achieves the best performance.
5 Experiments
Experiments evaluate Entrocraft on math reasoning tasks, comparing model sizes, inference-time scaling, entropy schedules, and long-term training against RL and entropy-preserving baselines. Entrocraft consistently improves performance, with decaying schedules remaining stable and linear decay slightly outperforming cosine decay.
- Experimental Setup: Experiments use Numina-Math for training and evaluate Qwen3 models and other model families on AMC-23 and AIME benchmarks.The setup includes 440K training questions, held-out data for general RL, and full-data experiments for long-term RL.
- Benchmark Evaluation: Entrocraft consistently improves final math-reasoning performance over RL and other entropy-preserving baselines.The comparisons include GRPO, GSPO, loss regularization, clipping, and positive-negative decoupling methods.
- Benchmark Evaluation: A 4B model with Entrocraft can outperform an 8B model trained with standard GRPO, while pass@K grows faster during inference-time scaling.The benchmark results also show that Entrocraft performs well under both mean@32 and pass@32 settings and avoids collapsing to a few solutions.
- Crafting Entropy Curves: After 200K training samples, fixed-target entropy control becomes unstable, whereas linear and cosine decay remain stable and sustain improvement beyond 400K samples.The fixed-target scheme is associated with sharp entropy fluctuations and rollout imbalance; linear decay is slightly better than cosine decay.
- Long-Term RL: Entrocraft achieves the strongest long-term performance by preventing entropy drift and saturation, while GRPO saturates after 100K samples.Clip-Cov later suffers a performance drop after 300K samples due to entropy explosion, whereas Entrocraft remains stable across the compared training stages.
6 Conclusion
The paper introduces Entrocraft to address entropy collapse and performance saturation in RL through precise entropy control. Its rejection-sampling method biases the advantage distribution, enabling customizable entropy curves and continual improvement across benchmarks.
- Conclusion: Entrocraft addresses entropy collapse and consequent performance saturation with precise entropy control.The method is presented as simple and applicable across existing policy-gradient methods.
- Conclusion: Entrocraft controls entropy by biasing the advantage distribution through rejection sampling.The paper first analyzes what drives entropy change, then uses that analysis to construct the control method.
- Conclusion: Experiments show customizable entropy curves and consistent performance gains over existing entropy-preserving methods across benchmarks.Entrocraft is described as a drop-in addition to policy-gradient methods that supports continual improvement on more data without saturation.
Ethics and Broader Impact Statement
The paper identifies potential misuse of stronger reasoning models as a broader downstream risk while emphasizing that its contribution is a training-stability technique rather than a new capability domain. It reports no human-subject data, personally identifiable information, or safety-critical deployment.
- Potential risks include misuse of stronger reasoning models in broader downstream applications.
- The paper primarily contributes a training-stability technique rather than a new capability domain.
- The study involves no human-subject data collection, personally identifiable information, or safety-critical deployment.
- The authors document datasets, implementation details, hyperparameters, and compute requirements to support external scrutiny.
A.2 Limitation and Future Directions
The analysis relates entropy changes to advantage signs under token- and sequence-level assumptions, while noting that the current method is optimized for single-turn dense-model math reasoning. More challenging multi-turn and MoE settings remain future directions because entropy instability is more catastrophic there.
- Limitation and Future Directions: The current method is optimized for single-turn math reasoning with dense models, while multi-turn RL and MoE settings remain unvalidated.The paper identifies more catastrophic entropy instability in these settings and plans future extension.
- Token-Level Entropy Change: For negative-advantage tokens, entropy increases, yielding ˆA_k · ∆H ≤0 with high probability.The result is stated symmetrically for tokens with negative advantage.
- Token-Level Entropy Change: Token-level entropy decreases when the sampled token's log probability exceeds the corresponding reference term.This condition follows from the Taylor-expanded entropy change.
- Token-Level Entropy Change: For a positive-advantage token, the RL update increases its probability while probability conservation decreases the probabilities of other tokens.The entropy condition is derived using the probability-sum constraint.
B.2 Proof of Theorem 2
Under a shared-outcome-reward assumption, the sequence-level derivation connects advantage signs to entropy changes when rollout likelihood is sufficiently high. The experiments report stable target control, reduced effective batch sizes, and lower computational needs than naive GRPO.
- Proof of Theorem 2: The sequence-level analysis assumes all tokens share the same outcome reward and relates the entropy-change sign to the response advantage.The derivation defines token probabilities at each response position.
- Proof of Theorem 2: For positive response advantage, all generated-token probability changes are positive, enabling an effective weighted-average token probability change.This simplifies the sequence-level entropy expression.
- Proof of Theorem 2: Sequence-level entropy change is negatively related to advantage when the generated response has sufficiently high likelihood.The same conclusion is stated for negative advantage, giving ˆA(x, y) · ∆H ≤0.
- Empirical Analysis: Entrocraft stabilizes entropy at the target 0.8, whereas GRPO collapses entropy and other preserving methods can remain unresponsive or unstable long term.The comparison is reported in the full entropy curves.
- Empirical Analysis: Dropping effective batch sizes show that Entrocraft requires less gradient computation than naive GRPO.The measurements use GRPO with Entrocraft, initial rollout.n=8, and a linear-decaying entropy range.
C.4 Full Evaluation on AIME
Full AIME results are consistent with the main benchmark conclusions: Entrocraft outperforms other entropy-preserving methods across diverse RL algorithms.
- Full Evaluation on AIME: Full AIME results are consistent with the conclusions reported in Table 2.The paper reports this consistency across all AIME benchmarks.
- Full Evaluation on AIME: Entrocraft outperforms all other entropy-preserving methods across diverse RL algorithms.This conclusion is stated for the full AIME evaluation.
- Full Evaluation on AIME: The AIME evaluation spans diverse RL algorithms rather than a single algorithmic setting.The reported comparison covers all AIME benchmarks discussed in the passage.
C.5 Results on Other Models
Entrocraft consistently improves RL performance across larger and different-family models, while its entropy control has practical boundaries: overly high entropy can destabilize long-term training.
- Results on Other Models: Entrocraft consistently improves performance over GRPO across larger models and different model families.This includes Qwen3-8B-Base, Qwen3-14B-Base, and Llama-3.1-8B-Instruct.
- Results on Other Models: The effective batch sizes of RL steps affected by Entrocraft decrease as model performance improves.
- Results on Other Models: Entrocraft consistently improves the final performance of RL algorithms compared with other entropy-preserving methods on AIME.
- Failure Case: Overly high entropy introduces instability that can accumulate and become more catastrophic during long-term RL training.