PaperlayerView exact revision on arXiv

Source-linked AI summary

EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control

Kai Yang, Xin Xu, Yangkun Chen, Weijie Liu, Jiafei Lyu, Zichuan Lin, Deheng Ye, Saiyong Yang

arXiv:2511.15248v2cs.LGcs.AI

TL;DR

Long-term LLM reinforcement learning needs stable entropy to sustain exploration, yet existing methods struggle to control it as positive and negative samples shift entropy in opposite directions. EntroPIC adjusts their weights using proportional or proportional-integral feedback on entropy deviations. The paper reports stable entropy control at scale and improved evaluation results over GRPO.

  • Problem

    Existing LLM reinforcement-learning methods struggle to maintain appropriate entropy, although it is important for exploration and avoiding premature convergence.

    Source: §1 · p. 2

  • Method

    EntroPIC uses proportional or proportional-integral feedback on target-entropy deviations to dynamically adjust positive- and negative-sample weights.

    Source: §1 · p. 2, §4.2 · pp. 4–5

  • Results

    EntroPIC improves average pass rate (avg@N) by 3.5% and pass rate (pass@N) by 3.8% over GRPO, while stabilizing entropy at the target during training.

    Source: §5.2.2 · p. 8

  • Takeaways & Limitations

    The paper reports stable entropy control in both on-policy and off-policy training and successful validation on over 1M prompts.

    Source: §1 · p. 2

    EntroPIC may add little when on-policy training already maintains stable entropy, and its manually set target entropy must be tailored to the model and task.

    Source: §6 · p. 9

Abstract

from arXiv · show

Long-term training of large language models (LLMs) requires maintaining stable exploration to prevent the model from collapsing into sub-optimal behaviors. Entropy is crucial in this context, as it controls exploration and helps avoid premature convergence to sub-optimal solutions. However, existing reinforcement learning methods struggle to maintain an appropriate level of entropy, as the training process involves a mix of positive and negative samples, each affecting entropy in different ways across steps. To address this, we propose Entropy stabilization via Proportional-Integral Control (EntroPIC), a novel method that adaptively adjusts the influence of positive and negative samples by dynamically tuning their loss coefficients. This approach stabilizes entropy throughout training, ensuring efficient exploration and steady progress. We provide a comprehensive theoretical analysis for both on-policy and off-policy learning settings, demonstrating that EntroPIC is effective at controlling entropy in large-scale LLM training. Experimental results show that our method successfully maintains desired entropy levels, enabling stable and optimal RL training for LLMs.

1 Introduction

Long-term LLM reinforcement learning needs stable entropy to sustain exploration, but existing methods struggle to maintain a target level without performance costs. EntroPIC uses feedback from entropy deviations to adjust sample weights, and the paper reports stable control in large-scale training.

1 Introduction

  • Entropy supports exploration and helps prevent premature convergence, while too much entropy can produce noisy outputs and instability.The introduction identifies output diversity, quantified by entropy, as important to training effectiveness and stability.Source: §1 · p. 2
  • Existing entropy-regulation methods slow entropy decline but struggle to maintain a target without sacrificing performance, and can lose token gradients or add hyperparameter sensitivity and complexity.The introduction also notes that on-policy entropy control is less explored despite its performance potential and early training fluctuations.Source: §1 · p. 2
  • EntroPIC uses proportional or proportional-integral control to adjust positive- and negative-sample weights from current and historical deviations from target entropy.The overview describes a correction factor based on current and historical entropy relative to the target, applied to high-probability samples.Source: §1 · p. 2, Figure 2 · p. 2
  • EntroPIC stabilizes entropy for both on-policy and off-policy training and was validated on over 1M prompts.Its dynamic coefficient tuning targets entropy control without global adjustments.Source: §1 · p. 2

2 Related Work

Prior LLM reinforcement-learning work spans feedback-based alignment and verifiable-reward training, while entropy regulation has used several strategies. These methods can slow entropy decline but still struggle to keep it high and stable over long training.

Reinforcement Learning with Large Language Models

  • RLHF commonly trains a reward model from human or AI feedback before policy optimization, whereas RLVR computes rewards by comparing outputs with ground-truth solutions.The paper describes RLVR as removing the reward-model stage and mitigating reward hacking.Source: §2 · pp. 2–3
  • RLVR performance is described as depending largely on entropy control rather than reward-model quality (Cui et al., 2025; Yu et al., 2025).The paper states that maintaining appropriate entropy enhances model diversity and performance potential.Source: §2 · p. 3

Entropy in Policy Optimization

3 Preliminary

The preliminary section introduces verifiable-reward training, policy-gradient methods, and policy entropy, then relates entropy control to feedback on deviations from a target. It describes GRPO and PPO as examples of policy-optimization methods.

Reinforcement Learning with Verifiable Rewards

  • RLVR trains LLMs on objectively verifiable tasks by comparing generated outputs with ground-truth solutions to compute rewards, without a reward model.The paper names mathematics and programming as examples.Source: §3 · p. 3

Policy-gradient methods

  • Policy-gradient methods estimate gradients to optimize expected reward, using the advantage at each token step to reflect the value of selecting that token in context.The preliminary passage introduces the objective and gradient estimate, then defines the stepwise advantage.Source: §3 · p. 3

Policy-optimization methods

  • GRPO normalizes response advantages within sampled groups, while PPO uses a clipped surrogate objective to limit excessive policy updates.The clipping is described as balancing optimization progress with training stability.Source: §3 · p. 3

Policy Entropy in Large Language Models

  • Policy entropy measures uncertainty in token selection; low entropy can cause mode collapse, while high entropy can produce unstable training and noisy generation.The section defines average token-level entropy for an LLM policy on training data.Source: §3 · pp. 3–4
  • Entropy dynamics depend on covariance between token log-probabilities and advantage: positive samples reduce entropy, while negative samples increase it.The section links this behavior to the imbalance that can arise when training mixes positive and negative samples.Source: §3 · p. 4

Proportional-Integral Control

  • In the paper’s entropy-control adaptation of PI feedback, target entropy is the setpoint, measured entropy is the observed value, and their error adjusts sample-weight coefficients.The proportional term responds to current error, while the integral term accumulates historical error to address steady-state bias.Source: §3 · p. 4

4 Adjust Entropy via PI Control

EntroPIC uses the opposing entropy effects of positive and negative samples to regulate policy entropy toward a target. Its convergence conditions differ by training setting, and the guarantees also hold when control is limited to high-probability tokens.

4.1 The Impact of Positive and Negative Samples on Entropy

  • With binary rewards and zero expected advantage, training only on positive samples decreases policy entropy, while training only on negative samples increases it.These opposing effects motivate dynamically changing the positive- and negative-sample loss weights.Source: §4.1 · p. 4

4.2 Adjusting Weights to Control Entropy

  • EntroPIC uses PI control to adjust positive- and negative-sample loss coefficients according to the gap between current and target entropy.The balancing coefficient combines proportional and integral control; α = 0 recovers the original loss, while α = 1 or −1 focuses updates on positive or negative samples, respectively.Source: §4.2 · pp. 4–5
  • On-policy training converges to target entropy with P- or PI-control, whereas off-policy training requires PI-control because P-control leaves steady-state error.The high-probability-token variant retains these convergence properties under the same conditions.Source: §4.2 · p. 5, §4.3 · p. 5

4.3 Simplify Loss by Controlling High-Probability Samples

  • Controlling only tokens with probability above a fixed threshold preserves the on- and off-policy convergence guarantees under the same conditions.The paper identifies tokens above 95% probability as high-probability tokens for easier identification.Source: §4.3 · pp. 5–6
  • Masking experiments found that modifying high-probability samples produces larger entropy changes than modifying low-probability tokens, while reducing high-probability positive weights encourages exploration.The paper also cautions that reducing low-probability negative weights can suppress rare events and degrade performance; the high-probability formulation has identical gradients to its on-policy version.Source: Figure 3 · p. 5, §4.3 · p. 6, Figure 4 · p. 6

5 Experiment

The experiments evaluate EntroPIC in on-policy and off-policy training, testing entropy control and performance on non-thinking and reasoning models. EntroPIC stabilizes entropy and achieves stronger reported performance than the compared methods.

5.1 Settings

  • The experiments use Qwen3-8B-Base for mathematical tasks and OpenReasoning-Nemotron-1.5B for reasoning-model evaluation.The reasoning model is trained on Polaris; the non-thinking model is first fine-tuned with supervised learning.Source: §5.1 · p. 7

5.2.1 Entropy Control Experiments

  • On-policy and off-policy control experiments show EntroPIC stabilizing entropy near its target, while off-policy P-only control fails to regulate it stably.Without entropy control, entropy continually decays; the off-policy comparison finds successful stabilization only with full PI control.Source: §5.2.1 · p. 7, Figure 5 · p. 7

5.2.2 Large-scale Training Results

  • During large-scale on-policy training, EntroPIC keeps entropy at the target while accuracy grows steadily on training and validation sets.Other methods show slow accuracy improvement after 1,000 steps; EntroPIC achieves the best final results.Source: §5.2.2 · p. 8, Figure 6 · p. 8
  • EntroPIC improves average pass rate (avg@N) by 3.5% and pass rate (pass@N) by 3.8% over GRPO.The comprehensive comparison covers standard on-policy and off-policy training.Source: §5.2.2 · p. 8, Table 1 · p. 8
  • On-policy training reaches a higher performance ceiling than off-policy training in the reported comparison.The authors consequently emphasize entropy-stabilized on-policy training for higher final performance.Source: §5.2.2 · p. 8
  • In the reasoning-model evaluation, EntroPIC outperforms QuestA-Nemotron and JustRL-Nemotron across most mathematical benchmarks and achieves the best overall performance.The comparison uses the Nemotron-1.5B backbone; Table 2 reports the benchmark results.Source: §5.2.2 · p. 9, Table 2 · p. 9

6 Conclusion

EntroPIC adjusts weights on high-probability samples to stabilize entropy at a predefined target, with theoretical and empirical validation. The paper also notes settings where it may add little benefit and the need to set target entropy manually.

6 Conclusion

  • EntroPIC adjusts high-probability sample update weights to stabilize entropy at a predefined target throughout training.The conclusion reports theoretical and empirical validation of entropy-control convergence, including toy experiments.Source: §6 · p. 9
  • EntroPIC may offer little improvement when on-policy training alone already maintains entropy stability.The target entropy also must be set manually to fit the model and task.Source: §6 · p. 9

A.1 Proof of Corollary 4.1

  • Positive samples decrease entropy, whereas negative samples increase it under the proof's stated assumptions.The proof derives these opposite effects for positive and negative samples.Source: §A.1 · p. 13

A.2 Proof of Theorem 4.2

The proof analyzes proportional and proportional-integral feedback for entropy control. Under its stability conditions, the error converges to zero, so entropy reaches its target.

A.2 Proof of Theorem 4.2

  • The coefficient function assigns 1 + α to positive-advantage samples and 1 − α to negative-advantage samples.This varies the relative influence of the two sample groups through α.Source: §A.2 · p. 13
  • Under P control, the entropy error contracts toward zero when 0 < 1 + C1 < 1.The proof gives |e_k+1| = |(1 + C1)e_k| under its stated conditions.Source: §A.2 · p. 15
  • For PI control, the proof establishes convergence of entropy error to zero under its stability conditions.The conditions are satisfied with a small learning rate and moderate proportional and integral gains.Source: §A.2 · pp. 16–17

A.3 Proof of Theorem 4.3

The off-policy analysis attributes P control's steady-state error to distribution shift and shows that PI control removes it. Under stated stability conditions, the entropy error converges to zero.

A.3 Proof of Theorem 4.3

  • With sufficiently small learning rate and moderate positive gains, the PI error dynamics satisfy the stated stability conditions.The proof requires the system eigenvalues to lie within the unit circle.Source: §A.3 · p. 19
  • Lyapunov stability yields asymptotic convergence of the entropy error to zero.The proof concludes that the limiting entropy error is zero.Source: §A.3 · p. 19
  • P control has nonzero steady-state error under off-policy distribution shift, whereas PI control achieves zero steady-state error.The proof attributes the P-control bias to µ ≠ π and states that PI control reaches zero error when Ki ≠ 0.Source: §A.3 · pp. 18–19

A.4 Proof of Corollary 4.4

Corollary 4.4 extends entropy control to high-probability tokens: reweighting is restricted to tokens above τ, and the convergence results hold under the stated conditions. In off-policy training, integral control removes the steady-state bias associated with importance weighting.

A.4 Proof of Corollary 4.4

  • For actions with positive or negative advantage and probability above τ, the modified coefficient is respectively 1 + α or 1 − α.The proof applies the PI-weight only to high-probability tokens; under the binary-reward assumption, these are the positive- and negative-advantage action sets.Source: §A.4 · pp. 19–20
  • In the on-policy masked case, both P and PI control converge to the target entropy under the stated small-gain conditions.The entropy error is defined relative to the target, and the entropy-change terms are restricted to the high-probability subset.Source: §A.4 · pp. 19–20
  • In off-policy training, importance weighting introduces a masked bias under P control, while adding an integral term removes it and yields a stable zero-error state.The stability statement is given in terms of eigenvalues within the unit circle.Source: §A.4 · p. 20

B.1 Training Details

The experiments train Qwen3-8B-Base and OpenReasoning-Nemotron-1.5B under specified data, entropy-control, and compute configurations. Evaluation uses repeated benchmark runs, and the method requires minimal code changes.

B.1 Training Details

  • Non-thinking experiments use Qwen3-8B-Base, lightweight SFT, and filtered math-training data, with GRPO as the baseline.Training uses DAPO-MATH, OpenReasonerZero, and DeepScaleR; samples with uniformly correct or incorrect preliminary responses are excluded, and dynamic sampling is not used.Source: §B.1 · p. 20
  • The default PI setup uses Kp = 1, Ki = 0.01, and τ = 0.95; reasoning-model experiments use OpenReasoning-Nemotron-1.5B trained on Polaris with target entropy 0.4.The reported 2,000-step training run uses about 1 million prompts and 8 million samples; reasoning-model training takes about 30,000 GPU-hours.Source: §B.1 · p. 20

B.2 Evaluation Details

  • Evaluation uses rollout temperature 0.6, top-p 0.95, and output limits of 8k tokens for non-thinking models and 64k for reasoning models.The final evaluation repeats runs four times on Math, Olympiad, Minerva, and Omni-math, and 32 times on other datasets, reporting avg@N and pass@N.Source: §B.2 · pp. 20–21

C Code

  • The entropy-control implementation requires minimal code changes, as illustrated by the code example.Figure 8 shows the code change for using EntroPIC.Source: C Code · p. 21, Figure 8 · p. 21

D.1 Generalization Analysis

EntroPIC improves generalization results and maintains performance across several training configurations. The experiments also report a target-entropy trade-off, the effects of thresholded reweighting, reflective outputs, and adaptive control coefficients.

D.1 Generalization Analysis

  • +6.5% MMLU-Pro accuracy and +12.6% LiveCodeBench over the Nemotron-1.5B backbone; EntroPIC improves across all three non-math benchmarks.The comparison contrasts with other RL methods, which regress on general benchmarks and approach zero coding performance; Table 3 summarizes generalization performance.Source: Table 3 · p. 21, §D.1 · pp. 21–22

D.2 Robustness to Temperature Settings

  • With sampling temperature 1.0 and target entropy 0.3, EntroPIC outperforms the baseline on all evaluation datasets.The model’s initial entropy is controlled at approximately 0.3; Table 4 reports results under the high-temperature setting.Source: §D.2 · p. 22, Table 4 · p. 22, Figure 9 · p. 22

D.3 Plug-and-Play Capability

  • Introducing EntroPIC before significant late-stage entropy collapse stabilizes entropy at the target and immediately improves training and testing metrics.The plug-and-play experiment is illustrated in Figure 10.Source: §D.3 · p. 22, Figure 10 · p. 22

D.4 Impact of Target Entropy

  • Higher target entropy tends to slightly lower avg@N while leaving pass@N stable or improving it; lower targets slightly raise avg@N but reduce pass@N.The configurations were each trained for 2,000 steps, and Table 5 reports results for different target-entropy settings.Source: §D.4 · p. 22, Table 5 · p. 23

D.5 Effect of High-Probability Threshold τ

  • Reweighting only tokens with probability above 0.95 produces the most stable and controllable training dynamics among the tested thresholds.The comparisons cover all tokens, tokens above 0.5, and tokens above 0.95; Figure 11 shows the training curves.Source: §D.5 · p. 23, Figure 11 · p. 23

D.6 Reflection Ability

  • EntroPIC-trained models more often produce reflective phrases such as “wait” and “let’s reconsider,” while late-stage GRPO outputs show less self-correction.Figure 12 tracks reflective-word frequency during training.Source: §D.6 · p. 23, Figure 12 · p. 23

D.7 Control Coefficient α

  • Across on-policy, off-policy, high-temperature, and plug-and-play settings, α rises when entropy decreases and falls when entropy increases.The reported α variations are shown in Figure 13; the passage describes stable convergence and long-term entropy control across these settings.Source: §D.7 · p. 23, Figure 13 · p. 24

E Case Study

The case study frames the problem as finding positive k values that give a function exactly two real minima. Its analysis associates that condition with a double root in the critical-point equation and reports three candidate values.

E Case Study

  • The case study seeks positive k values for which f has exactly two distinct real minima.The stated task is to find the sum of the three values; the supplied passages do not give that sum.Source: E Case Study · pp. 24, 26
  • The analysis connects exactly two minima with a double root in the critical-point equation.The text says the critical point equation must possess a double root for certain k values.Source: E Case Study · pp. 25–26
  • The reported three k values are 46, 72, and 122.The values are presented as the middle value and its two symmetric counterparts.Source: E Case Study · p. 25
Loading 2511.15248v2…