PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy-Preserving Reinforcement Learning

Aleksei Petrenko, Ben Lipkin, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, Philipp Krähenbühl

arXiv:2603.11682v1cs.LGcs.AI

TL;DR

Policy-gradient training can collapse entropy, reducing trajectory diversity and exploration. The paper analyzes the mechanisms and implementation factors behind this behavior, then introduces explicit entropy-control methods; entropy-preserving methods perform strongly while retaining compatibility with sequential learning and asynchronous training benefits.

  • Problem

    Policy-gradient algorithms can narrow trajectory distributions around high-probability solutions, limiting exploration and diverse reasoning paths.

  • Method

    The paper analyzes entropy dynamics theoretically and empirically, then proposes REPO, which modifies advantages, and ADAPO, which adaptively adjusts asymmetric clipping.

  • Results

    Entropy-preserving REPO and ADAPO achieve strong off-policy performance, closing the gap to on-policy training and retaining trainability for sequential learning.

  • Takeaways & Limitations

    Entropy and exploration capability should be treated as first-class concerns in reinforcement-learning training pipelines.

  • Takeaways & Limitations

    Finite-precision implementations and fixed entropy-bonus coefficients can distort entropy control or impose memory costs.

Abstract

from arXiv · show

Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy -- and thus the diversity of explored trajectories -- as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we argue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.

1 INTRODUCTION

Policy-gradient RL can lose exploration by collapsing entropy during training. The paper argues for monitoring and controlling entropy because its trajectory, rather than only final entropy, is associated with downstream performance.

  • Motivation: Entropy collapse narrows exploration toward already high-probability solutions, often improving pass@1 while reducing pass@k.This can cause premature convergence to a local optimum.
  • Contribution: The paper proposes analyzing and actively controlling entropy throughout reinforcement-learning training.Its framing unifies entropy dynamics across policy-gradient objectives and motivates explicit control mechanisms.
  • Motivation: Lower entropy throughout training yields lower performance, whereas differences confined to the final steps have little effect.The paper therefore treats the entropy trajectory as more informative than final entropy alone.
  • Contribution: REPO modifies the advantage function, while numerical fixes alone reach 79% Test Normal and 71% Test Challenge on AppWorld.REPO and ADAPO are presented as entropy-preserving methods with strong off-policy performance.

2 PRELIMINARIES

The paper formulates language modeling as policy-gradient reinforcement learning over token-generation trajectories. It introduces entropy as policy uncertainty and reviews on-policy, clipped, and sequence-level optimization methods.

  • Language modeling: A language model defines an autoregressive probability distribution over token strings.Token probabilities factor over preceding context, with an end-of-sequence token terminating generation.
  • RL formulation: In the MDP formulation, tokens are actions, generated text is state context, and trajectories receive terminal rewards.State transitions append generated actions to the context and terminate at EOS or another environment condition.
  • Policy gradients: REINFORCE estimates the policy gradient using advantages multiplied by trajectory log-probability gradients.The advantage shifts return by a baseline, and RLOO estimates it from independent samples.
  • Optimization methods: PPO permits limited deviation from a sampling policy by clipping importance weights, while GSPO clips tokens and trajectories differently.GRPO and LOOP combine the PPO objective with leave-one-out advantage estimates.
  • Entropy: Policy entropy measures uncertainty over generations or actions at a given state.The paper studies how state-wise entropy evolves under policy-gradient objectives.

3 THEORY: ENTROPY DYNAMICS OF POLICY GRADIENT

Entropy dynamics are governed by how advantages correlate with action log-probabilities. Clipping can limit or bias entropy changes: PPO bounds them, while asymmetric clipping in DAPO and GSPO implicitly favors preservation, though not universally.

  • General entropy dynamics: Positive advantages sharpen high-probability actions and increase low-probability actions, respectively decreasing or increasing entropy.Negative advantages produce the opposite pattern.
  • General entropy dynamics: Entropy change is driven by a multiplicative relationship between action log-probabilities and their advantages.Under simplifying assumptions, the change is weighted by action probabilities and reflects their correlation.
  • PPO: PPO clipping bounds updated entropy between (1 − ϵlow) · Hπθold and (1 + ϵhigh) · Hπθold.The clipping thresholds directly limit the maximum induced entropy change per token.
  • DAPO and GSPO: DAPO’s asymmetric clipping permits larger entropy increases than decreases, producing an overall entropy increase over sufficient samples.The proposed thresholds ϵlow = 0.2 and ϵhigh = 0.28 stabilize entropy experimentally.
  • DAPO and GSPO: GSPO’s sequence-level asymmetric clipping provides a trajectory-length-dependent entropy bound and can similarly favor entropy increases.For trajectories shorter than approximately 600 tokens, its bound is tighter than DAPO’s under the stated parameters.
  • Implications: RLOO avoids entropy-collapse amplification from off-policy drift and repeated recycled-advantage updates but does not eliminate the underlying entropy decrease.Even strictly on-policy training can reduce entropy when advantages correlate positively with log-probabilities.

4 EMPIRICAL FINDINGS: IMPLEMENTATION DETAILS AFFECTING ENTROPY

Implementation details can qualitatively alter entropy dynamics. In particular, finite-precision output casting biases clipping toward entropy decrease, while full-precision fixes and FP16 training can reverse apparent entropy collapse.

  • 16-bit quantization: BF16 casting introduces an upward bias in observed importance ratios, with E[robserved | rtrue] > rtrue.The bias arises when model outputs are cast to the training dtype before ratio computation.
  • 16-bit quantization: Quantization reaches ϵhigh more often and ϵlow less often, limiting probability increases for low-probability actions and reducing entropy preservation.This creates an effective clipping asymmetry that favors entropy decrease.
  • 16-bit quantization: Using full precision for model outputs throughout ratio calculations is a direct fix for the finite-precision clipping bias.The issue exists in any finite-precision implementation but is stronger with BF16’s limited precision.
  • FP16 versus BF16: FP16 reduces discrepancies between inference and training and tends to produce more stable, predictable entropy behavior.With the log-probability rounding fix, DAPO changes from entropy collapse to rapid entropy increase on Qwen-3-8B AppWorld.

5 EXPLICIT ENTROPY CONTROL METHODS

The paper proposes explicit entropy-control methods because implicit clipping can be unstable and standard entropy bonuses are costly. REPO regulates entropy through advantage modification, while ADAPO adapts asymmetric clipping using observed entropy.

  • Motivation: Explicit entropy bonuses require a fixed coefficient and full-logit materialization, motivating an adaptive controller and paired-sampling estimator.The proposed estimator jointly estimates policy and entropy gradients without materializing the full vocabulary logits.
  • REPO: REPO modifies the advantage function with a scaled policy log-likelihood term to control entropy at the token level.The induced control increases entropy when the coefficient is positive and decreases it when negative.
  • REPO: REPO-D counteracts per-token entropy change, aiming to neutralize the default entropy dynamics.Its coefficient is set in proportion to the negative estimated entropy change.
  • REPO: REPO-R increases entropy by upweighting rare correct solutions while reducing the average penalty on rare incorrect solutions.It implements this intuition by rescaling advantages according to action probabilities.
  • ADAPO: ADAPO adaptively changes the upper asymmetric clipping threshold while keeping the lower threshold fixed, providing bidirectional entropy control.It increases the upper threshold when entropy falls below its initial level and decreases it when entropy rises above that level.

6 EXPERIMENTS

Experiments compare policy-gradient algorithms on AppWorld and AIME, finding that entropy dynamics vary substantially across algorithms and implementation details. Entropy-preserving methods generally outperform their off-policy baselines and better support subsequent training on another task.

  • Entropy dynamics: GRPO reduces entropy by nearly 90% over training, whereas strictly on-policy RLOO loses considerably less entropy.PPO-like methods deplete entropy faster than strictly on-policy training, while LOOP behaves similarly to GRPO.
  • Entropy dynamics: DAPO and GSPO retain considerably more entropy through clipping modifications, although DAPO can uncontrollably increase entropy without control.These observations appear across AppWorld and AIME experiments.
  • Performance: Entropy-preserving methods yield higher final test accuracy when they maintain higher cumulative per-token entropy, with stronger trends on AppWorld than AIME.The paper hypothesizes that Qwen-3 is more overfit to AIME, while AppWorld requires more exploration.
  • Sequential learning: During sequential training, GRPO policies perform poorly on the opposing task, whereas DAPO and especially REPO retain exploration ability.The experiment trains first on AIME or AppWorld and then continues training on the other environment.
  • Implementation sensitivity: Numerical fixes qualitatively changed DAPO from entropy collapse to rapid entropy increase in Qwen-3-8B AppWorld training.The result indicates that observed entropy dynamics can be highly sensitive to implementation details such as numerical precision.
  • Performance: 79% Test Normal and 71% Test Challenge were achieved by the best FP16 RLOO checkpoint with Qwen-3-32B on AppWorld.The paper reports this as the highest score at submission time.

7 RELATED WORK

Related work frames reinforcement learning as a way to sharpen pretrained policies around existing solutions, while addressing entropy collapse through modified objectives or reference-policy constraints. These approaches differ in how they preserve exploration and limit policy change.

  • RL for alignment: Reinforcement learning has become a dominant approach for aligning pretrained language models in environments with verifiable rewards.The cited applications include programming and mathematics.
  • Policy sharpening: Prior work commonly views training as sharpening the base policy around existing solutions rather than producing new ones.This perspective treats the pretrained policy as already calibrated to many reward functions.
  • Modified objectives: Several methods address entropy collapse by modifying policy-gradient objectives, including rank-based rewards, wider PPO clipping, sequence-level clipping, and direct pass@k optimization.These methods target low-probability correct actions or reduce dependence on individual action probabilities.
  • Reference-policy constraints: Other approaches preserve the base policy with a DKL penalty, but prior work reports that this can limit how much the policy learns.Subsequent methods remove, selectively apply, or iteratively reset the reference-policy constraint.

8 CONCLUSION

The paper analyzes entropy dynamics in policy-gradient RL and proposes entropy-preserving mechanisms that improve performance while retaining exploration and trainability. It also connects entropy preservation with implementation choices and with the trade-off between strictly and weakly on-policy training.

  • 8 CONCLUSION: Entropy-preserving methods perform strongly on AIME and AppWorld, outperforming GRPO and DAPO baselines while improving sequential learning.The paper also reports 79% Test Normal and 71% Test Challenge on AppWorld with RLOO and FP16 training.
  • 8 CONCLUSION: RLOO achieves the best performance overall with proper numerical handling, whereas weakly on-policy methods provide asynchronous throughput benefits.Strictly on-policy training requires synchronous updates, while weakly on-policy training can overlap trajectory collection and policy updates.
  • 8 CONCLUSION: REPO and ADAPO are compatible with both on-policy paradigms and can help weakly on-policy methods approach strictly on-policy performance while preserving asynchronous execution.REPO modifies the advantage function, while ADAPO adaptively adjusts clipping thresholds.
  • 8 CONCLUSION: Entropy should be treated as a first-class concern in RL training because it corresponds to the exploration capability of the policy.The paper argues that entropy should be actively monitored and controlled throughout training.
  • 8 CONCLUSION: The paper’s theoretical analysis and algorithm development are primarily focused on entropy collapse in policy-gradient algorithms for language-model reasoning.The paper also acknowledges potential misuse risks involving biased, harmful, or misleading language-model outputs.

A.8 PROOF OF THEOREM 3 (BF16 MULTIPLICATIVE BIAS)

The appendix analyzes how BF16 quantization and output casting bias importance ratios and alter entropy-related clipping behavior. These effects are concentrated in off-policy clipped algorithms and can materially affect entropy dynamics and performance despite changing fewer than 0.1% of output tokens.

  • A.8 PROOF OF THEOREM 3 (BF16 MULTIPLICATIVE BIAS): Under bf16 quantization, the observed importance-ratio estimate has a multiplicative upward bias: E[robserved | rtrue] > rtrue.The bias is proportional to rtrue, so larger ratios experience proportionally larger absolute bias.
  • A.8 PROOF OF THEOREM 3 (BF16 MULTIPLICATIVE BIAS): FSDP2 casts module outputs to the chosen floating-point type, including final model outputs, even when logits are computed in full 32-bit precision.Preserving full-precision log probabilities requires explicitly overriding the MixedPrecisionPolicy output dtype.
  • A.8 PROOF OF THEOREM 3 (BF16 MULTIPLICATIVE BIAS): Half-precision downcasting affects importance-weight numerical stability and can alter clipping in off-policy algorithms such as LOOP, GRPO, and DAPO.The passage reports no measurable difference for fully on-policy RLOO, while off-policy clipped methods are affected.
  • A.8 PROOF OF THEOREM 3 (BF16 MULTIPLICATIVE BIAS): Fewer than 0.1% of output tokens change their clipping outcome under 16-bit rounding, yet the resulting bias reduces DAPO’s entropy-preservation effectiveness.More tokens exceed the upper clipping range, while fewer exceed the lower range.
  • A.8 PROOF OF THEOREM 3 (BF16 MULTIPLICATIVE BIAS): FP16 training significantly reduces the discrepancy between inference and training policies compared with BF16 training.The discrepancy arises from implementation differences and lack of batch-size invariance in GPU kernels.

B.3 ABLATION STUDY

The ablation shows that numerical precision changes can reverse DAPO’s entropy dynamics, while a separate softmax-gradient issue distorts updates for highly confident tokens. Targeted numerical fixes improve accuracy without full-vocabulary 64-bit overhead.

  • Precision ablations: Using the MPP fix and FP16 training together changes DAPO from entropy collapse and poor exploration to rapidly increasing entropy during Qwen-3-8B training.The combined changes also improved training across models and algorithm variants.
  • Softmax-gradient precision: For high-probability tokens, 1 − p can round to zero, making the sampled-token gradient vanish while other-token gradients remain non-zero.This affects roughly 30–40% of tokens in the reported experiments, often involving formatting or programming syntax.
  • Softmax-gradient precision: The resulting gradients are incorrect for high-confidence tokens and can distort learning where the model is already very confident.The issue persists even with full-precision training because it arises from the softmax backward computation.
  • Softmax-gradient precision: Recalculating problematic high-probability-token gradients in 64-bit precision provides an O(1) per-time-step fix without full-vocabulary 64-bit memory overhead.The method identifies the highest-probability tokens at each time step and recomputes only their gradients.
  • Relative impact: The softmax-gradient issue is less noticeable than FP16 training and the MPP fix because affected gradients are tiny relative to gradients for tokens with p ≪1.The paper presents the combined numerical effects in Figure 8.

C.3 SUMMARY OF RESULTS

Across the reported experiments, entropy-preserving methods provide strong AppWorld performance and more stable entropy dynamics, while AIME differences are small and peak performance appears early. REPO modifies advantages to favor low-probability actions, with practical implementations and adaptive control described for training.

  • Results: Entropy-preserving methods show strong performance relative to DAPO and GRPO baselines, together with more stable entropy dynamics.The comparison is summarized for the AppWorld and AIME result tables.
  • Results: ADAPO and REPO-R perform competitively on AIME, but algorithm results are very close and peak performance is reached very early in training.The experiments limit maximum context length to 4096 tokens to make reasoning more compact and training dynamics more demanding.
  • Results: On-policy RLOO reaches 79% success on Test Normal and 71% on Test Challenge after the reported numerical improvements.These results exceed the highest scores reported at submission time for an agentic GPT-4.1-based system.
  • REPO mechanism: REPO transformations rotate advantage–log-probability pairs, reinforcing low-probability correct actions, especially when their outcomes are substantially better than the batch average.REPO-D uses surprisal-based advantage adjustment, while REPO-R additionally scales by advantage magnitude.
  • REPO-R implementation: REPO-R uses A(s, a) · (1 + ζ · L(s, a)) in its general formulation, with positive ζ reducing penalties on rare incorrect actions and strengthening penalties on common incorrect actions.In practice, the centered L(s, a) can be replaced with the raw log-probability, avoiding expensive recomputation with negligible practical effect.
  • REPO-R implementation: The implementation includes an adaptive controller that adjusts ζ according to whether current entropy is above or below a target.The reference implementation includes bidirectional per-token REPO-R rescaling and an iteration-level controller.

E QWEN 2.5 EXPERIMENTS

Qwen 2.5 32B behaves differently from Qwen 3 in entropy dynamics and baseline performance: LOOP and GRPO avoid rapid collapse, while RLOO does not. Numerical changes also improve LOOP’s AppWorld results over prior work.

  • Initial performance: Qwen 2.5 32B starts near 40% on AppWorld Test Normal, compared with under 10% for Qwen 3 models.The paper attributes this difference to Qwen3’s excessively verbose thinking blocks hindering task progress.
  • Cross-model comparison: Qwen 2.5 achieves substantially lower best performance on Test Challenge than Qwen 3, which the paper most likely attributes to base-model limitations.The comparison concerns the hardest reported AppWorld test split.
  • Comparison with prior work: LOOP’s best Qwen 2.5 32B checkpoints exceed prior reported success rates by approximately 7% on Test Normal and 9% on Test Challenge.The paper most likely attributes the improvement to numerical changes because the remaining setup and hyperparameters closely match prior work.
  • Entropy dynamics: Unlike Qwen 3, Qwen 2.5 does not show rapid LOOP/GRPO entropy collapse, whereas RLOO does, indicating that base-model characteristics substantially affect training entropy dynamics.The conclusion is based on the reported model comparison and entropy trajectories.
Loading 2603.11682v1…