PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning

Zhenpeng Su, Leiyu Pan, Minxuan Lv, Tiehua Mei, Zijia Lin, Yuntao Li, Wenping Hu, Ruiming Tang, Kun Gai, Guorui Zhou

arXiv:2512.05591v2cs.LGcs.CL

TL;DR

Off-policy reinforcement learning can push LLM policies beyond the trust region, while PPO-Clip leaves shifts among unsampled actions unconstrained. The paper uses entropy ratio to measure global exploration change and adds bidirectional ERC constraints; across multiple model scales and benchmarks, ERC consistently outperforms baseline methods.

  • Problem

    Off-policy updates can deviate from the trust region, and PPO-Clip constrains sampled actions while leaving unsampled-action probability shifts uncontrolled.

    Source: §1 · pp. 1–2

  • Method

    ERC uses the relative entropy change between new and old policies as a global exploration measure and imposes bidirectional constraints on it.

    Source: §1 · pp. 2–3, §3.1 · p. 4

  • Results

    ERC consistently outperforms baseline methods across multiple model scales and improves performance across nearly all mathematical benchmarks.

    Source: §4.2 · p. 5, §6 · p. 9

  • Takeaways & Limitations

    ERC provides a global distribution-level constraint that complements importance-ratio clipping of sampled actions.

    Source: §1 · p. 2, §6 · p. 9

    Generalization to code generation and agent-based reinforcement learning remains an open question because of computational constraints.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language model post-training relies on reinforcement learning to improve model capability and alignment quality. However, the off-policy training paradigm introduces distribution shift, which often pushes the policy beyond the trust region, leading to training instabilities manifested as fluctuations in policy entropy and unstable gradients. Although PPO-Clip mitigates this issue through importance clipping, it still overlooks the global distributional shift of actions. To address these challenges, we propose using the entropy ratio between the current and previous policies as a new global metric that effectively quantifies the relative change in policy exploration throughout updates. Building on this metric, we introduce an \textbf{Entropy Ratio Clipping} (ERC) mechanism that imposes bidirectional constraints on the entropy ratio. This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions. We integrate ERC into both DAPO and GPPO reinforcement learning algorithms. Experiments across multiple benchmarks show that ERC consistently improves performance.

1 Introduction

Off-policy reinforcement learning supports LLM post-training but can push policies beyond the trust region, producing entropy and gradient instability. The section motivates ERC as a global constraint on exploration change that complements PPO-Clip.

1 Introduction

  • Off-policy updates can deviate from the trust region, with entropy fluctuations and exploding or vanishing gradient magnitudes impairing training.The data may come from older behavior policies, and importance sampling's high variance can destabilize update step size.Source: §1 · p. 1
  • PPO-Clip constrains sampled-action probability changes but leaves unsampled actions unconstrained, allowing global distribution drift to accumulate.The introduction illustrates that drift can occur even when the sampled action's probability changes only slightly.Source: §1 · p. 2
  • ERC adds hard truncation based on entropy change to PPO-Clip, constraining the entropy ratio and complementing its local sampled-action constraint.The authors report that this stabilizes entropy values and gradients throughout training.Source: §1 · p. 2
  • The entropy ratio measures relative change in policy exploration and is proposed as a global measure of policy-distribution drift across updates.The work evaluates ERC across multiple reinforcement-learning algorithms and benchmarks.Source: §1 · pp. 2–3
  • ERC consistently improves performance across multiple benchmarks when integrated into reinforcement-learning algorithms.The reported constraint also narrows and stabilizes the effective trust region, while stabilizing entropy and gradients.Source: §1 · pp. 2–3

2 Preliminary

The preliminary section introduces PPO's trust-region constraints and summarizes GRPO and its DAPO and GPPO variants. These methods differ in how they constrain policy updates and handle sampling or gradients.

2.1 Proximal Policy Optimization

  • PPO stabilizes training by restricting deviation between old and new policies; its two constraint approaches are a KL penalty and probability-ratio clipping.Directly optimizing the standard objective can let the importance ratio deviate excessively and destabilize training.Source: §2.1 · p. 3
  • PPO-Penalty adds a KL-divergence penalty to constrain distributional differences, but pointwise constraints may suppress exploration and tuning its coefficient can be heuristic.The penalty coefficient is β.Source: §2.1 · p. 3
  • PPO-Clip clips the importance ratio within fixed bounds to form a local trust region, but constrains only sampled actions.Unsampled actions remain unconstrained and may drift beyond the trust region.Source: §2.1 · p. 3

Group Relative Policy Optimization (GRPO)

  • GRPO removes explicit value-function estimation and applies standardized group-response advantages in a clipped policy-gradient objective.DAPO builds on GRPO with asymmetric clipping bounds, dynamic sample filtering, and token-level loss aggregation with reward shaping.Source: Group Relative Policy Optimization… · p. 3
  • GPPO preserves gradients when the importance ratio exceeds the clipping range, using constant-scale updates to stabilize training and alleviate gradient truncation.The described concern is that traditional clipping can suppress gradients of high-entropy tokens and slow convergence of negative samples.Source: Group Relative Policy Optimization… · pp. 3–4

3 Method

The method measures policy-distribution change with an entropy ratio, then uses Entropy Ratio Clipping (ERC) to constrain updates beyond PPO-Clip’s local sampled-action constraint. ERC complements PPO-Clip and is reported to stabilize entropy and gradients while preserving exploration.

3.1 Entropy Ratio

  • The entropy ratio measures relative entropy change between new and old policies on the same data, including shifts across the full action distribution and unsampled actions.At each decoding step, it is defined at the token level.Source: §3.1 · p. 4

3.2 Entropy Ratio Clip

  • ERC discards token gradients when the entropy ratio falls outside a predefined range, applying hard truncation to limit sharp shifts in global output distribution and entropy.It becomes active when entropy is about to deviate substantially, unlike KL constraints that continuously restrict the policy.Source: §3.2 · p. 4
  • ERC complements PPO-Clip by addressing global distribution shifts that importance-ratio clipping misses when it considers only sampled actions.Clipping samples with large entropy-ratio deviations is intended to maintain stable entropy between old and new policies.Source: §3.2 · p. 4
  • Compared with PPO-Clip, ERC is reported to stabilize entropy, reduce gradient variance, and support conservative updates while maintaining exploration.The mechanism integrates orthogonally with reinforcement-learning objectives that use importance-ratio clipping.Source: §3.2 · p. 4

4 Experiment

Experiments evaluate ERC on mathematical reasoning benchmarks and compare training dynamics with and without the mechanism. ERC consistently improves performance across nearly all mathematical benchmarks at both tested model scales, while the reported training trajectories show more stable entropy and gradient norms.

4.1 Experimental Setup

  • Training uses KlearReasoner-MathSub-30K, filtering queries based on whether most of 16 distilled responses pass rule-based validators.The dataset is described as containing 30k mathematical reasoning samples and as undergoing filtering and decontamination.Source: §4.1 · pp. 4–5
  • Training uses 1.5B and 7B models, eight sampled responses per query, off-policy batches of 128 prompts, and entropy-ratio bounds βlow = 0.05 and βhigh = 0.05.The stated learning rate is 1e-6; mini-batches contain 16 prompts.Source: §4.1 · p. 5
  • Evaluation covers mathematical reasoning, code reasoning, and instruction-following benchmarks, with task-specific scoring and inference-length settings.Instruction-following results are reported at both prompt and instance levels.Source: §4.1 · p. 5

4.2 Main Results

  • ERC consistently improves performance on nearly all mathematical benchmarks, with larger gains on AIME25 and HMMT25 and improvements at both 1.5B and 7B scales.Table 1 compares baselines with ERC-augmented DAPO across mathematical reasoning benchmarks and model scales.Source: §4.2 · p. 5, Table 1 · p. 6
  • ERC yields smoother training, with more stable entropy trajectories and bounded gradient norms, as Figure 2 compares training dynamics with and without ERC.The text attributes instability in traditional clipping methods to their failure to regulate drift in unsampled actions.Source: §4.2 · pp. 5–6, Figure 2 · p. 5

5 Analysis

ERC improves performance across PPO-Clip and non-clipping RL settings, while analyses show how its global entropy constraint complements local clipping and preserves exploratory updates. Comparisons also report advantages over KL regularization, entropy regularization, and sequence-level clipping.

5.1 ERC Enhances Trust Region Constraints

  • ERC clips tokens near trust-region boundaries that PPO-Clip's local constraints overlook, and its upper and lower bounds restrain drastic distribution fluctuations.The authors describe the two mechanisms as complementary in mitigating trust-region divergence and improving training stability.Source: §5.1 · p. 6

5.2 Maintaining Exploration through ERC

  • ERC preferentially clips low-entropy tokens while generally preserving high-entropy tokens, including reasoning-related words, supporting exploration alongside stability.The clipped set is described as containing mainly deterministic mathematical symbols and computation operators.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 Clipping Ratio Analysis

  • Approximately 20% of tokens are clipped by ERC, versus around 0.02% by PPO-Clip, reflecting ERC's global entropy-ratio signal beyond sampled-action ratios.The authors report that ERC still surpasses PPO-Clip baselines in final performance and training stability.Source: §5.3 · p. 7, Table 2 · p. 7

5.5 ERC vs. KL Regularization

  • ERC outperforms PPO-penalty on AIME24 and AIME25 in the reported evaluation.The comparison uses KL-regularized methods with varying coefficients.Source: Figure 6 · p. 8, §5.5 · p. 8

5.4 The Broader Applicability of ERC

  • ERC improves performance with PPO-Clip in DAPO and with non-clipping GPPO, where it can serve as a primary stability constraint.The GPPO comparison reports consistent gains across mathematical-reasoning benchmarks.Source: §5.4 · pp. 7–8, Table 3 · p. 8

5.6 ERC vs. Entropy Regularization

  • ERC achieves significantly better performance than entropy regularization on AIME24 and AIME25.The authors attribute the difference to ERC's bidirectional clipping, which addresses both directions of entropy fluctuation.Source: §5.6 · p. 8, Figure 7 · p. 9

5.7 Comparison with Sequence-Level Clipping

  • ERC-DAPO consistently outperforms sequence-level clipping on AIME24 and AIME25, and the two clipping approaches are described as orthogonal.The comparison uses DS-R1-Distill-Qwen-7B; the authors state the approaches can be used together.Source: Figure 8 · p. 9, §5.7 · p. 9

6 Conclusion

Trust-region deviation during optimization is identified as a source of LLM reinforcement-learning instability, and PPO-Clip leaves shifts among unsampled actions unconstrained. The conclusion proposes ERC's bidirectional global entropy-ratio constraint and reports consistent baseline improvements alongside stable training and preserved exploration.

  • Trust-region deviation during optimization is identified as a longstanding source of instability in reinforcement learning for large language models.Source: §6 · p. 9
  • ERC uses the entropy ratio between new and old policies to constrain global distribution change bidirectionally, addressing unsampled-action shifts left uncontrolled by PPO-Clip.Source: §6 · p. 9
  • Across multiple model scales, ERC consistently outperforms baseline methods while suppressing trust-region drift, stabilizing training, and preserving necessary exploration.Source: §6 · p. 9

Limitations

The reported evaluation is on mathematical reasoning tasks; generalization to code generation and agent-based reinforcement learning remains an open question because of computational constraints.

  • Generalization beyond mathematical reasoning, including to code generation and agent-based reinforcement learning, remains unvalidated because of computational constraints.Source: Limitations · p. 9

Related papers

Loading 2512.05591v2…