PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models

Shumin Wang, Yuexiang Xie, Wenhao Zhang, Yuchang Sun, Yanxi Chen, Yaliang Li, Yanyong Zhang

arXiv:2602.03392v1cs.LGcs.AI

TL;DR

RFT uses entropy to monitor output diversity, but existing entropy-control methods lack a principled account of entropy dynamics. The paper develops a single-logit and GRPO framework centered on an entropy discriminator, finding empirical support for its reliability and for clipping methods that stabilize entropy and promote exploration.

  • Problem

    Existing RFT entropy methods often rely on heuristic designs, while conflicting entropy effects reveal limited understanding of entropy dynamics during training.

  • Method

    The paper derives token-level entropy change from a single-logit update, extends it to first-order GRPO dynamics, and uses the resulting discriminator to design and interpret entropy controls.

  • Results

    Experiments show that S∗ reliably discriminates entropy dynamics and that derived clipping methods stabilize entropy in RFT to promote model exploration.

  • Takeaways & Limitations

    The framework explains entropy collapse through update direction and token probability, while providing theoretical support for entropy-control strategies in RFT.

  • Takeaways & Limitations

    The first-order analysis assumes auxiliary regularization is inactive and ignores tokens that trigger logit clipping.

Abstract

from arXiv · show

Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning (RFT), a principled understanding of entropy dynamics during this process is yet to be thoroughly investigated. In this paper, we establish a theoretical framework for analyzing the entropy dynamics during the RFT process, which begins with a discriminant expression that quantifies entropy change under a single logit update. This foundation enables the derivation of a first-order expression for entropy change, which can be further extended to the update formula of Group Relative Policy Optimization (GRPO). The corollaries and insights drawn from the theoretical analysis inspire the design of entropy control methods, and also offer a unified lens for interpreting various entropy-based methods in existing studies. We provide empirical evidence to support the main conclusions of our analysis and demonstrate the effectiveness of the derived entropy-discriminator clipping methods. This study yields novel insights into RFT training dynamics, providing theoretical support and practical strategies for optimizing the exploration-exploitation balance during LLM fine-tuning.

1. Introduction

RFT uses entropy to diagnose output diversity and the exploration–exploitation trade-off, but existing entropy controls lack a principled account of entropy dynamics. The paper develops a theoretical framework, extends it to GRPO, and supports entropy-discriminator clipping experimentally.

  • RFT frames LLM fine-tuning as policy optimization, making exploration versus exploitation a central challenge for training stability and avoiding local optima.
  • Entropy measures output diversity and is used to monitor training dynamics and regulate policy behavior in RFT.
  • Existing entropy methods often use heuristic, isolated adjustments, creating unclear guidance and labor-intensive hyperparameter tuning.
  • The framework characterizes token-level entropy change, extends the analysis to a practical GRPO step, and relates change direction to token updates and discriminator score S∗.
  • The analysis provides a principled lens for designing and interpreting entropy-control methods, including a unified account of their effects and mechanics.
  • Experiments find S∗ reliable for discriminating entropy dynamics and show derived clipping methods stabilize entropy to promote exploration in RFT.

2. Preliminaries

The preliminaries define GRPO as a group-based policy-optimization procedure and introduce entropy dynamics as the study of how token-triggered updates alter output-distribution entropy. This setup connects probability updates to response diversity and repetitive outputs.

  • GRPO samples a group of responses for each query, assigns scalar rewards, and updates the policy using token-level response contributions.
  • Under strict on-policy training, the importance ratio equals 1, while positive advantages increase sampled-token probability and negative advantages decrease it.
  • Entropy dynamics examines how a parameter update triggered by one sampled token changes the output token distribution’s entropy at that step.
  • The framework relates entropy change ΔH_t to the policy probability of the sampled token to determine whether updates encourage diversity or repetitive outputs.

3. Analysis of the Entropy Dynamics in RFT

The analysis derives entropy dynamics first for a single-logit perturbation and then for GRPO. It shows that entropy change depends on update direction, token probability, entropy, and a policy-weighted discriminator baseline, motivating principled control strategies.

  • Single-logit analysis: The framework quantifies how a single-token update changes policy entropy and derives a first-order expression for a GRPO optimization step.
  • Single-logit analysis: A logit perturbation δz = ε·e_k rewards a token when ε is positive and penalizes it when ε is negative.
  • Single-logit analysis: A single-logit update increases the selected token’s probability and redistributes probability mass proportionally across all other tokens.
  • Single-logit analysis: The entropy-change direction is determined by the signs of the update ε and discriminator S∗, whose sign depends on token probability relative to e^-H(p).
  • Single-logit analysis: Rewarding a low-probability token increases entropy, whereas rewarding a high-probability token decreases entropy; penalization reverses these effects.
  • Single-logit analysis: Consistently rewarding high-probability responses can produce rapid entropy collapse and gradual loss of exploratory capability.
  • GRPO extension: For GRPO, entropy change depends on S∗’s deviation from the policy-weighted expectation E_i∼p[S_i], which acts as a dynamic baseline.
  • GRPO extension: With on-policy sampling, the expected entropy-change factor is zero across the vocabulary and training batch, supporting constraints based on S∗ relative to its expectation.

4. Bridging Entropy Dynamics to Entropy Control Methods

The paper connects entropy dynamics to practical control methods by using the discriminator score to identify token updates that influence entropy and motivate clipping strategies. It then interprets existing clipping, entropy-regularization, and probability-weighted methods through this framework.

  • Entropy Discriminator Guided Clipping: The discriminator score S∗ and its relationship to entropy dynamics provide the basis for identifying tokens with disproportionate effects on entropy changes.This motivates selectively mitigating outlier-token influence for fine-grained entropy control.
  • Entropy Discriminator Guided Clipping: Batch-normalized clipping computes token-level discriminator statistics and masks updates that contribute to severe fluctuations in ∆H.The method uses batch means and variances to define token masks, while operating on scalar values with negligible additional computation.
  • Entropy Discriminator Guided Clipping: Vocabulary-normalized clipping is motivated by the GRPO-specific characterization of entropy change and evaluates a vocabulary-level discriminator normalization.The required expectation can use logits already available from the forward pass, limiting additional computational cost.
  • Interpreting Existing Methods through Entropy Dynamics: The framework reinterprets existing entropy-based methods as amplifying entropy-increasing token effects or suppressing entropy-decreasing ones.The related methods include clipping mechanisms, entropy regularization, and probability-weighted updating.
  • Interpreting Existing Methods through Entropy Dynamics: For positive samples, higher-probability tokens are often associated with entropy decreases, whereas lower-probability tokens are often associated with entropy increases; negative samples show the reverse trend.This relationship links token probability and entropy through the discriminator-based analysis.
  • Interpreting Existing Methods through Entropy Dynamics: GRPO clipping primarily affects low-probability tokens, and relaxing positive-sample clipping preserves updates associated with increasing token entropy.The clip-higher interpretation therefore connects a clipping choice to counteracting entropy decline and promoting exploration.

5. Experiments

Experiments support the proposed entropy discriminator and show that ClipB and ClipV stabilize entropy while improving exploration-related performance over standard GRPO.

  • 5.2. Empirical Observations of the Entropy Dynamics: Selective updates and gradient masking support a relationship between the sign of S∗ and the direction of token-entropy change.Masking S∗>0 or S∗<0 tokens produces opposite entropy behaviors across positive and negative samples.
  • 5.2. Empirical Observations of the Entropy Dynamics: The batch-averaged deviation S∗−E_i∼p[S_i] approaches zero and is three orders of magnitude smaller than the batch-averaged S∗.This empirically supports Corollary 3.5.
  • 5.3. Effects of Entropy Discriminator Clipping Methods: ClipB and ClipV use µ to control clipping and both mitigate entropy decay to excessively low levels relative to standard RFT.The experiments evaluate their effects on clip fraction and entropy across different µ values.
  • 5.3. Effects of Entropy Discriminator Clipping Methods: Both ClipB and ClipV outperform standard GRPO across AIME24, AIME25, and DAPO500, supporting entropy control as a way to preserve exploration.The comparison uses Avg@K and Pass@K, with K=32 for AIME24/25 and K=8 for DAPO500.
  • 5.4. Analysis of Exploration versus Exploitation: The methods retain consistent performance gains when integrated with PPO and evaluated on Qwen3-4B-Base, DeepSeek-Distilled-Llama3-8B, and InternLM3-8B.These results are reported as evidence of effectiveness across training algorithms and model families.
  • 5.4. Analysis of Exploration versus Exploitation: Stabilizing entropy improves both Pass@K and Avg@K and encourages discovery of correct reasoning paths across a wider range of problems.Pass@K reflects solution diversity, while Avg@K primarily reflects exploitation of similar high-reward patterns.
  • 5.4. Analysis of Exploration versus Exploitation: ClipB yields fewer completely solved or completely failed problems than standard GRPO, indicating a more exploratory pass-rate distribution.Standard GRPO shows a higher proportion of problems at the two extremes across multiple rollouts.

6. Related Works

Related work applies entropy-based strategies to RFT, while recent studies motivate a more systematic analysis of entropy dynamics and performance.

  • 6. Related Works: RFT methods include entropy regularization, entropy-based token selection, flexible clipping schemes, and other entropy-oriented strategies.These approaches target model behavior from different aspects of reinforcement fine-tuning.
  • 6. Related Works: Recent work connects entropy changes with sampling distributions and performance, highlighting the importance of studying entropy dynamics in RFT.This motivates the paper’s theoretical framework.

7. Conclusions

The paper presents a theoretical framework linking token updates, policy distributions, and entropy changes, and uses these insights to explain entropy collapse and guide entropy control.

  • 7. Conclusions: The framework characterizes token-level entropy change and derives a first-order expression for entropy change during a GRPO policy update.It provides a microscopic analysis of single-token updates before extending the analysis to GRPO.
  • 7. Conclusions: The analysis links entropy fluctuations to a token’s update direction, probability, and policy entropy, providing a unified view of entropy-control methods.The framework is used to interpret mechanisms that amplify entropy-increasing tokens or suppress entropy-decreasing tokens.

A. Proof of Corollaries

The corollaries show that expected entropy-change factors vanish under on-policy GRPO sampling at both token and batch levels.

  • A. Proof of Corollaries: To first order, the expected entropy change of a token is zero under on-policy sampling in GRPO.This is stated as Corollary 3.4.
  • A. Proof of Corollaries: For an on-policy GRPO batch, the expected entropy-change factor S_i is zero across the batch.This is stated as Corollary 3.5.
  • A. Proof of Corollaries: The proof conditions on each token’s probability, averages over the batch using linearity of expectation, and then removes conditioning with the tower property.The derivation proceeds through conditional expectation and batch averaging.

B. Detailed Experiment Setup

Experiments use specified hardware, framework, optimizer settings, rollout configurations, and correctness-based rewards.

  • Experiments run on NVIDIA A100 and H20 GPUs using the Trinity-RFT framework.
  • Training uses Adam with hyperparameters (0.9, 0.999), batch size 64, and learning rate 4 × 10−7.Rollouts number 16 for Qwen2.5-7B/14B-Instruct and 8 for other models.
  • Sampling and evaluation temperatures are 1.0 and 0.7, respectively.
  • Each response receives reward 1 when its answer and format are correct, and 0 otherwise.

C. Extension to Advantage-Aware Analysis

The advantage-aware analysis extends token-level entropy theory to GRPO sampling and batches, yielding covariance-based expressions and practical discriminator-based clipping, while exposing an implementation limitation.

  • Extension to Advantage-Aware Analysis: The analysis extends entropy-change results to advantage estimation from both model-sampling and batch-averaging perspectives.
  • Model Sampling: Corollary C.1 gives an on-policy GRPO expression that decouples advantage from the core entropy-change term through covariance.
  • Model Sampling: Corollary C.1 cannot be directly implemented because advantages for unsampled tokens are undefined and non-computable.
  • Batch Perspective: Corollary C.2 provides a computable batch-level covariance form and motivates monitoring −CovB(A, S∗−E_i∼p[S_i]) during training.
  • Entropy Control: The clipping algorithms compute S∗−E_i∼p[S_i] and filter tokens contributing strongly to entropy fluctuations, with batch-level normalization improving efficiency.
  • Parameter Sharing: The framework models token effects as linearly superimposed, while rigorous treatment of parameter-sharing interference remains open.

D. Extension to off-policy scenarios

The paper extends its GRPO entropy-change analysis to off-policy sampling and evaluates entropy-control methods with PPO, reporting applicability beyond the original setting.

  • Off-Policy Extension: The GRPO derivation extends to off-policy sampling by incorporating the importance ratio r = πθ/πθsample.
  • Off-Policy Extension: The expected entropy-change factor r(S∗−E_i∼p[S_i]) is zero to first order under the stated GRPO conditions.
  • Off-Policy Extension: Off-policy corollaries provide first-order token-wise and batch-wise entropy-change expressions using sampling and current-policy distributions.
  • PPO Extension: The paper evaluates ClipB/ClipV with PPO using GAE advantage signals on Qwen2.5-7B-Instruct.
  • PPO Extension: PPO experiments report significant improvements for the proposed methods.
  • Implications: The authors identify entropy-control methods tailored to different RFT algorithms as a promising direction.

E.3. Experiments with More Models

Experiments across Qwen3, Distilled-Llama, and InternLM test whether the entropy-control methods generalize across model families and training dynamics.

  • Experimental Scope: Additional experiments use Qwen3-4B-Base, DeepSeek R1-Distill-llama-8B-Instruct, and InternLM3-8B-Instruct.
  • Results: The methods outperform baselines in most scenarios, indicating generalized effectiveness across different models.
  • Qwen3: For Qwen3, the methods alleviate entropy collapse during training.
  • Distilled-Llama: For Distilled-Llama, the method stabilizes entropy and achieves competitive model performance despite different training dynamics.
  • InternLM: For InternLM, the method provides useful benefits for stabilizing training.
Loading 2602.03392v1…