PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin

arXiv:2506.01939v2cs.CLcs.AIcs.LG

TL;DR

RLVR improves LLM reasoning, but its token-level mechanisms remain unclear. The paper analyzes entropy patterns in Chain-of-Thought reasoning and RLVR training, then restricts policy-gradient updates to high-entropy forking tokens, achieving comparable or superior performance to full-token training while low-entropy-only training declines.

  • Problem

    RLVR is widely used for training reasoning models, but which token types contribute most to reasoning-model learning remains insufficiently studied.

  • Method

    The paper analyzes token entropy patterns in Chain-of-Thought reasoning and RLVR training, then retains policy-gradient updates only for the highest-entropy tokens.

  • Results

    Using 20% high-entropy tokens achieves performance comparable to full-gradient updates on Qwen3-8B and surpasses them on Qwen3-32B and Qwen3-14B, while low-entropy-only training degrades performance.

  • Takeaways & Limitations

    High-entropy minority tokens are critical decision points that account for nearly all reported RLVR performance gains and can guide more targeted RLVR optimization.

  • Takeaways & Limitations

    The findings come from specific experimental settings and may not generalize to all RLVR scenarios; the effective token proportion may require adjustment elsewhere.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs), while its mechanisms are not yet well understood. In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence reasoning performance. By examining token entropy patterns in Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of tokens exhibit high entropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways. Furthermore, studying how entropy patterns evolve during RLVR training reveals that RLVR largely adheres to the base model's entropy patterns, primarily adjusting the entropy of high-entropy tokens. These findings highlight the significance of high-entropy tokens (i.e., forking tokens) to RLVR. We ultimately improve RLVR by restricting policy gradient updates to forking tokens and uncover a finding even beyond the 80/20 rule: utilizing only 20% of the tokens while maintaining performance comparable to full-gradient updates on the Qwen3-8B base model and significantly surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71 on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models, highlighting a strong scaling trend. In contrast, training exclusively on the 80% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that decide reasoning directions. Collectively, our results highlight the potential to understand RLVR through a token-entropy perspective and optimize RLVR by leveraging high-entropy minority tokens to further improve LLM reasoning.

1 Introduction

The paper studies RLVR through token entropy, finding that a small high-entropy minority acts as reasoning forks and that RLVR mainly modifies these tokens. Restricting policy-gradient updates to the highest-entropy tokens preserves or improves performance while low-entropy tokens contribute little or can hinder reasoning.

  • RLVR is analyzed through token entropy patterns to identify which tokens most influence reasoning performance.
  • High-entropy minority tokens act as forks that guide Chain-of-Thought reasoning toward diverse pathways, while most low-entropy tokens complete ongoing linguistic structures.
  • 20% high-entropy tokens preserve competitive Qwen3-8B performance and outperform full-gradient updates by +11.04 on AIME’25 and +7.71 on AIME’24 for Qwen3-32B.
  • 20% high-entropy tokens also outperform full-gradient updates by +4.79 on AIME’25 and +5.21 on AIME’24 for Qwen3-14B, while 80% low-entropy-only training severely degrades performance.
  • During RLVR training, entropy patterns largely remain aligned with the base model, with RLVR primarily adjusting high-entropy tokens.
  • The paper discusses high-entropy minority tokens as a possible explanation for why supervised fine-tuning memorizes whereas reinforcement learning generalizes.

2 Preliminaries

This section defines token-generation entropy and introduces PPO, GRPO, and DAPO as RLVR optimization frameworks. It also distinguishes entropy at a generation index from the identity of a sampled token.

  • 2.1 Token entropy calculation: Token entropy H_t is the uncertainty of the model’s vocabulary distribution at generation index t.The distribution is produced by π_θ conditioned on the query and previously generated tokens.
  • 2.1 Token entropy calculation: Token entropy refers to the generation distribution at an index, so repeated token identities can have different entropy values.Entropy is associated with p_t rather than uniquely with the sampled token o_t.
  • 2.2 RLVR Algorithms: PPO stabilizes RLVR by constraining policy updates near the old policy through a clipped surrogate objective.Its objective uses the probability ratio between the current and old policies and an estimated advantage.
  • 2.2 RLVR Algorithms: GRPO removes the value network by estimating advantages from average rewards within groups of sampled responses, while adding a KL penalty.Its reward assigns 1 for an equivalent answer and 0 otherwise.
  • 2.2 RLVR Algorithms: DAPO builds on GRPO with clip-higher, dynamic sampling, token-level policy gradients, and overlong reward shaping, and serves as the experimental baseline.The paper describes DAPO as an RLVR algorithm without a value network.

3 Analyzing Token Entropy in Chain-of-Thought Reasoning

Token entropy in CoT is highly uneven: most tokens have low entropy, while a minority bridge reasoning components and act as forking points. Raising temperature specifically for these tokens benefits AIME performance more than raising temperature for other tokens.

  • 3 Analyzing Token Entropy in Chain-of-Thought Reasoning: 50.64% of CoT tokens have entropy below 10^-2, while only 20% exceed 0.672.The distribution therefore contains a low-entropy majority and a high-entropy minority.
  • 3 Analyzing Token Entropy in Chain-of-Thought Reasoning: Highest-entropy tokens typically bridge consecutive reasoning parts, whereas lowest-entropy tokens complete sentences or words.Intermediate tokens can combine bridging and continuation functions to varying degrees.
  • 3 Analyzing Token Entropy in Chain-of-Thought Reasoning: High-entropy tokens are termed forking tokens because they can lead to different potential reasoning branches.The paper tests their role by assigning separate temperatures to forking and other tokens.
  • 3 Analyzing Token Entropy in Chain-of-Thought Reasoning: The 0.672 entropy threshold identifies forking tokens as the top 20% of sampled tokens, while T_high and T_low control their respective temperatures.The threshold is estimated as the 80th percentile among the sampled tokens.
  • 3 Analyzing Token Entropy in Chain-of-Thought Reasoning: Lowering T_high degrades performance more than lowering T_low, while increasing T_high improves performance more than increasing T_low.Increasing T_low can cause nonsensical outputs, according to the reported AIME experiments.

4 RLVR Preserves and Reinforces Base Model Entropy Patterns

RLVR largely preserves the base model’s entropy pattern while selectively changing high-entropy tokens. Tokens with higher initial entropy tend to receive larger entropy increases, whereas low-entropy tokens remain comparatively stable.

  • Entropy-pattern preservation: RLVR preserves the base model’s entropy pattern by tracking overlap in the positions of the top 20% high-entropy tokens across training.The comparison uses the base model at step 0 and the RLVR-trained model at step 1360.
  • Entropy-pattern reinforcement: RLVR primarily alters high-entropy tokens, while low-entropy tokens undergo comparatively minimal entropy variation.This selective change is reported across the entropy-percentile analysis during training.
  • Entropy-pattern reinforcement: Tokens with higher initial entropy tend to experience greater entropy increases after RLVR.Figure 4 groups tokens into 5% entropy percentile ranges of the base model, with entropy change shown on a log-scaled y-axis.

5 High-Entropy Minority Tokens Drive Effective RLVR

This section evaluates RLVR training that retains policy-gradient updates only for high-entropy tokens, comparing it with full-token training across model sizes and token-selection ranges. The results show that top-20% high-entropy tokens preserve or improve reasoning performance, whereas low-entropy tokens contribute little and can reduce performance.

  • 5.1 Formulation of RLVR Using Only Policy Gradients of the Highest-Entropy Tokens: RLVR training retains policy gradients only for the top-ρ high-entropy tokens selected by a batch entropy threshold.The method masks low-entropy-token gradients while normalizing over the selected tokens.
  • 5.3 Main Results: 20% high-entropy tokens maintain Qwen3-8B performance, while removing the bottom 80% low-entropy tokens stabilizes training and improves AIME’24 results for Qwen3-32B and Qwen3-14B.Figure 6 reports AIME’24 gains of 7.73 for Qwen3-32B and 5.21 for Qwen3-14B.
  • 5.3 Main Results: +11.04 on AIME’25 and +7.71 on AIME’24: Qwen3-32B improves over vanilla DAPO when only the top 20% high-entropy tokens receive policy-gradient updates.Qwen3-14B also improves by +4.79 on AIME’25 and +5.21 on AIME’24, while Qwen3-8B remains comparable to full-gradient training.
  • 5.3 Main Results: Retaining only the bottom 80% low-entropy tokens causes a substantial performance decline despite covering most training tokens.The result indicates that low-entropy tokens contribute minimally to enhancing reasoning capabilities.
  • 5.3 Main Results: Approximately 20% high-entropy tokens balances exploration and training stability, while selecting 10%, 50%, or 100% produces lower overall entropy during early training.For Qwen3-14B and Qwen3-32B, reducing the fraction to 10% slightly lowers performance and increasing it to 100% causes a notable decline; Qwen3-8B is relatively consistent across tested proportions.
  • 5.3 Main Results: The performance gain over vanilla DAPO becomes increasingly significant as model size increases when policy gradients focus on forking tokens.Increasing the maximum response length from 20480 to 29696 raises Qwen3-32B AIME’24 performance from 63.54 to 68.12.

6 Discussions

The discussion situates high-entropy tokens as distinctive to LLM reasoning and argues they may help explain RLVR’s behavior and generalization. It also contrasts entropy-targeting choices in RLVR with traditional reinforcement learning.

  • Discussion 1: High-entropy minority tokens may help explain why reinforcement learning generalizes while supervised fine-tuning tends to memorize training data.The paper presents this as a hypothesis grounded in prior empirical findings.
  • Discussion 2: LLM Chain-of-Thought contains low-entropy majority tokens and high-entropy minority tokens, unlike traditional RL’s more uniform action entropy.The paper attributes this difference to pretrained prior knowledge and the requirement for fluent language output.
  • Discussion 3: Entropy bonus may be suboptimal for RLVR because it raises entropy in low-entropy majority tokens, whereas clip-higher promotes entropy in high-entropy minority tokens.This comparison concerns how the two mechanisms distribute exploration pressure across tokens.

7 Related Work

The related work reviews reinforcement learning for LLM alignment and reasoning, then positions this paper’s entropy-based token criterion alongside prior analyses of reasoning and critical tokens.

  • Reinforcement learning for LLM: RL was first widely used in RLHF to improve LLM instruction-following and alignment with human preferences.The section notes online and offline preference-optimization categories, including methods such as PPO and GRPO.
  • Analysis on reinforcement learning with verifiable rewards: Prior RLVR studies identify reasoning behavior, long Chain-of-Thought structure, and critical decision tokens as important to learning or model behavior.These studies differ in whether they emphasize behaviors, chain structure, or error-prone decision points.
  • Analysis on reinforcement learning with verifiable rewards: Unlike prior work judging token importance by output correctness, this paper proposes token entropy as a criterion for reflecting LLM mechanisms.The criterion is presented as a different lens for analyzing token importance.

8 Conclusion

The paper concludes that high-entropy minority tokens are central to understanding and optimizing RLVR, while acknowledging limits in model, domain, and experimental-setting coverage. It points to broader applications of these insights as future work.

  • 8 Conclusion: High-entropy tokens form a small subset of Chain-of-Thought tokens that act as reasoning forks, and RLVR mainly modifies their entropy.The model largely retains the base model’s entropy patterns during RLVR training.
  • 8 Conclusion: Restricting policy-gradient updates to the top 20% highest-entropy tokens matches or surpasses full-token RLVR, while optimizing low-entropy tokens significantly reduces performance.The reported gains show a strong scaling trend with model size.
  • Limitations: The experiments are limited to specific settings, primarily Qwen models and mathematics benchmarks, so the findings may not generalize to all RLVR scenarios.The authors specifically note that the effective 20% proportion may need adjustment in other settings.
  • Future Directions: Future work includes leveraging high-entropy tokens in RLVR and exploring applications to SFT, distillation, inference, and multimodal training.The paper frames these as directions for developing new algorithms and extending the analysis.
Loading 2506.01939v2…