PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning with Exploration: An Entropy Perspective

Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, Furu Wei

arXiv:2506.14758v4cs.CL

TL;DR

The paper addresses the tendency of accuracy-driven RL to favor exploitation and limit exploratory reasoning in LLMs. It analyzes entropy as an exploration signal and augments standard RL advantages with a clipped, gradient-detached entropy term. The method consistently improves Pass@1 and substantially improves Pass@K, including at large K values across benchmarks.

  • Problem

    Accuracy-driven RL can narrow LLM behavior and limit exploratory capacity, motivating analysis of entropy’s relationship with exploratory reasoning.

  • Method

    The method augments standard PPO or GRPO advantages with a clipped, gradient-detached entropy term to promote longer and deeper reasoning chains.

  • Results

    The method consistently outperforms baselines across benchmarks and RL algorithms, with improvements extending to Pass@K at large K values.

  • Takeaways & Limitations

    Entropy-based advantage shaping supports exploratory reasoning while preserving the original policy optimization direction.

Abstract

from arXiv · show

Balancing exploration and exploitation is a central goal in reinforcement learning (RL). Despite recent advances in enhancing large language model (LLM) reasoning, most methods lean toward exploitation, and increasingly encounter performance plateaus. In this work, we revisit entropy -- a signal of exploration in RL -- and examine its relationship to exploratory reasoning in LLMs. Through empirical analysis, we uncover positive correlations between high-entropy regions and three types of exploratory reasoning actions: (1) pivotal tokens that determine or connect logical steps, (2) reflective actions such as self-verification and correction, and (3) rare behaviors under-explored by the base LLMs. Motivated by this, we introduce a minimal modification to standard RL with only one line of code: augmenting the advantage function with an entropy-based term. Unlike traditional maximum-entropy methods which encourage exploration by promoting uncertainty, we encourage exploration by promoting longer and deeper reasoning chains. Notably, our method achieves significant gains on the Pass@K metric -- an upper-bound estimator of LLM reasoning capabilities -- even when evaluated with extremely large K values, pushing the boundaries of LLM reasoning.

1 INTRODUCTION

Accuracy-driven RL methods can improve LLM reasoning but may narrow behavior and weaken sustained exploration. The paper links entropy to exploratory reasoning and proposes entropy-based advantage shaping, reporting gains in Pass@1 and Pass@K.

  • Accuracy-driven RL can converge on narrow, over-optimized behaviors, weakening sustained multi-step reasoning and causing plateaus or regressions.
  • High entropy correlates with pivotal tokens, reflective actions, and rare behaviors associated with exploratory reasoning.
  • The method adds a clipped, gradient-detached entropy term to the advantage function, encouraging longer and deeper reasoning while preserving policy optimization direction.Clipping prevents the entropy term from dominating or reversing the original advantage, while gradient detachment preserves the original optimization behavior.
  • The approach is evaluated with GRPO and PPO and consistently improves Pass@1 accuracy across benchmarks.
  • The method achieves substantial Pass@K improvements, extending gains to a metric treated as an upper-bound estimator of reasoning capability.

2 PRELIMINARY ANALYSIS: ENTROPY AND EXPLORATORY REASONING

The analysis examines whether entropy identifies exploratory reasoning in LLM outputs. Higher entropy is associated with pivotal tokens, reflective sentences, and rare behaviors, which are operationalized through token analysis, sentence classification, and embedding distance.

  • Token-level entropy visualization on Qwen2.5-Base-7B mathematical responses distinguishes high-entropy reasoning dynamics from low-entropy ones.
  • Pivotal Tokens: Pivotal tokens such as “first,” “recall,” and “thus” have significantly higher entropy and mark decision points connecting reasoning steps.
  • Reflective Actions: Reflective sentences containing actions such as “verify” or “check” consistently exhibit higher average entropy.Responses are segmented into sentences, whose average entropy is compared after reflective actions are identified with regular expressions.
  • The parallel analysis on DeepSeek-R1-Distill-Qwen-1.5B code reasoning tasks yields consistent observations.
  • Rare Behaviors Emergent During RL: Rare behaviors are defined as RL-generated sentences in the top 10% of average SBERT distance from their five nearest base-model neighbors.The analysis uses semantic isolation from the base model’s output distribution to identify under-explored behaviors.

3 METHOD

The method shapes PPO or GRPO advantages with a clipped, gradient-detached entropy term to encourage exploratory reasoning while preserving the original optimization direction. Its self-regulating dynamics amplify high-entropy actions initially, then reduce shaping as confidence increases.

  • Entropy-Based Advantage Shaping: Entropy-based advantage shaping injects an entropy-derived term into per-token advantages during PPO or GRPO policy optimization.The method is designed as a minimal modification to existing RL training pipelines before policy-loss computation.
  • RL Baselines: The method integrates with mainstream policy-optimization algorithms, including PPO and GRPO, by replacing the original advantage with the shaped advantage.PPO uses likelihood ratios and advantages, while GRPO uses group-relative rewards as its baseline.
  • Entropy-Based Advantage Shaping: The entropy term is clipped so it cannot dominate the original advantage or reverse its sign when the advantage is negative.Clipping preserves the original optimization direction while adjusting update magnitude.
  • Self-Regulating Dynamics: High entropy initially amplifies the advantage, increasing confidence in selected tokens; sharper distributions then lower entropy and reduce later entropy-based shaping.The self-regulating effect is described as empirically validated in Figure 7.
  • Entropy Regularization Comparison: Gradient detachment makes entropy act through adjusted advantage values rather than an additional entropy gradient, preserving the base RL gradient flow.This distinguishes the method from entropy regularization, which adds a policy-dependent entropy-gradient component.

4 EXPERIMENT SETTINGS

Experiments use Qwen2.5 base and math-adapted 7B models, DAPO-sourced data, and GRPO or PPO training with established baseline techniques. Evaluation covers mathematical benchmarks using Pass@1 and Pass@K-oriented reasoning assessments.

  • Backbone Models: Experiments use Qwen2.5-Base-7B and Qwen2.5-Math-Base-7B as the two backbone models.Initial Llama-series RL training was also attempted, but intermediate reasoning chains were abandoned within a few iterations.
  • RL Training Configuration: Training data come from DAPO, with +1 reward for correct final answers and -1 otherwise, using GRPO and PPO in veRL.Baselines incorporate Clip-Higher, Token-level Loss, Critic-Pretraining, and Group-Sampling.
  • RL Training Configuration: The entropy-shaping experiments fix κ to 2, using α = 0.4 for GRPO and α = 0.1 for PPO.These coefficients are held as the reported settings across the experiments.
  • Evaluation Benchmarks and Metrics: Evaluation uses AIME 2025/2024, AMC 2023, and MATH500 with temperature 0.6, maximum response length 8K tokens, and top-p 0.95.The datasets are evaluated multiple times, reporting average Pass@1 accuracy and also assessing reasoning boundaries with Pass@K.

5 RESULTS

The method consistently outperforms baselines across benchmarks and RL algorithms, with gains extending to large-K Pass@K evaluations where most baselines plateau.

  • The method consistently outperforms baselines across benchmarks and RL algorithms, achieving superior average performance.
  • Pass@K improvements persist at large K values, where most baselines plateau.
  • On AIME2024, AMC2023, and MATH500, base models can surpass RL-finetuned models at sufficiently large K, indicating limited exploratory capacity under conventional RL.
  • On AIME2025, the method exceeds both RL baselines and the base model’s performance ceiling.

6 ANALYSIS

Training and testing analyses show that entropy-based advantage shaping sustains longer responses, preserves stable entropy dynamics, and strengthens exploratory reasoning without increasing redundancy.

  • 6.1 RL TRAINING PROCESS: Entropy-based advantage produces slightly higher rewards in later training stages than the compared methods.
  • 6.1 RL TRAINING PROCESS: Response length continues rising beyond 1000 steps with entropy-based advantage, surpassing both the RL baseline and entropy regularization.
  • 6.1 RL TRAINING PROCESS: At step 2000, entropy is 0.17 for the method and 0.34 for the baseline, while removing clip-higher lowers entropy to 0.03.
  • 6.1 RL TRAINING PROCESS: Entropy regularization increases entropy but spikes after step 1500, whereas entropy-based advantage is reported to improve stable training and reasoning performance.
  • 6.1 RL TRAINING PROCESS: The method amplifies high-entropy exploratory actions while gradually building confidence at those decision points rather than increasing entropy uniformly.
  • 6.2 EXPLORATORY REASONING DYNAMICS: Testing shows significantly more pivotal tokens and reflective actions, alongside longer responses with repetition rates comparable to the RL baseline.
  • 6.2 EXPLORATORY REASONING DYNAMICS: Case studies show more accurate and mathematically rigorous solutions through explicit constraints, systematic case analysis, and adjustments after failed attempts.

7 RELATED WORK

The work builds on reinforcement-learning research that treats exploration as a central problem and entropy as a signal for guiding policy exploration, alongside varied reinforcement fine-tuning signals.

  • Prior RL research addresses exploration through theoretical frameworks and empirical heuristics.
  • Reinforcement fine-tuning uses supervised signals such as human feedback or verifiable correctness, while unsupervised approaches reduce dependence on human annotations through consistency-based signals.

8 CONCLUSION

The paper links entropy to exploratory reasoning and introduces entropy-based advantage shaping to promote deeper reasoning, reporting substantial Pass@K improvements across benchmarks with GRPO and PPO.

  • Pivotal tokens, reflective actions, and rare behaviors consistently align with higher-entropy regions in LLM reasoning.
  • The method adds a clipped, gradient-detached entropy term to standard RL advantages while preserving the original policy optimization direction.
  • Validation with GRPO and PPO shows substantial Pass@K improvements across diverse benchmarks.

A EXTENDED PRELIMINARY ANALYSIS

The code-domain analysis evaluates entropy patterns using DeepSeek-R1-Distill-Qwen-1.5B on LiveCodeBench, following the DeepCoder training configuration.

  • DeepSeek-R1-Distill-Qwen-1.5B is used to validate entropy findings in the code domain.The evaluation uses LiveCodeBench, with RL training configured according to DeepCoder.
  • Figure 10 compares entropy between exploratory reasoning and other reasoning on the code domain.

B EXPERIMENT SETTINGS

The experiments use specified training data, verifiers, and RL hyperparameters for Qwen-based models under GRPO and PPO.

  • Training Data and Reward Function: Qwen2.5-Base-7B uses DAPO’s original prompt template and verifier for training.Qwen2.5-Math-Base-7B receives a step-by-step reasoning suffix and uses Math-Verify as verifier.
  • RL Training Configuration: GRPO and PPO training uses the hyperparameters listed in Table 4 without entropy or KL losses.For PPO, the critic uses learning rate 2e−6 and 10-step pre-training; both methods use GAE with λ = 1.0 and γ = 1.0.

C CASE STUDY

The case study constructs a positive-integer list satisfying a sum of 30, unique mode 9, and a median absent from the list, while illustrating systematic trial-and-check reasoning.

  • Question and Ground Truth: The task asks for the sum of squares of a positive-integer list whose sum is 30, unique mode is 9, and median is absent from the list.
  • Question and Ground Truth: The case study also includes an earlier attempted configuration [1, 9, 9, 9, 7] and reports a conflicting answer of 251.The supplied task statement separately gives a ground-truth value of 236.
  • Constructing the List: The solution represents the sorted list as a1, a2, . . . , an and analyzes odd- and even-length median cases.For even n, the median is the average of the two middle values; for odd n, it is the middle value.
  • Constructing the List: The case study tests candidate list sizes and value assignments while preserving the sum and unique-mode constraints.The n = 7 and n = 8 trials with three 9s leave insufficient positive remainder, motivating a return to n = 6.
  • Constructing the List: The n = 6 construction uses [1, 1, 2, 9, 7, 10], with sum 30, unique mode 9, and median 4.5 absent from the list.
Loading 2506.14758v4…