PEAR: Phase Entropy Aware Reward for Efficient Reasoning

• The listed authors are Chen Huang, Wei Lu, and Wenxuan Zhang.Source: PEAR: PHASE ENTROPY AWARE REWARD FOR… · p. 1

1 INTRODUCTION

• LRMs can generate excessively long chain-of-thought responses with redundant calculations or verbose explanations, making efficient reasoning without sacrificing performance a key challenge.Source: §1 · p. 1
• Concise-data training may hinder adaptation to novel reasoning styles or out-of-distribution problems and risks discarding useful intermediate reasoning.Source: §1 · p. 1
• Entropy is positively correlated with response length, and thinking-phase entropy is higher than final-answer entropy, reflecting exploratory versus more deterministic generation.Figure 2 presents entropy–length relationships and compares average entropy across the two phases.Source: §1 · p. 2, Figure 2 · p. 3
• PEAR penalizes excessive thinking-phase entropy while moderating final-answer entropy to encourage concise reasoning without explicit length targets or rigid truncation.The approach uses entropy differences between the two response phases during training.Source: §1 · p. 2, Figure 1 · p. 2
• 37.8% to 59.4% shorter responses, with accuracy decreases of less than 1%, were reported across four benchmarks and different model scales.The benchmarks are GSM8K, MATH500, AIME24, and AMC23; the paper also reports strong generalization to out-of-distribution tasks.Source: §1 · pp. 2–3

2.1 ENTROPY AND RESPONSE LENGTH

• Across examined model families and benchmarks, higher average entropy aligns with longer, more exploratory responses, making entropy an interpretable signal for response-length control.The analysis compares each model’s average token-level entropy with the total number of generated tokens.Source: §2.1 · pp. 3–4

2.2 PHASE-DEPENDENT ENTROPY ANALYSIS

• The thinking phase has higher entropy than the final-answer phase, reflecting exploration versus a more deterministic commitment to a solution.The paper argues that these complementary phases should be optimized differently.Source: §2.2 · p. 4

2.3 ENTROPY FILTERING EXPERIMENTS

• The filtering experiment removes higher-entropy thinking tokens from generated traces, then feeds the retained low-entropy tokens back to the model to produce answers.The study measures the effects on reasoning efficiency and task accuracy.Source: §2.3 · p. 4
• At 80% or 60% low-entropy-token retention, accuracy stays stable or improves; at 40% or less, accuracy drops sharply, while final-answer length remains relatively unchanged.The filtering results locate redundancy mainly in the thinking phase, where high-entropy tokens are associated with over-elaboration.Source: Figure 3 · p. 4, §2.3 · p. 4

3.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)

• GRPO normalizes rewards across responses to the same prompt to estimate advantages, and its objective includes KL regularization against a reference policy.The normalization emphasizes differences among candidate outputs and stabilizes the gradient signal under sparse rewards.Source: §3.1 · pp. 4–5

3.2 PHASE ENTROPY AWARE REWARD (PEAR)

• PEAR combines average thinking- and final-answer-phase entropy in a phase reward, with α tuning the final-answer entropy contribution.The method aims to reduce thinking-phase entropy while preserving or encouraging final-answer entropy to maintain flexibility and completeness.Source: §3.2 · p. 5
• For correct responses, the phase-aware reward uses a base score and replaces the original GRPO reward before the same advantage normalization is applied.Malformed or incorrect responses receive a format score; the text also specifies fallback handling when the closing thinking token is absent or the answer cannot be parsed.Source: §3.2 · pp. 5–6
• PEAR discourages excessive exploratory reasoning while using final-answer entropy to help stabilize and structure conclusions without sacrificing accuracy.The method is motivated by an observed correlation between entropy and response length.Source: §3.2 · pp. 5–6

4.1 EXPERIMENT SETTING

• Experiments use 7,473 GSM8K training samples and evaluate three LRMs on four mathematical reasoning benchmarks using accuracy and generated tokens.Generation is capped at 16,384 tokens; sampling uses temperature 0.6 and top-p 0.95.Source: §4.1 · p. 6

4.2 EFFECTIVENESS OF PEAR

• For Qwen3-8B, PEAR limits the accuracy decline to 0.91%, versus 1.23% for Step Entropy and 2.68% for LCPO, while compressing more.The authors report that 4B and 8B models achieve over 50% response-length reduction compared with the 1.5B model’s results.Source: §4.2 · pp. 6–7
• 37.8% to 55.2%: PEAR reduces average response length versus original models, with an accuracy decrease of only 0.9%.The comparison spans all benchmarks and evaluated models; Table 1 reports Acc@1 and relative token changes.Source: §4.2 · p. 6, Table 1 · p. 7
• Despite GSM8K-only training, PEAR shows OOD robustness; on Qwen3-4B it matches vanilla accuracy on AIME24 and AMC23 using 34% and 45% of the original reasoning budget.The evaluation covers GSM8K, MATH500, AIME24, and AMC23.Source: §4.2 · p. 7

4.3 HOW PEAR AFFECTS REASONING

• PEAR reduces entropy most in the thinking phase, slightly increases it in final answers, and reduces both reasoning steps and tokens per step.On AIME24, the number of thinking steps falls by more than half; the reduction is concentrated in the thinking phase.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 HYPERPARAMETER STUDY

• α balances answer-phase flexibility and efficiency: α = 0 or −1 harms accuracy, moderate values such as 1 balance performance and redundancy reduction, and very high values diminish efficiency gains.At α = 0, post-thinking entropy is ignored; at α = −1, entropy is penalized in both phases. Very high α makes behavior converge toward the baseline.Source: §4.4 · p. 8, Figure 5 · p. 8

5.1 EFFICIENT REASONING

• Efficiency research includes early-exit rules, CoT compression, adaptive reasoning, and length-constrained rewards; the authors contrast PEAR’s intrinsic entropy signal with methods that discard intermediate reasoning.The cited approaches use internal states, generation behavior, compressed traces, or specified length constraints.Source: §5.1 · p. 9

5.2 REASONING THROUGH ENTROPY CONTROL

• Prior entropy work targets capability, calibration, or exploration, while PEAR uses entropy to control reasoning length without explicit token budgets.The section also describes entropy minimization and selective optimization of high-entropy tokens in RLVR.Source: §5.2 · p. 9

6 CONCLUSION

• The paper reports a positive relation between entropy and response length, with higher entropy during thinking than in the final answer phase.Source: §6 · p. 9
• PEAR discourages thinking-phase entropy while preserving final-answer flexibility, enabling response-length control without explicit targets or rigid truncation.Source: §6 · p. 9
• Across four benchmarks, PEAR reduces token redundancy by 37.8% to 59.4% while preserving accuracy and showing OOD generalization.Source: §6 · p. 9

A ENTROPY FILTERING EXPERIMENTS FOR QWEN3-8B

• Retaining 80% or 60% of low-entropy tokens keeps accuracy stable or improves it, while retaining 40% or fewer causes a sharp accuracy drop.The authors attribute the degradation under aggressive filtering to loss of essential reasoning steps.Source: A ENTROPY FILTERING EXPERIMENTS FOR… · p. 14
• The final-answer phase length remains relatively unchanged across filtering levels, while the authors locate redundancy in the thinking phase.Source: A ENTROPY FILTERING EXPERIMENTS FOR… · p. 14
• The authors report that removed high-entropy tokens mainly drive excessive exploration rather than correct reasoning, and that their removal reduces reasoning noise.Source: A ENTROPY FILTERING EXPERIMENTS FOR… · p. 14

B EXPERIMENT DETAILS FOR BASELINE METHODS

• GRPO uses the original rule-based reward, assigning 1 for correct answers and 0 otherwise, with 8 rollouts and a 1 × 10^-3 KL penalty (Shao et al., 2024).The implementation uses the open-source verl framework.Source: B EXPERIMENT DETAILS FOR BASELINE… · p. 14
• Step Entropy uses two-stage training: SFT with pruned chain-of-thought data, followed by reinforcement learning with GRPO.The SFT stage uses FP16, a 2 × 10^-5 learning rate, and 0.01 weight decay; the RL stage uses a 1 × 10^-5 learning rate and 0.1 KL penalty.Source: B EXPERIMENT DETAILS FOR BASELINE… · pp. 14–15
• LCPO follows the L1-Exact setup, applying GRPO with length control and a maximum-length constraint (Aggarwal & Welleck, 2025).Training uses a 1×10^-6 learning rate, batch size 64, 4K-token context, 8 rollouts, temperature 0.6, and a 1 × 10^-3 KL penalty.Source: B EXPERIMENT DETAILS FOR BASELINE… · p. 15

C EVALUATION BENCHMARKS

• GSM8K comprises 1,319 gradeschool mathematics word problems, typically requiring two to eight sequential reasoning steps.The problems primarily involve basic arithmetic across multiple intermediate stages.Source: C EVALUATION BENCHMARKS · p. 15
• MATH500 is a curated subset of 500 high-school competition problems, using the evaluation setup adopted by OpenAI.The subset is drawn from the broader MATH dataset.Source: C EVALUATION BENCHMARKS · p. 15
• AIME24 contains 30 problems from the 2024 American Invitational Mathematics Examination, while AMC23 contains 40 problems from the 2023 American Mathematics Competition.These competition benchmarks cover topics including algebra, combinatorics, geometry, number theory, and probability.Source: C EVALUATION BENCHMARKS · p. 15