PaperlayerView exact revision on arXiv

Source-linked AI summary

ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning

Xin Jiang, Minhao Wang, Wen Wu, Zhentao Xie, Shangheng Du, Jinxin Shi, Jiabao Zhao

arXiv:2608.28771v1cs.LG

TL;DR

RLVR rewards correctness but provides limited guidance about reasoning quality and offers weak preferences among equally correct responses. ERR+ sequentially rewards entropy resolution before applying a difficulty-aware relative-length objective, and experiments report consistent accuracy gains with shorter responses across benchmarks and backbones. The paper also identifies a limitation: the objectives do not directly cover human-centered qualities such as explainability or pedagogical clarity.

  • Problem

    RLVR provides limited guidance on reasoning structure, while equal correctness rewards offer little signal to prefer concise, well-structured responses.

  • Method

    ERR+ first rewards length-normalized cumulative entropy drops without suppressing high-entropy exploration, then scores response length against co-generated peers using a difficulty-aware relative signal.

  • Results

    Experiments on five benchmarks show consistent accuracy improvements and meaningful response-length reductions over binary-reward and entropy-based baselines across model backbones.

  • Takeaways & Limitations

    Entropy resolution and relative peer-based length scoring can be combined sequentially to improve reasoning accuracy and conciseness together.

  • Takeaways & Limitations

    ERR+ targets accuracy and response length but does not directly optimize explainability, pedagogical clarity, or stylistic consistency valued by human evaluators.

Abstract

from arXiv · show

Large reasoning models achieve strong performance on complex tasks by generating extended chain-of-thought (CoT) traces via reinforcement learning with verifiable rewards (RLVR). While current RLVR methods have achieved strong results with correctness-based reward signals, they provide limited guidance on the quality of the reasoning process itself, leaving the internal reasoning structure largely unoptimized. Through empirical analysis across multiple model families, we identify a consistent pattern: correct reasoning trac es exhibit more frequent and larger token-level entropy drops within the thinking phase than incorrect ones. We propose ERR+, a two-phase RLVR framework grounded in this observation. The first phase trains with the Entropy Relief Reward (ERR), a bonus proportional to cumulative token-level entropy drops in the thinking phase, log-normalized by response length. Unlike prior methods that suppress entropy, ERR rewards the resolution of uncertainty while leaving exploratory high-entropy states unconstrained. The second phase introduces the Robust Relative Efficiency Reward, which scores each response's length against co-generated peers via a $\tanh$-transformed within-group $z$-score. We provide a formal analysis showing that joint optimization of the two objectives induces gradient conflict in early training, motivating the sequential design . Experiments on five datasets demonstrate consistent improvements in both accuracy and response conciseness across model backbones. Our code is available at https://github.com/XrkArul/err_response

1 Introduction

Existing RLVR methods provide limited guidance on reasoning structure and offer little preference among equally correct responses. ERR+ addresses this gap by rewarding entropy resolution and then optimizing difficulty-aware conciseness through sequential training.

  • Motivation: Within-group binary rewards provide limited signal to prefer concise, well-structured reasoning among equally correct responses.When correct responses receive identical rewards, within-correct-group variance collapses.
  • Motivation: Correct reasoning traces exhibit more frequent and deeper token-level entropy drops than incorrect traces across model families and reasoning tasks.This observation motivates using entropy changes as a reasoning-quality signal.
  • Method: ERR+ combines an entropy-resolution reward that preserves high-entropy exploration with a difficulty-aware, outlier-robust length reward.ERR uses cumulative entropy drops with length normalization, while RRER scores length relative to co-generated peers.
  • Method: Formal analysis shows that the two reward objectives produce conflicting policy-gradient directions early in training, motivating sequential rather than joint optimization.The conflict resolves after first-phase convergence.
  • Results: Evaluations across five benchmarks and four model backbones show consistent accuracy improvements and response-length reductions over baselines.The reported contribution emphasizes simultaneous gains in both metrics.

2 Related Work

Prior RLVR and entropy-control methods often apply uniform length pressure or suppress exploratory entropy, making simultaneous gains in accuracy and conciseness difficult. ERR+ instead rewards entropy changes and normalizes length relative to co-generated peers.

  • RLVR context: The broader RLVR literature includes GRPO, DAPO, VAPO, Dr. GRPO, GDPO, LASER, and ConMax, which address optimization, normalization, or length-control issues through different mechanisms.These methods form the comparison context for ERR+.
  • Length control: Existing length signals apply equal pressure regardless of problem difficulty, whereas ERR+ uses within-group z-score normalization as a difficulty proxy.The co-generated group mean supplies the relative reference for response length.
  • Entropy control: Entropy-based methods identify high-entropy tokens as important reasoning junctions, but entropy suppression can degrade accuracy by constraining exploration.Prior work reports that the highest-entropy tokens account for much of RLVR's gains, while PEAR and related approaches can suppress them.
  • Research gap: A common limitation of prior methods is difficulty improving accuracy and reducing response length simultaneously.Optimizing one objective often comes at the expense of the other.
  • ERR+: ERR rewards uncertainty resolution at each step without constraining exploration, enabling joint improvement in accuracy and conciseness.The method operates on entropy changes rather than absolute entropy levels.

3 Preliminary Analysis

The preliminary analysis measures token-level entropy during the thinking phase and compares its dynamics in correct and incorrect responses. Correct traces show sharper and more sustained entropy declines, establishing entropy drops as a robust reasoning-quality signal.

  • Measurement: Token-level entropy is computed from per-token logits, and a negative entropy change indicates a local drop during the thinking phase.The analysis partitions responses into correct and incorrect groups for comparison.
  • Comparison: Correct and incorrect traces are compared using running-mean entropy trajectories to examine how uncertainty evolves during reasoning.The figures provide both instance-level and aggregated views.
  • Single-case result: In a single AIME24 problem, the correct trace declines sharply after exploration while the incorrect trace remains persistently high.The decline is described as signaling decisive commitment, whereas the high trajectory reflects unresolved uncertainty.
  • Dataset-level result: At the dataset level, pooled correct traces consistently show sharper entropy declines than incorrect traces across 16 bootstrap samples.The aggregation is designed to reflect population-level trends rather than problem-specific selection artifacts.
  • Implication: These observations establish entropy drops as a reliable reasoning-quality signal and directly motivate the reward design.The finding supports rewarding entropy changes rather than simply penalizing entropy presence.

4 Method

ERR+ uses entropy resolution to shape reasoning quality, then applies difficulty-aware relative length control. The two objectives are trained sequentially because their gradients conflict early, while correctness gating and bounded normalization limit undesirable incentives.

  • Entropy Relief Reward: ERR accumulates thinking-phase entropy drops as a correctness-gated reward for improving reasoning quality.It does not treat entropy drops as standalone semantic verification.
  • Entropy Relief Reward: Raw entropy-drop sums could reward longer traces, so ERR uses log(Tk + 1) normalization to impose diminishing length pressure.The logarithmic form avoids unfairly penalizing difficult problems requiring extended reasoning.
  • Relative Efficiency Reward: RRER scores each response length relative to co-generated peers, using group statistics as an implicit difficulty proxy.This avoids uniform absolute length pressure across prompts of different difficulty.
  • Relative Efficiency Reward: A tanh mapping bounds the length signal and improves robustness to outlier responses, while asymmetric allocation prevents incorrect short responses from receiving length bonuses.Equal-length groups deactivate RRER.
  • Sequential Training: Theorem 1 motivates sequential training because early length pressure can remove exploratory high-entropy context needed for later entropy relief.The predicted conflict weakens after phase 1, when pruned spans are low-entropy and contribute little to ERR.
  • Training Procedure: Algorithm 1 computes entropy-based rewards independently in phase 1 and group-relative length rewards after full-group statistics are computed in phase 2.Both phases use standard GRPO advantage normalization, with checkpoints selected by validation accuracy subject to length-outlier monitoring.

5 Experiments

Experiments evaluate ERR+ across four reasoning-model backbones, five benchmarks, and multiple RLVR and length-control baselines. The reported results show simultaneous accuracy gains and shorter responses, with ablations supporting the necessity of both sequential phases.

  • Setup: Experiments use four model backbones trained on 7,473 GSM8K problems and evaluate them with Pass@1 accuracy and response length.The setup uses a maximum response length of 16,384 tokens.
  • Setup: Evaluation covers GSM8K, MATH-500, AIME24, AMC23, and MMLU-STEM.The benchmarks span arithmetic, competition mathematics, and out-of-domain STEM tasks.
  • Main Results: ERR+ achieves the highest accuracy while reducing response length on the 1.5B backbone, unlike baselines that trade accuracy against concision.The gains also transfer when DAPO is used as the base optimizer.
  • Generalization Across Backbones: ERR+ achieves the highest average accuracy on all three additional backbones while reducing response length relative to GRPO.PEAR is shorter on some backbones but degrades accuracy, especially on AIME24.
  • Ablation Study: Removing phase 1 causes 5.5% average accuracy loss, while removing phase 2 leaves accuracy at 66.8% but produces 21.5% longer traces.The phase-2-only variant reaches 1,244 tokens on GSM8K but does not preserve overall quality.
  • Ablation Study: Joint training performs worst overall, with 5.8% lower accuracy and 18.5% longer responses than full ERR+.This ablation supports the predicted early gradient conflict.

6 Ablation on Length Normalization in ERR

ERR uses logarithmic length normalization to reward entropy resolution without imposing a fixed drop density, preserving accuracy on hard problems while leaving later length compression to Phase 2. The anti-reward-hacking check finds no evidence of artificial entropy cycling.

  • Length normalization: log(Tk + 1) provides a middle ground between no normalization and linear normalization, avoiding verbosity rewards and fixed per-token density requirements.The diminishing per-token requirement preserves exploration on genuinely difficult, longer traces.
  • Length normalization: Linear normalization produces shorter outputs but incurs clear accuracy loss relative to the logarithmic form.The comparison evaluates average accuracy and token count across five benchmarks on DeepSeek-R1-Distill-Qwen-1.5B.
  • Sequential design: The final response-efficiency gain comes from Phase-2 RRER after Phase 1 establishes stable reasoning structure, not from the Phase-1 normalizer alone.This separates structural learning from subsequent response-length compression.
  • Anti-reward-hacking analysis: ERR’s entropy-drop proxy could incentivize inflate-then-drop cycles or repetitive deterministic patterns without improving reasoning quality.The paper identifies these as potential reward-hacking modes requiring intervention checks.
  • Anti-reward-hacking analysis: Up/Drop remains approximately 1.0 and sign-change counts decrease during training, while accuracy does not show the pattern expected from artificial cycling.These observations support the interpretation that rewarded drops track genuine commitment events rather than engineered entropy oscillations.

7 Conclusion

ERR+ uses sequential entropy-resolution and efficiency rewards because correct reasoning traces show more frequent and larger entropy drops than incorrect traces. Across five datasets, the framework improves accuracy while reducing response length.

  • Empirical basis: Correct reasoning traces exhibit more frequent and larger token-level entropy drops than incorrect traces.This observation motivates using entropy resolution as a process signal during the thinking phase.
  • ERR+ framework: Phase 1 rewards entropy resolution without constraining high-entropy exploration, while Phase 2 scores response length against co-generated peers with a difficulty-aware, outlier-robust signal.The two phases separate reasoning-structure formation from response-efficiency optimization.
  • Results: ERR+ improves accuracy over binary-reward and entropy-based baselines while meaningfully reducing response length on GSM8K, AIME 2024, AMC23, MATH-500, and MMLU-STEM.The reported pattern spans five datasets and evaluates the framework across multiple reasoning tasks.

Limitations

The paper reports consistent accuracy and conciseness improvements but acknowledges that its objectives do not cover several human-valued reasoning qualities. It also formally characterizes conditions under which sequential optimization avoids early gradient conflict.

  • Scope: The objectives target accuracy and response length, while explainability, pedagogical clarity, and stylistic consistency remain unoptimized.The paper proposes preference learning or multi-task reinforcement learning as future directions for balancing efficiency with human-centric qualities.
  • Optimization boundary: The formal analysis identifies early gradient conflict between entropy relief and length-efficiency objectives, motivating sequential rather than joint optimization.The conflict is associated with length pressure removing exploratory context before it becomes rewarded entropy relief.
  • Optimization boundary: The analysis predicts that gradient conflict diminishes after Phase 1 convergence, with no degradation to the Phase 1 objective at the relevant limit.This is a theoretical scope condition rather than a claim that all training dynamics are fully characterized.

B Hyperparameter Sensitivity Analysis

The sensitivity analysis varies entropy-relief and efficiency weights across five benchmarks. Values λ = 0.3 and α = 0.3 consistently provide the best accuracy–length trade-off, with a broader robust region around them.

  • Phase-1 sensitivity: Increasing λ from 0.1 to 0.3 improves accuracy, while λ ≥ 0.4 can overwhelm normalization and bias the model toward verbose justifications.Accuracy plateaus or degrades slightly beyond λ = 0.3.
  • Phase-2 sensitivity: Increasing α to 0.3 progressively improves the accuracy–length trade-off, whereas larger values saturate compression and begin degrading accuracy.The tanh transformation limits runaway penalization at extreme z-scores.
  • Cross-benchmark results: λ = 0.3 and α = 0.3 consistently yield the best accuracy–length trade-off across all five benchmarks.The analysis varies each weight from 0.1 to 0.5.
  • Cross-benchmark results: Harder competition-math benchmarks benefit more from careful tuning than GSM8K.The paper identifies AIME24 and AMC23 as more sensitive to the parameter choices.
  • Robustness region: The region λ ∈ [0.2, 0.4] and α ∈ [0.2, 0.4] maintains > 87% average accuracy and > 18% length compression.Values outside this region show either accuracy degradation or inefficient compression.

E Statistical Performance Analysis of Different Methods

Across five datasets and independent runs, ERR+ is reported to improve performance and stability relative to other methods. The analysis also presents a consistently favorable accuracy–length trade-off under default settings.

  • Statistical performance: ERR+ achieves statistically significant accuracy improvements over other methods on DeepSeek-R1-Distill-Qwen-1.5B, with all comparison t-tests having p-values below 0.05.The reported accuracy standard deviation is 0.61%, and variance is 0.3773 across independent evaluations.
  • Hyperparameter comparison: λ = 0.3 and α = 0.3 consistently provide the best accuracy–length trade-off across five benchmarks.Figure 3 reports accuracy and token count while varying the two phase-specific weights.
  • Efficiency: ERR+ is reported to improve output efficiency in addition to its accuracy advantage.The supplied passage characterizes this as a breakthrough improvement in output efficiency, without giving a separate numeric value.

F Trace Structure Analysis: Why Phase 1 Enables Phase 2

The trace analysis supports a structure-then-compress mechanism: Phase 1 creates identifiable entropy-relief commitment points, enabling Phase 2 to remove low-relief spans without reducing safe-prune accuracy. The pattern transfers across benchmarks and policy-optimization backbones.

  • Trace definitions: A large drop is defined as entropy relief rt > 0.5 nats, while a low-relief span contains no such large drop.Large drops represent commitment points, and low-relief spans are explanatory tokens between commitments.
  • Mechanism diagnostics: Phase 1 reduces the mid-exploration fraction from 0.28 to 0.08 without collapsing average entropy, which changes from 0.58 to 0.49.The gradient cosine improves from −0.45 to near zero after Phase 1 and becomes slightly positive after Phase 2.
  • Phase 1: structure: Phase 1 doubles large-drop frequency from 6.5 to 12.8 and raises safe-prune accuracy from 72.4% to 84.3%.These changes indicate that traces become organized around identifiable commitment points.
  • Phase 2: compression: Phase 2 reduces low-relief span percentage from 35.8% to 24.5% while safe-prune accuracy changes from 84.3% to 84.8%.The reported pattern supports compression of redundant material after Phase 1 has made it safer to remove.
  • Backbone transfer: ERR+ improves both accuracy and response length over DAPO alone across four benchmarks, indicating transfer beyond GRPO.Table 10 reports Pass@1 accuracy and average response length for DAPO and DAPO+ERR+.
  • Cross-domain evaluation: ERR+ also improves accuracy and response length over GRPO on GPQA Diamond and MMLU-STEM after GSM8K-only training.The evaluation is presented as a cross-domain stress test without cross-domain fine-tuning.
Loading 2608.28771v1…