Source-linked AI summary
LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models
Songtao Wei, Yi Li, Zhikai Li, Xu Hu, Yuede Ji, Guanpeng Li, Feng Chen, Carl Yang, Zhichun Guo, Bingzhe Li
TL;DR
Large reasoning models often produce unnecessarily long chains of thought, while static reward weights and global length budgets cannot accommodate changing training signals and problem-specific reasoning needs. LEAD uses online Potential-Scaled Instability weighting and per-problem targets from correct rollouts, achieving the best reported accuracy–efficiency score across five mathematical benchmarks without sacrificing reasoning performance.
Problem
Static correctness–efficiency weighting and global length constraints struggle with non-stationary reward usefulness and heterogeneous reasoning requirements across problems.
Method
LEAD separately normalizes reward channels, adapts their weights online with Potential-Scaled Instability, and calibrates a symmetric per-problem target length from correct rollouts.
Results
Across five mathematical reasoning benchmarks, LEAD achieves the best Accuracy-Efficiency Score among the evaluated RL and length-control baselines without requiring a hand-tuned weight schedule.
Takeaways & Limitations
LEAD adapts computational footprint to problem difficulty and optimization progress while improving efficiency without sacrificing reasoning performance.
Takeaways & Limitations
Decoupled normalization alone still assumes a fixed correctness–efficiency weighting throughout training, leaving the non-stationary reward-balance problem unresolved.
Abstract
from arXiv · showhide
Large reasoning models, such as OpenAI o1 and DeepSeek-R1, tend to become increasingly verbose as their reasoning capabilities improve. These inflated Chain-of-Thought (CoT) trajectories often exceed what the underlying problems require, wasting compute, latency, and context budgets. While introducing length-based efficiency rewards during reinforcement learning offers a natural remedy, existing methods struggle with two fundamental challenges: the optimal balance between correctness and efficiency is non-stationary throughout training, and intrinsic reasoning budgets vary drastically across problems. Relying on static reward weights and global length constraints inevitably forces a compromise between degraded accuracy and unrealized compression. To overcome these limitations, we propose LEAD (Length-Efficient Adaptive and Dynamic reasoning), a method that replaces static heuristics with online, self-adaptive mechanisms. LEAD dynamically calibrates the correctness-efficiency trade-off at each step using a Potential-Scaled Instability, directing optimization capacity to the most informative learning signal. Furthermore, it estimates an adaptive per-problem target length online based on the model's own correct rollouts, applying a symmetric efficiency reward that penalizes both overthinking and over-compression. Evaluated on five mathematical reasoning benchmarks, LEAD achieves the highest accuracy and Accuracy-Efficiency Score among RL-trained efficient-reasoning methods while producing substantially shorter outputs than the base model.
1 Introduction
LEAD addresses inefficiently verbose reasoning by adapting both the correctness–efficiency trade-off during training and the target length for each problem. Across five math benchmarks, it preserves or improves accuracy while substantially shortening solutions.
- Reasoning models often generate unnecessarily long chains of thought, wasting compute, latency, and context budget.
- Static efficiency methods struggle because reward usefulness changes during training and reasoning requirements differ across prompts.
- LEAD dynamically updates correctness and efficiency weights online according to which signal remains informative.
- LEAD estimates a per-problem target length from correct rollouts and symmetrically penalizes both overthinking and over-compression.
- Across five math benchmarks, LEAD matches or exceeds baseline accuracy, substantially reduces solution length, and outperforms DRPO and ShorterBetter on accuracy–efficiency score.
2 Related Work
Prior work strengthens reasoning through outcome-based reinforcement learning, while efficient-reasoning methods add length penalties, pruning objectives, or budget constraints to reduce verbosity.
- Outcome-based reinforcement learning underlies reasoning models including OpenAI o1, DeepSeek-R1, Kimi-k1.5, and Qwen-QwQ.
- GRPO samples multiple rollouts per prompt and computes group-relative advantages without a critic.
- Efficient-reasoning methods use length constraints, pruning, length penalties, or decoupled learning signals to reduce redundant reasoning.
3 Limitations of Static Length Control
Static length control fails because combined reward normalization can collapse one learning signal, while a global budget cannot match heterogeneous problem difficulty. These failures create competing risks of over-compression and wasted length feedback.
- Standard GRPO combines correctness and length rewards with fixed non-negative weights before computing a shared group-relative advantage.
- Combined-then-normalized rewards create structural pathologies when accuracy and efficiency are optimized jointly.
- Reward Collapse under Static Weighting: When all rollouts are correct, any λℓ>0 causes the length signal to determine the normalized advantage, regardless of the intended coefficient.
- Reward Collapse under Static Weighting: In all-incorrect groups, efficiency drives the entire advantage, while mixed groups let the higher-variance component dominate after normalization.
- Reward Collapse under Static Weighting: Fixed weights cannot track changing reward informativeness: length variance collapses as responses cluster, while correctness can remain informative on hard prompts.
- Global Length Budget Ignores Problem Difficulty: A global budget forces hard problems toward potentially incorrect truncation when aggressive, but yields little compression on easy problems when loose.
4 Method
LEAD combines separately normalized correctness and efficiency advantages with online instability-driven weighting, then calibrates a symmetric, per-problem length target from correct rollouts. The design adapts optimization to both training progress and prompt difficulty.
- Method: LEAD replaces scalar-combined advantages and a global budget with decoupled reward normalization and per-problem target-length calibration.
- Dynamic Reward Weighting with Decoupled Group Normalization: Each reward is normalized separately before its advantage channel is combined under weights that sum to one.
- Dynamic Reward Weighting with Decoupled Group Normalization: Decoupled normalization removes scale mismatch but still assumes fixed weights, so LEAD adds online dynamic weighting for non-stationary reward learnability.
- Dynamic Reward Weighting with Decoupled Group Normalization: Potential-Scaled Instability weights rewards using both relative variability and remaining headroom, favoring signals that are informative and improvable.
- Per-problem Online Target-Length Calibration: The symmetric efficiency reward peaks at the target and penalizes deviations on both sides, including under-length solutions.
- Dynamic Reward Weighting with Decoupled Group Normalization: The controller smooths per-batch instability values with an EMA and enforces a lower bound on the correctness weight.
- Per-problem Online Target-Length Calibration: LEAD filters correct rollouts to estimate each prompt’s target length, clamping it between Lmin and Bmax; unsolved prompts use Bmax.
- Per-problem Online Target-Length Calibration: Recomputing targets from current correct rollouts lets them decrease as the model discovers shorter valid solutions.
5 Experiment
LEAD is evaluated across five mathematical benchmarks, two model sizes, and several controlled ablations. It achieves the strongest reported accuracy–efficiency trade-offs while showing that dynamic reward weighting and correct-rollout target calibration are important design choices.
- Experimental setup: The evaluation uses five math benchmarks, two DeepSeek-R1-Distill-Qwen model sizes, and a held-out MATH-500 test set.Training uses 8,521 MATH Level 3–5 problems; evaluation covers AIME 2024/2025, AMC 2023, MATH-500, and OlympiadBench.
- Math reasoning results: LEAD achieves the highest average accuracy and AES among RL-trained methods across both 1.5B and 7B models.At 1.5B, it reaches 53.36 accuracy and 0.68 AES; at 7B, it reaches 65.17 accuracy and −0.11 AES.
- Math reasoning results: 53.36 accuracy and 0.68 AES make LEAD the only 1.5B method that improves over the base model while reducing average length.Against DRPO, LEAD improves accuracy by 2.62 points and AES by 0.18.
- Static vs. dynamic reward weighting: Dynamic LEAD reaches 0.68 AES and 53.36 accuracy, outperforming the best static setting without manual ratio selection.The efficiency weight starts near 0.5 and decays to approximately 0.07 as the efficiency signal saturates.
- Per-problem target aggregation: The mean of correct rollouts gives the best target-length aggregator, reaching 53.36 accuracy and 0.68 AES.The minimum aggregator compresses most aggressively but lowers accuracy by over 3 points; mean-of-all trails LEAD by 1.9 accuracy points.
6 Conclusion
LEAD combines adaptive reward weighting with per-problem target-length calibration to address changing optimization priorities and heterogeneous reasoning demands. Across five mathematical reasoning benchmarks, it achieves the best reported accuracy–efficiency trade-off while retaining strong accuracy and reducing response length.
- LEAD adapts correctness–efficiency weights using per-reward instability and remaining headroom, rather than a fixed training schedule.This controller is paired with online per-problem target-length estimation and symmetric efficiency rewards.
- Across five mathematical reasoning benchmarks, LEAD achieves the best Accuracy-Efficiency Score among the evaluated baselines and length-control variants.The reported comparison includes GRPO, GDPO, DRPO, ShorterBetter, and their length-control variants.
- Figure 2 tracks on-policy batch accuracy, rollout-batch mean response length, per-problem target L∗q, and symmetric efficiency reward across four aggregator variants.These trajectories provide the reported training-dynamics view for DeepSeek-R1-Distill-Qwen-1.5B.
A.1 Limitations
The appendix describes LEAD’s scope, deployment boundary, and implementation context. The method assumes reliably evaluable correctness and multiple rollouts, and it learns efficiency during training rather than enforcing a hard inference-time budget.
- LEAD is designed for settings with reliable correctness evaluation and multiple sampled rollouts per prompt.The paper identifies mathematical reasoning as a suitable setting because answer verification is relatively precise.
- Extending LEAD to open-ended generation, instruction following, or subjective preference tasks may require task-specific reward models or validators.The dynamic weighting and per-problem calibration components are described as agnostic to the particular reward source.
- Target-based compression may emerge later on very hard prompts because LEAD waits for at least one correct rollout before applying target pressure.This prioritizes correctness before compression and avoids suppressing exploration on unsolved problems.
- A single scalar target length per prompt may not represent tasks with multiple valid solution styles of substantially different lengths.The paper identifies distributional or multi-target length models as a future direction.
- LEAD learns efficient reasoning through training-time policy optimization rather than enforcing a hard inference-time token budget.It is therefore not a replacement for deployment-time budget controllers when strict latency or cost limits are required.
- LEAD preserves the GRPO objective and changes only the construction of the group-relative advantage used in training.Algorithm 1 combines decoupled normalization, PSI-driven weighting, and online target-length calibration in that advantage computation.
- Accuracy-Efficiency Score measures accuracy preservation and length reduction relative to the pre-RL base model.Higher AES indicates a better accuracy–efficiency trade-off, with α=1, β=3, and γ=10 in the reported formulation.
E Additional Experiment
The appendix states that it contains additional empirical results and complementary training-dynamics material.
- The appendix includes additional empirical results, training dynamics, and complementary results.
E.1 Full Hyperparameter Specification
The appendix records the training settings used in Section 5.1, including scale-specific differences.
- Table 4 lists every training setting referenced in Section 5.1, separating differing entries into 1.5B and 7B columns.Most training values are shared between model scales.
E.2 Training dynamics on DeepSeek-R1-Distill-Qwen-1.5B
LEAD shifts optimization toward correctness early, then maintains rising accuracy while shortening responses and tightening per-problem length targets. Its token allocation remains difficulty-sensitive, spending comparatively more extra tokens on harder prompts than compression baselines.
- Training dynamics: LEAD smoothly reweights correctness and efficiency without manual scheduling as training progresses.The run tracks dynamic weights, target-length statistics, response length, and validation accuracy.
- Training dynamics: By step 50, the efficiency weight falls to approximately 0.08, while correctness plateaus near 0.93 for the remainder of training.Weights initialize at (λc, λℓ)=(0.5, 0.5) and shift rapidly toward correctness during early training.
- Training dynamics: Most length compression occurs during the early learn-to-solve phase, while the per-problem target length tightens most rapidly.Later, the efficiency signal saturates and optimization capacity is held on correctness, whose signal remains informative.
- Token allocation by prompt difficulty: LEAD preserves difficulty-sensitive allocation, with Spearman ρ= + 0.67 versus + 0.71 for the base model and + 0.53 for GDPO.Higher ρ indicates greater difficulty sensitivity; GDPO compresses most uniformly.
- Token allocation by prompt difficulty: +1,540 to +3,540 tokens on Q1 versus baselines falls to +850 to +1,510 on Q4, concentrating LEAD’s extra budget on hardest prompts.Against GDPO, ShorterBetter, and GRPO, the hardest-tier gap is 3.4×, 2.3×, and 2.2× the easiest-tier gap, respectively.
E.4 Results at 8K budget on DeepSeek-R1-Distill-Qwen-1.5B
At an 8K training-time response limit, LEAD retains its advantage across accuracy and efficiency, achieving the strongest reported accuracy and AES while avoiding the over-compression seen in ShorterBetter.
- 8K-budget results: 54.44 accuracy, +2.59 over the base, and 0.54 AES make LEAD best on both metrics at the 8K budget.GDPO trails LEAD by 3.17 accuracy points despite closing the AES gap to 0.01.
- 8K-budget results: The same conclusions as the 4K results hold when the training-time maximum response length increases from 4K to 8K.All methods use the same evaluation protocol and unchanged hyperparameters apart from the maximum response length.
- 8K-budget results: ShorterBetter incurs a 7.3-point accuracy regression and the worst AES among baselines at the 8K budget.Forcing correct rollouts toward a single minimum length still over-compresses hard problems.