PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn to Reason Efficiently with Adaptive Length-based Reward Shaping

Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, Junxian He

arXiv:2505.15612v1cs.CLcs.AIcs.LG

TL;DR

LRMs often generate redundant, lengthy reasoning traces, creating a need for more efficient chain-of-thought generation without sacrificing reasoning quality. The paper unifies RL-based length reward shaping, introduces LASER and its dynamic, difficulty-aware extensions LASER-D and LASER-DE, and reports improved accuracy-efficiency trade-offs, including +6.1 on AIME2024 with 63% lower token usage.

  • Problem

    LRMs can overthink by producing unnecessarily long and redundant reasoning trajectories, increasing token usage and potentially compounding errors.

  • Method

    The paper unifies RL-based length reward shaping and proposes LASER, LASER-D, and LASER-DE with target lengths that can adapt to training dynamics and query difficulty.

  • Results

    +6.1 percentage points on AIME2024 with 63% lower token usage is reported for LASER-D/LASER-DE on DeepSeek-R1-Distill-Qwen-1.5B.

  • Takeaways & Limitations

    The LASER series provides a Pareto-optimal balance between reasoning performance and token efficiency while producing more concise reasoning with fewer redundant self-reflections.

  • Takeaways & Limitations

    Fixed-length truncation can disproportionately reduce performance on harder tasks, motivating difficulty-aware target lengths.

Abstract

from arXiv · show

Large Reasoning Models (LRMs) have shown remarkable capabilities in solving complex problems through reinforcement learning (RL), particularly by generating long reasoning traces. However, these extended outputs often exhibit substantial redundancy, which limits the efficiency of LRMs. In this paper, we investigate RL-based approaches to promote reasoning efficiency. Specifically, we first present a unified framework that formulates various efficient reasoning methods through the lens of length-based reward shaping. Building on this perspective, we propose a novel Length-bAsed StEp Reward shaping method (LASER), which employs a step function as the reward, controlled by a target length. LASER surpasses previous methods, achieving a superior Pareto-optimal balance between performance and efficiency. Next, we further extend LASER based on two key intuitions: (1) The reasoning behavior of the model evolves during training, necessitating reward specifications that are also adaptive and dynamic; (2) Rather than uniformly encouraging shorter or longer chains of thought (CoT), we posit that length-based reward shaping should be difficulty-aware i.e., it should penalize lengthy CoTs more for easy queries. This approach is expected to facilitate a combination of fast and slow thinking, leading to a better overall tradeoff. The resulting method is termed LASER-D (Dynamic and Difficulty-aware). Experiments on DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and DeepSeek-R1-Distill-Qwen-32B show that our approach significantly enhances both reasoning performance and response length efficiency. For instance, LASER-D and its variant achieve a +6.1 improvement on AIME2024 while reducing token usage by 63%. Further analysis reveals our RL-based compression produces more concise reasoning patterns with less redundant "self-reflections". Resources are at https://github.com/hkust-nlp/Laser.

1 Introduction

The paper targets over-thinking in LRMs by using RL to compress redundant chains of thought while preserving or improving reasoning accuracy. It unifies length-based reward shaping and introduces LASER, LASER-D, and LASER-DE to improve the accuracy-efficiency trade-off.

  • Motivation: Long reasoning trajectories increase token usage and may include compounding errors and unnecessary verbosity, even for elementary questions.This over-thinking issue motivates RL-based chain-of-thought compression.
  • Results: The LASER series achieves a Pareto-optimal balance between accuracy and response length, with the main figure showing AIME2024 runs for DeepSeek-R1-Distill-Qwen-1.5B.Each plotted point represents a fully trained run with a distinct hyperparameter configuration.
  • Motivation: RL-based efficiency methods typically trade reduced token usage for lower reasoning accuracy, although recent RL approaches offer a more favorable balance.The paper positions its methods against this recurring efficiency-accuracy trade-off.
  • Methods: LASER unifies length-based reward shaping methods and uses a target-length-controlled step reward to improve the performance-efficiency trade-off.It extends a truncation-based RL baseline and is reported as the best trade-off among evaluated baselines.
  • Methods: LASER-D dynamically adjusts target lengths during training and assigns different limits according to query difficulty, while LASER-DE additionally encourages exploration on incorrect responses.Harder questions receive higher token limits, whereas easier questions are constrained more strongly.

2 Preliminary

The preliminary section frames reasoning enhancement and efficiency improvement as RL objectives under a KL-constrained optimization formulation. It uses rule-based trajectory rewards and length-based reward shaping to encourage more efficient responses.

  • RL for Efficient Reasoning: RL can improve reasoning capabilities across mathematical, coding, and agentic tasks, often accompanied by emergent self-reflection-style thinking.The paper follows prior work using rule-based outcome rewards for mathematical reasoning.
  • RL Objective: The KL-constrained objective optimizes expected trajectory reward while penalizing divergence between the trained policy and a reference model.The response is y, the question is x, and β controls the two optimization targets.
  • RL Objective: GRPO is used to optimize the KL-constrained reinforcement-learning objective.The reference policy is the model before RL training.
  • RL for Efficient Reasoning: Length-based reward shaping incentivizes shorter responses by assigning higher rewards to more concise outputs.This is the common mechanism used by several efficient-reasoning approaches.

3 Truncation: A Simple Yet Effective Baseline

The truncation baseline trains LRMs under a smaller generation limit, treating responses beyond that limit as incorrect to encourage concise accurate reasoning. It substantially improves token efficiency, but its benefits vary with task difficulty and can disproportionately hurt harder tasks.

  • Baseline: An 8,192-token training limit replaces the models’ original 32,768-token context window and truncates responses beyond the allowed length.The baseline therefore pushes models toward accurate responses under strict token constraints.
  • Effectiveness: 71% improvement in token efficiency is achieved with the 8,192-token limit, while average accuracy decreases by 1.6 absolute points.This result is reported relative to the original model across the evaluated benchmarks.
  • Effectiveness: The truncation baseline improves reasoning efficiency while maintaining competitive accuracy on average.The paper characterizes it as a simple yet effective approach.
  • Efficacy-Efficiency Trade-off: AIME accuracy drops 4.1 points with an 8,192-token limit and 9.7 points with a 4,096-token limit, compared with a 7% decrease on MATH500.The larger degradation on AIME is linked to its longer reasoning trajectories: over 75% exceed 8,192 tokens versus 15% for MATH500.

4 A Unified View on Efficient Reasoning with RL

The paper unifies efficient-reasoning approaches as length-based reward shaping, then introduces LASER to replace hard truncation with target-length step rewards. LASER improves the accuracy–token-efficiency Pareto trade-off by preserving correctness incentives while rewarding concise correct responses.

  • Truncation Baseline: Truncation is a special case with C(y) = 0, enforcing target length LT through the context window.ThinkPrune extends this family with an adaptive target length LA and three training stages.
  • Unified Formulation: Length-based reward shaping unifies efficient-reasoning methods through designs of correctness C(y), control variable λ(y), and length reward S(y).The framework connects truncation and other approaches using a shared reward formulation.
  • Prior Reward Designs: Group-based rewards encourage shorter responses within rollout groups but can induce reward hacking and overly concise answers, especially for simple questions.Budget-based rewards mitigate this issue but can destabilize training when target lengths are sparsely distributed.
  • LASER: LASER uses a target-length step reward instead of hard truncation, assigning a length bonus to correct responses shorter than LT.Its context window is larger than LT, so long responses are rarely truncated while correctness remains part of the reward.
  • LASER: LASER achieves improved Pareto-optimality over previous methods and simultaneously improves accuracy and token efficiency on AIME24.The reported empirical evidence positions LASER as a reward-design framework for balancing reasoning efficacy and efficiency.

5 Adaptive Length-based Step Reward Shaping

LASER-D addresses LASER’s fixed-length and query-insensitive limitations by dynamically adapting target lengths and assigning different budgets to easy, medium, and hard questions. Its monitoring-based mechanism selects difficulty-specific targets while adding minimal computational overhead.

  • 5.1 Design Principles: LASER’s fixed target length may become suboptimal as model reasoning changes during training, while questions of different difficulty require different reasoning lengths.These limitations motivate dynamic and difficulty-aware reward shaping.
  • 5.1 Design Principles: LASER-D dynamically adjusts target lengths during training and decouples them across queries of different difficulty.It assigns smaller targets to easier questions and larger targets to harder questions.
  • 5.2 Automatic Adapting Mechanism: Queries are classified as easy, medium, or hard from correctness rates in rollout batches using k/3 and 2k/3 thresholds.The classification uses realtime RL-training rollouts and incurs negligible computational overhead.
  • 5.2 Automatic Adapting Mechanism: Every N training steps, LASER-D evaluates a small monitoring dataset and searches target lengths for each difficulty level.The monitoring set mirrors the training-data distribution; an example uses 500 samples and updates every 20 steps.
  • 5.2 Automatic Adapting Mechanism: Expected Correct Responses estimates complete correct responses under length limits by combining coverage ratios with difficulty-specific minimum correct-response counts.The metric samples K responses per monitoring query and uses fixed |C_d| values based on minimum correct responses.
  • 5.2 Automatic Adapting Mechanism: The adaptive target is the smallest length with ECR_d ≥ 1, intended to avoid lengths where correct responses are unlikely without retaining redundant extra tokens.The search spans from LT to the 16,384-token context limit in intervals of I.
  • 5.2 Automatic Adapting Mechanism: The automatic adapting mechanism increases computation by just 3.5% in the reported experiments.This overhead comes from periodic evaluation on a small monitoring dataset.
  • 5.3 LASER-DE: LASER-DE extends LASER-D by encouraging policy-model exploration on incorrect responses using additional test-time-compute observations.The variant is motivated by findings that incorrect responses tend to use more tokens and that additional compute can improve reasoning.

6 Experiments

Experiments across three model sizes and four reasoning benchmarks compare LASER variants with prior length-control methods. LASER-D and LASER-DE generally provide the strongest accuracy–token-efficiency trade-offs, including on out-of-domain evaluations.

  • Experimental setup: Experiments use DeepSeek-R1-Distill-Qwen models with 1.5B, 7B, and 32B parameters across MATH500, OlympiadBench, AIME2024, and AMC2023.Training uses the DeepScaleR-Preview-Dataset containing 40K competition-level question-answer pairs.
  • Evaluation protocol: Pareto-frontier evaluation varies key reward hyperparameters to compare accuracy and response length across training runs.The 1.5B backbone enables multiple experiments exploring different trade-offs.
  • 1.5B results: LASER-DE reaches 35% AIME2024 accuracy using just over 5,500 tokens, while LASER-D reaches 34% with 4,600+ tokens.Across all benchmarks, LASER-DE is strongest below 3,500 average tokens, whereas LASER-D performs best at higher token usage.
  • Larger models: LASER-D achieves 58.3% AIME accuracy with 5,379 tokens on the 7B model, a +5.2-point gain over the original model’s result.The original model uses 13,414 tokens on this dataset.
  • Larger models: LASER-DE preserves competitive 32B accuracy with only a minor 1% drop while significantly reducing output length.The baseline already exceeds 76% accuracy on the training dataset, limiting room for improvement.
  • Out-of-domain evaluation: On GPQA, LSAT, and MMLU, LASER variants improve both accuracy and token usage over the original model, with LASER-D and LASER-DE maintaining the best trade-offs.These benchmarks evaluate generalization beyond the RL training distribution.

7 Analysis

The analysis examines how RL-based compression changes reasoning length and behavior on AIME2024. Shorter responses coincide with fewer self-reflection signals and substantially less backtracking, while several productive behaviors remain stable.

  • Analysis design: Analysis on AIME2024 uses 16 samples per question and tracks reasoning changes through keyword counts and reasoning-behavior ratios.The study uses DeepSeek-R1-Distill-Qwen-1.5B for this analysis.
  • Self-reflection: Self-reflection keywords decline as response length decreases across LASER methods during RL training.The tracked keywords include “recheck,” “rethink,” “try again,” “wait,” “alternatively,” “retry,” and “however.”
  • Reasoning behaviors: Backtracking falls from over 30% to just above 10% as response length is reduced.Verification, Enumeration, and Subgoal Setting remain stable, with a slight increase in Subgoal Setting.
  • Qualitative analysis: Qualitative examples show the LASER-D-trained model answering “1+1=?” directly instead of repeating self-reflection, while MATH500 analysis examines similar efficiency changes.The comparison is between the original and LASER-D-trained DeepSeek-R1-Distill-Qwen-1.5B models.

8 Conclusion

The paper presents a unified view of RL-based chain-of-thought compression and reports that adaptive LASER methods improve the accuracy–efficiency frontier. Behavioral analysis links compression with more concise and productive reasoning.

  • Conclusion: The paper unifies reward-shaping and truncation methods under a framework for RL-based chain-of-thought compression.This framework motivates adaptive, length-based reward shaping.
  • Conclusion: Figure 3 tracks average self-reflection keyword counts and response length over AIME24 RL training.The truncation method uses an 8192-token context window, while LASER variants use LT = 2048.
  • Conclusion: Figure 4 tracks reasoning-behavior ratios and response length over LASER-DE training iterations on AIME2024.The plotted LASER-DE runs use LT = 2048.
  • Conclusion: RL-based compression encourages models to reason more concisely and productively.The analysis emphasizes changes in reasoning behaviors rather than response length alone.
  • Conclusion: LASER, LASER-D, and LASER-DE achieve significant improvements in accuracy and token usage across benchmarks, with LASER-D and LASER-DE Pareto-optimal against other methods.Figure 5 compares accuracy and response length across all benchmarks and on AIME2024.

B Ratio of Truncated Responses During Training with Truncation

The analysis measures how often training responses exceed an 8192-token limit. Truncation remains frequent after extended training, indicating that this context-window constraint is sub-optimal.

  • Truncation during training: More than 45% of responses are initially truncated under the 8192-token training limit.The proportion remains above 10% even after 200 rollout steps.
  • Truncation during training: The truncation rate stays above 10% after 200 rollout steps, highlighting persistent context-window constraints.The authors characterize this training setting as sub-optimal.

C Dynamics of Accuracy and Rewards Throughout Training

Across training, truncation-based and LASER-family methods improve rewards and accuracy together, while group-based rewards and some budget-based methods show important trade-offs or instability. LASER-D adapts target lengths by problem difficulty, and the methods generalize beyond the training distribution.

  • Truncation methods, LASER, LASER-D, and LASER-DE show simultaneous increases in rewards and accuracy throughout training.
  • Group-based rewards exploit length rewards but cause a significant accuracy drop, while L1-Max-16384 can train unstably with a large context window.
  • LASER-D and LASER-DE maintain strong accuracy-efficiency trade-offs under moderately constrained inference budgets, especially on harder questions.
  • L1-Max performs best under extremely tight budgets but plateaus with more available budget, whereas group-based methods can produce overly brief responses.
  • Adaptive target lengths quickly become shorter for easy problems, decrease from 10,000+ to 3000-4000 for medium problems, and remain near the maximum for difficult problems.
  • LASER, LASER-D, and LASER-DE consistently improve accuracy and efficiency on GPQA, LSAT, and MMLU, including the knowledge-intensive MMLU benchmark.

K Analysis of Reasoning Behaviros

The paper analyzes how reasoning behaviors change during long-to-short RL using a cognitive behavior framework and gpt-4.1-mini. The analysis samples AIME2024 reasoning traces repeatedly to obtain fine-grained behavioral information.

  • The authors apply Gandhi et al.’s cognitive behavior framework to analyze reasoning changes during long-to-short RL.
  • gpt-4.1-mini performs fine-grained identification and analysis of reasoning behaviors throughout training.
  • The AIME2024 analysis samples one question 16 times to examine reasoning behavior during training.

L Qualitative Analysis on Efficient Reasoning

Qualitative analyses compare original and LASER-D-trained reasoning on trivial questions and MATH500 examples. The trained model uses more concise, structured reasoning with less repetitive self-reflection and backtracking, while broader validation remains focused mainly on mathematics.

  • On “1+1=?”, the original model produces repetitive self-reflection, whereas the LASER-D-trained model directly provides the answer.
  • On MATH500, LASER-D replaces verbose explanations of single ideas with more succinct expressions using structured formulas.
  • The qualitative analysis suggests RL-based compression reduces unproductive backtracking and encourages more concise, direct expression.
  • The paper focuses primarily on mathematics and identifies code generation and agentic tasks as areas needing further validation.
  • A case study reports over 17K tokens for the original model versus only 1K+ tokens for the trained model on the same MATH500 task.
Loading 2505.15612v1…