PaperlayerView exact revision on arXiv

Source-linked AI summary

ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning

Jingyang Yi, Jiazheng Wang, Sida Li

arXiv:2504.21370v4cs.AI

TL;DR

Extended CoT reasoning can become inefficient and redundant, motivating methods that identify a task-appropriate reasoning length. ShorterBetter uses Sample Optimal Length as a reinforcement-learning reward, reducing output length by 50%-80% across in-domain and out-of-domain tasks while maintaining accuracy. Its trace analysis links this efficiency to less repetition, self-verification, and alternative-path exploration.

  • Problem

    Excessively long reasoning can cause computational inefficiency, diminishing returns, and even reduced model performance through overthinking.

  • Method

    ShorterBetter uses Sample Optimal Length, the shortest correct response among multiple generations, as a dynamic reinforcement-learning reward for learning efficient CoT lengths.

  • Results

    50%-80% reductions in output token length were achieved on in-domain mathematical reasoning and diverse out-of-domain benchmarks while maintaining accuracy.

  • Takeaways & Limitations

    ShorterBetter reduces redundant reasoning and refines traces toward more confident and targeted reasoning strategies.

  • Takeaways & Limitations

    The strongest empirical results are currently reported on tasks with verifiable answers, while evaluation on open-ended settings and larger-scale models remains a future direction.

Abstract

from arXiv · show

Recent models such as OpenAI o1 and DeepSeek-R1 have demonstrated strong performance on reasoning-intensive tasks by generating extended Chain-of-Thought (CoT) traces. While longer reasoning helps with thorough exploration of solution paths for complex problems, it also often leads to inefficient and redundant outputs--a phenomenon commonly described as overthinking. In this paper, we propose ShorterBetter, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the Sample Optimal Length (SOL) as the length of the shortest correct response among multiple generations, which serves as a dynamic reward signal to guide the model toward efficient reasoning. Applied to DeepSeek-Distill-Qwen-1.5B/7B as base models, ShorterBetter achieves 50%-80% reduction in output lengths in both in-domain and out-of-domain reasoning tasks while maintaining accuracy. Our reasoning trace analysis shows that ShorterBetter refines the structure of the reasoning traces by reducing unnecessary repetition, excessive self-verification, and over-exploration of alternatives.

1 Introduction

ShorterBetter targets overlong reasoning by learning an optimal CoT length through reinforcement learning and Sample Optimal Length rewards. The method is motivated by overthinking and aims to reduce redundant reasoning while preserving accuracy.

  • Motivation: Extended reasoning can become counterproductive through computational inefficiency, diminishing returns, and impaired performance.The paper identifies repetition, premature stopping, continued alternative exploration, and degraded coherence as problematic patterns in lengthy traces.
  • Method: The proposed optimal-length hypothesis places effective reasoning between insufficiently short analysis and redundant overlong reasoning.The optimal length is described as dependent on model capacity and problem difficulty.
  • Method: ShorterBetter is a reinforcement learning method that uses Sample Optimal Length as a dynamic reward signal.It guides reasoning models to learn their optimal reasoning lengths autonomously rather than relying on manual supervision.
  • Contributions: 50%-80% reductions in output token length were achieved on in-domain mathematical and diverse out-of-domain benchmarks while maintaining accuracy.The result applies to 1.5B and 7B base models.
  • Contributions: ShorterBetter analyzes reasoning traces by reducing verbosity after correct answers and refactoring traces toward more confident, targeted strategies.The analysis framework is presented as evidence of improved reasoning efficiency.

2 Related Work

Related work frames efficient reasoning as a response to overthinking in large reasoning models. Existing reinforcement-learning approaches mainly control length through explicit budgets or incentives to follow shorter paths.

  • Reasoning in LLMs: Large reasoning models scale test-time computation through extended CoT generation, parallel sampling, and iterative refinement.Examples include OpenAI o1, DeepSeek-R1, and Qwen-QwQ.
  • The Overthinking Phenomenon: Overthinking is associated with computational inefficiency because models generate excessively lengthy and redundant reasoning.The related-work discussion also describes excessive reasoning as potentially harmful to practical performance.
  • The Overthinking Phenomenon: Excessive reasoning can degrade logical correctness, introduce irrelevant information, and reduce accuracy beyond a threshold.These effects are attributed to misjudged problem difficulty and compounding errors in the cited studies.
  • Efficient Reasoning with RL: Existing reinforcement-learning methods use explicit length budgets or reward shorter reasoning paths relative to reference models.L1 specifies a user-controlled budget, while DAST adapts the budget to problem difficulty.

3 Method

ShorterBetter estimates an optimal reasoning length from sampled responses and uses it to reward concise, correct reasoning. The method integrates this reward with GRPO so the policy can adapt length to problem difficulty and model capability without manual thresholds.

  • 3.1 Optimal Reasoning Length (OL): Formal Definition: Optimal reasoning length is the minimum length at which the model reliably produces sufficiently correct responses under probability and correctness thresholds.It becomes longer for harder problems and shorter when the model can consistently generate concise, correct responses.
  • 3.2 Sample Optimal Length (SOL): A Practical Approximation: ShorterBetter approximates optimal reasoning length with Sample Optimal Length (SOL), a sample-based measure derived from multiple candidate rollouts.The exact optimal length is impractical to compute because it requires complete knowledge of the model’s output distribution.
  • 3.2 Sample Optimal Length (SOL): A Practical Approximation: The SOL-based reward balances response correctness against a length penalty using hyperparameters α and β.The reward is defined for each response in a sampled group, with α and β balancing correctness reward and length penalty.
  • 3.2 Sample Optimal Length (SOL): A Practical Approximation: SOL adapts to problem difficulty and model capability, updating during training toward the shortest correct reasoning length achieved by the current policy.It requires no manual tuning of problem-specific length thresholds and uses sampled responses to adjust dynamically.
  • 3.2 Sample Optimal Length (SOL): A Practical Approximation: When no correct response appears in a sampled group, SOL uses the average response length as a neutral baseline.This avoids biasing training toward either overthinking or underthinking without a correct signal.
  • 3.3 Policy Optimization via GRPO: GRPO normalizes rewards within rollout groups to estimate advantages, then updates the policy to favor high-reward samples while constraining deviation from a reference policy.The objective uses a clipped policy-improvement term and a KL-divergence penalty.

4 Experimental Setup

The experiments fine-tune DeepSeek-R1-Distill-Qwen models on mixed mathematical data and evaluate efficiency and accuracy in held-out and broader benchmark settings. Comparisons include reasoning, non-reasoning, and prior efficiency-oriented baselines using accuracy, output length, and Accuracy-Efficiency scores.

  • Datasets: DeepScaleR-preview provides 40K mathematical problems spanning AIME, AMC, Omni-MATH, and Still, from high-school mathematics to Olympiad-level challenges.In-domain evaluation uses held-out subsets totaling 1K problems.
  • Base Models: The study trains ShorterBetter from DeepSeek-R1-Distill-Qwen-1.5B and 7B base reasoning models.The resulting models are referred to as ShorterBetter-1.5B and ShorterBetter-7B.
  • Evaluations: Evaluation reports Accuracy, Output Length, and Accuracy-Efficiency (AE) Score for each benchmark.Accuracy is the fraction of correct responses, Output Length is average tokens per benchmark, and AE Score balances shortening against accuracy preservation.
  • Implementation Details: Training uses Verl with rollout group size n = 8, batch size N = 64, a learning rate of 1 × 10^-6, and 300 or 200 steps for the 1.5B and 7B variants.Maximum inference lengths are constrained by computational limits.
  • Baselines: Baselines include the distilled DeepSeek models, Qwen2.5-Instruct models, Training Efficient, and O1-Pruner.The Instruct models provide a fast-thinking non-reasoning contrast, while other baselines target reasoning efficiency.

5 Results and Discussion

Across in-domain and out-of-domain evaluations, ShorterBetter substantially reduces inference length while preserving competitive accuracy. Its trace analyses indicate more efficient reasoning structure, while ablations show that SOL’s design and correctness reward are important for stable, accurate training.

  • Experimental Results: 77.6% and 79.2% average in-domain length reductions accompany accuracy changes of +2.5% and +3.2% for ShorterBetter-1.5B variants.The α = 1 and α = 2 variants show similar length-reduction and accuracy-preservation behavior.
  • Experimental Results: 77.0% and 75.5% average out-of-domain length reductions accompany accuracy changes of –0.8% and –1.6% for ShorterBetter-1.5B variants.These results are averaged across the evaluated out-of-domain benchmarks.
  • Experimental Results: 62.1% in-domain and 62.3% out-of-domain average length reductions are achieved by ShorterBetter-7B, with accuracy changes of +7.1% and -0.48%.The 7B model reaches up to 70% output-length reduction and the highest AE Scores across evaluated tasks.
  • Reasoning Structure Analysis: ShorterBetter-7B reduces post-answer tokens by 15%, 20%, 17%, and 19% on AMC, MATH, Olympiad, and Minerva, respectively.The analysis measures output remaining after the first correct answer for questions both models answer correctly.
  • Reasoning Structure Analysis: On MATH, meaningful categories increase by 15.6% and 6.8%, while Non-Substantive Statements, Verification & Self-Correction, and Exploring Alternatives decrease by 4.8%, 14.1%, and 3.9%.These shifts indicate denser solution-oriented reasoning with fewer filler tokens, unnecessary checks, and alternative-solution detours.
  • Reasoning Structure Analysis: The authors recommend more rigorous quantitative metrics for reasoning structure to validate and extend the analysis.This is stated as a limitation of the current reasoning-structure evaluation.
  • Ablation Studies: Ablations show that replacing SOL with the shortest response length causes training collapse within 100 steps, whereas removing correctness reward keeps training effective but destabilizes accuracy.The findings support SOL’s combination of correctness and conciseness, while retaining correctness reward to prevent accuracy degradation.

6 Conclusion and Future Directions

ShorterBetter dynamically optimizes CoT lengths without manual supervision and reduces output length while maintaining accuracy across in-domain and out-of-domain benchmarks. Its current empirical evidence is strongest on tasks with verifiable answers, while future work includes open-ended settings and larger reasoning models.

  • ShorterBetter dynamically optimizes CoT lengths without manual supervision using a self-supervised reward signal.
  • Strong empirical results are currently reported on tasks with verifiable answers.
  • The formulation naturally extends to open-ended settings with non-binary correctness scores.
  • Future directions include evaluating ShorterBetter on larger-scale reasoning models and developing finer-grained, behavior-aware optimization strategies.

Broader Impact

ShorterBetter targets efficient reasoning and may improve deployment practicality by reducing redundant reasoning while maintaining accuracy. The evaluation framework combines output length and accuracy through the AE Score, with accuracy degradation penalized more heavily than gains.

  • Reducing redundant reasoning steps while maintaining accuracy could lower computational costs and improve deployment in resource-constrained settings.
  • The AE Score evaluates whether inference output length decreases without sacrificing accuracy.
  • Positive AE Scores indicate shorter outputs with maintained or improved accuracy, whereas negative scores penalize accuracy degradation.
  • The asymmetric AE Score design penalizes accuracy drops more heavily than accuracy gains because avoiding degradation is prioritized.

A.3 Experimental Details

The experiments distinguish in-domain from out-of-domain tasks according to the training data used for ShorterBetter and the evaluated baselines. Out-of-domain benchmarks are held out from all models’ training data to test generalization, with model comparisons using downloaded open-source checkpoints.

  • In-domain tasks are held-out subsets of datasets included in ShorterBetter’s DeepScaleR-Preview training corpus.
  • Out-of-domain tasks include MathQA, BBH, MMLU, LiveCodeBench, MBPP, and HumanEval, which are absent from ShorterBetter’s training corpus.
  • Out-of-domain benchmarks are held out from all models’ training data and therefore serve as tests of generalization.
  • Training Efficient and O1-Pruner use different training datasets, with partial overlap between their data and ShorterBetter’s data.
  • Comparisons used downloaded open-source Training Efficient 1.5B/7B and O1-Pruner 7B checkpoints; O1-Pruner 1.5B was unavailable for evaluation.

A.4 On the Practical Choice of Hyperparameters

The hyperparameter α controls the trade-off between accuracy and efficiency relative to the fixed length-penalty β. Results favor balanced settings, while the preferred choice depends on the model family and whether efficiency or accuracy is prioritized.

  • β is fixed at 0.001 while α is varied because GRPO uses relative rather than absolute rewards.
  • α controls the trade-off between accuracy and efficiency, with its value relative to β determining the key balance.
  • High α, such as 5.0, stabilizes accuracy gains but produces weaker length reduction and longer outputs.
  • Low α, such as 0.1, can degrade reasoning capability and fail to learn effective concise reasoning paths.
  • Balanced α values, such as 1.0 or 2.0, preserve or improve accuracy while exerting substantial pressure to reduce output length.
  • The authors recommend starting with α in [2, 5] and tuning it according to the model family and objective.
  • ShorterBetter-1.5B used 4 A100 GPUs for approximately 16 hours, while ShorterBetter-7B used 8 A100 GPUs for less than 12 hours.

B.1 Coding Benchmarks

Coding-benchmark performance is reported in Table 4 and Figure 6, with Table 4 comparing accuracy and average inference length across model sizes and training methods.

  • B.1 Coding Benchmarks: Figure 6 presents model performance on coding benchmarks.
  • B.1 Coding Benchmarks: Table 4 reports coding-benchmark accuracy rates and average inference lengths in tokens across model sizes and training methods.Both ShorterBetter models use α = 2; baseline models are from Arora and Zanette (2025).

B.2 Training dynamics of ShorterBetter

Training analysis tracks output length, accuracy, SOL, and deviations from SOL to characterize how ShorterBetter learns efficient inference lengths.

  • Training dynamics of ShorterBetter: During GRPO training, output-length and accuracy trends are shown for ShorterBetter-1.5B and ShorterBetter-7B over the first 200 steps.
  • Training dynamics of ShorterBetter: For each problem, SOL is computed from eight sample generations and averaged batch-wise, while mean deviation measures relative distance from SOL.The deviation is defined as d_i = ℓ(y_i)−ℓ_SOL(G(x_i)) / ℓ_SOL(G(x_i)).
  • Training dynamics of ShorterBetter: Both SOL and mean deviation steadily decrease during training, especially within the first 100 steps.The trend suggests outputs increasingly cluster around the optimal length.
  • Training dynamics of ShorterBetter: Length-gap analysis uses 150 problems across AIME, AMC, MATH, Olympiad-bench, and Minerva, generating 10 responses per problem before filtering.

B.5 DeepSeek-R1-Distill-Llama-8B Results

ShorterBetter generalizes to the DeepSeek-R1-Distill-Llama-8B architecture, with α = 5 selected to balance accuracy and length reduction, although out-of-distribution performance declines relative to Qwen models.

  • B.5 DeepSeek-R1-Distill-Llama-8B Results: ShorterBetter is evaluated on DeepSeek-R1-Distill-Llama-8B to test generalization beyond the Qwen model series.
  • B.5 DeepSeek-R1-Distill-Llama-8B Results: α = 5 provides the chosen balance between accuracy and length reduction for the Llama-based model.
  • B.5 DeepSeek-R1-Distill-Llama-8B Results: Out-of-distribution performance shows a larger deduction than for Qwen-series models, suggesting that fewer training steps or larger α may be needed.
  • B.5 DeepSeek-R1-Distill-Llama-8B Results: Table 6 compares 8B Llama-based models across diverse tasks.
  • B.5 DeepSeek-R1-Distill-Llama-8B Results: The reasoning-trace analysis uses an LLM judge to categorize lines by their function in the reasoning process.The setup uses Gemini 2.5 Flash Preview with top_p = 0.95 and temperature = 0.7.
  • B.5 DeepSeek-R1-Distill-Llama-8B Results: The analysis labels adopted reasoning as steps explicitly corresponding to distinct elements of the condensed final solution.
Loading 2504.21370v4…