PaperlayerView exact revision on arXiv

Source-linked AI summary

The Art of Efficient Reasoning: Data, Reward, and Optimization

Taiqiang Wu, Zenan Xu, Bo Zhou, Ngai Wong

arXiv:2602.20945v3cs.CLcs.AI

TL;DR

Efficient reasoning seeks to reduce the computational burden of long Chain-of-Thought while retaining accuracy, but prior work has emphasized reward design over the broader training recipe. This paper systematically studies data, reward, and optimization with fine-grained metrics, finding two-stage training, the importance of positive reward density, and length-bias generalization across domains and model scales.

  • Problem

    Prior efficient-reasoning methods focus mainly on reward design while giving less attention to data composition and optimization strategy.

  • Method

    The paper uses a unified protocol to analyze prompts, rollouts, reward shaping, and optimization with correctness-conditioned lengths and token-budget evaluations from 2k to 32k.

  • Results

    The study identifies length adaptation and reasoning refinement as two training stages, while learned length bias generalizes across domains and the strategy reduces CoT length by 15% to 47.3% while preserving or improving Mean@8.

  • Takeaways & Limitations

    Stable efficient reasoning requires sufficient positive reward signals and avoiding the short-is-correct trap; the resulting guidelines generalize across domains and difficulty levels.

  • Takeaways & Limitations

    The study trains on mathematical prompts and validates on math and coding benchmarks, leaving broader domains such as creative writing for future evaluation.

Abstract

from arXiv · show

Large Language Models (LLMs) consistently benefit from scaled Chain-of-Thought (CoT) reasoning, but also suffer from heavy computational overhead. To address this issue, efficient reasoning aims to incentivize short yet accurate thinking trajectories, typically through reward shaping with Reinforcement Learning (RL). In this paper, we systematically investigate the mechanics of efficient reasoning for LLMs. For comprehensive evaluation, we advocate for more fine-grained metrics, including length distribution conditioned on correctness and performance across a wide spectrum of token budgets ranging from 2k to 32k. First, we reveal that the training process follows a two-stage paradigm: length adaptation and reasoning refinement. Through extensive experiments (about 0.2 million GPU hours) in a unified protocol, we deconstruct training prompts and rollouts, reward shaping, and optimization strategies. A central finding is to maintain a sufficient density of positive reward signals and avoid the short-is-correct trap. Moreover, the learned length bias generalizes across domains and difficulty levels. We distill these findings into valuable insights and practical guidelines, and validate them across the Qwen3 models ranging from 0.6B to 30B, demonstrating the robustness and generalization. Weights are available at https://wutaiqiang.github.io/project/Art

1 Introduction

Efficient reasoning addresses the latency and computational costs of long Chain-of-Thought by rewarding short, accurate trajectories. The paper systematically studies data, reward, and optimization, revealing two training stages and validating practical guidelines across Qwen3 models.

  • Motivation: Longer Chain-of-Thought reasoning improves LLM capabilities but introduces substantial deployment latency.Efficient reasoning targets this trade-off by encouraging shorter accurate trajectories.
  • Research gap: Prior work largely emphasizes reward design while overlooking training data composition and optimization strategy.The paper addresses this gap through a unified investigation of the broader training recipe.
  • Key findings: Training follows two stages: length adaptation, followed by reasoning refinement within the learned length scope.The paper characterizes this paradigm as a central mechanism of efficient reasoning.
  • Evaluation: Fine-grained evaluation compares correctness-conditioned length distributions and performance across token budgets from 2k to 32k.These metrics expose budget-dependent differences between efficiency strategies.
  • Method and contribution: The study deconstructs data, rollout, reward, and optimization choices, then distills the findings into practical training guidelines.The analysis covers prompt difficulty, rollout number, reward assignment, and optimization strategies.

2 Preliminary

The paper studies efficient reasoning through RL reward shaping and evaluates it with correctness-conditioned lengths and performance across token budgets. Training follows length adaptation before reasoning refinement, with behavior varying substantially by budget and generalizing from math training to code.

  • Experimental setup: RL generates multiple reasoning trajectories per prompt and updates the policy using rollout-derived reward signals.The experimental setup uses N trajectories for each input prompt and policy-gradient optimization.
  • Reward engineering: Reward shaping supplements correctness-based rewards to incentivize concise yet accurate rollouts under a target length.The truncation baseline treats correctness and the target token length as central reward conditions.
  • Evaluation protocol: The evaluation tracks correctness-conditioned length distributions and performance at inference budgets B ∈{2k, 4k, 8k, 16k, 32k}.The benchmarks include mathematical reasoning tasks and LiveCodeBench.
  • Two-stage paradigm: Training exhibits two stages: rapid length adaptation followed by reasoning refinement within the stabilized target-length region.Stage I reduces average token consumption, while Stage II improves performance after length curves plateau.
  • Budget dependence: Reward-shaping behaviors depend strongly on token budget: aggressive penalties help at 2k but can cause collapse at 32k, whereas Laser later recovers.At 32k, Laser follows a U-shaped trajectory as reasoning refinement offsets an initial compression-related decline.
  • Cross-domain generalization: Length bias learned from mathematical prompts transfers to code, with Kimi strongest at 2k on LiveCodeBench and comparable at larger budgets.The reported cross-domain similarity indicates that the learned length preference extends beyond the training domain.

3 Experiments and Guidelines

The experiments show that efficient reasoning depends on sufficient positive reward density, careful reward assignment, and optimization choices that balance speed, accuracy, and stability.

  • Prompt Difficulty: Training on hard prompts can cause entropy spikes, premature length collapse, and degraded downstream performance.Sparse positive samples let length penalties on incorrect rollouts dominate, encouraging overfitting to short outputs.
  • Prompt Difficulty: Easier prompts provide denser positive signals and produce more stable training, while retaining performance on relatively tough tasks.Training on easy prompts yields performance on AIME’25 comparable to or slightly exceeding training on the full dataset.
  • Rollout Number: More rollouts accelerate length adaptation and improve reasoning refinement on mathematical benchmarks, but increase computational cost.With larger N, short correct trajectories are easier to discover; all settings reach a similar length floor, while larger N achieves higher asymptotic Mean@8.
  • Reward on Negative Rollouts: Masking all incorrect rollouts creates the short-is-correct trap, causing entropy explosion and precipitous rollout-length collapse.The resulting signal contains positive rewards for short correct outputs and negative rewards only for overlong correct outputs.
  • Reward on Negative Rollouts: Masking overlong correct rollouts improves performance but produces longer outputs, whereas target-length sampling avoids the length trap and achieves a better trade-off.Target-length sampling reaches the optimal Pareto frontier by keeping negative samples shorter while preserving similar positive samples.
  • Off-policy Optimization: Higher off-policy staleness shortens length adaptation and can improve accuracy, but increases instability risks such as rising entropy.The authors recommend on-policy training for larger and more fragile models despite acceleration before 800 steps.

4 Extensive Analysis

The proposed strategy generalizes across Qwen3 model scales and changes reasoning trajectories toward shorter, more streamlined outputs.

  • Cross-scale Evaluation: The strategy is evaluated across Qwen3 models ranging from 0.6B to 30B on AIME’25.The experiments test whether the derived guidelines remain robust across model scales.
  • Quantitative Results: On Qwen3-0.6B, Mean@8 increases from 13.33 to 24.58 while average response length decreases from 14.9k to 8.9k.Pass@8 also rises to 36.67 compared with the vanilla score of 26.67.
  • Case Study: Qualitative comparisons indicate that optimized models produce shorter outputs and reorganize CoTs into a more streamlined, expert-like format.The comparison is made between vanilla and optimized reasoning trajectories.

5 Related Work

Related work reduces reasoning cost through shorter outputs, latent-space reasoning, or efficient decoding; this paper instead studies the mechanics of RL-based efficiency optimization.

  • Efficient Reasoning: Efficient reasoning methods target overthinking and the inference costs of long-form CoT through SFT- or RL-based shortening approaches.These methods train long reasoning chains to become shorter.
  • Alternative Directions: Other research explores architectural innovations such as latent-space reasoning and more efficient decoding.These directions differ from the paper’s focus on RL-based efficiency optimization.
  • Reward Shaping: Reward-shaping work commonly promotes shorter correct responses, while some approaches promote longer incorrect rollouts for exploration.The literature therefore considers different treatments of negative or incorrect rollouts.

6 Conclusion

The paper identifies a two-stage training paradigm, develops fine-grained evaluation criteria, and derives practical guidelines for robust efficient reasoning.

  • Core Findings: Efficient-reasoning training follows length adaptation followed by reasoning refinement.The model first adapts to token constraints, then optimizes performance within the resulting length scope.
  • Practical Guidelines: The analysis emphasizes sufficient and effective rewards, especially through relatively easier training prompts.The learned reasoning-length bias generalizes across domains and difficulty levels, with evaluations extending to models up to 30B.

Limitation and Future Work

The paper identifies limitations in domain coverage, length control, model scale, and supervision granularity, leaving several directions for future research.

  • Domain diversity: The evaluation trains on mathematical prompts and validates on math and coding benchmarks, leaving domains such as creative writing untested.The authors also leave open whether more diverse prompts improve performance.
  • Adaptive length: The study uses fixed rollout and target lengths rather than adapting them to prompts or the current language model.Adaptive target length is explored only in a toy setting, while adaptive rollout length could be precomputed.
  • Evaluate on larger models: Experiments focus mainly on DeepSeek-R1-Distill-Qwen-1.5B, with some Qwen3 validation but no evaluation on extremely large models such as Qwen3-235B-A22B-Instruct-2507.The authors cite limited GPU availability as the reason for this boundary.
  • More fine-grained supervision: The work studies reward-shaping-based RL but does not apply fine-grained refinement to the generated chains of thought.The authors identify tool creation and reuse during reasoning as another open topic.

A Detailed Results on Reward Engineering

The reward-engineering results show that efficient-reasoning strategies behave differently across token budgets, making broad budget evaluation necessary.

  • Reward formulations: Kimi-1.5 normalizes rollout length using the minimum and maximum lengths within each rollout group before constructing its reward.Its reward is described as the sum of a correct reward and an incorrect penalty.
  • Reward formulations: Laser combines a correctness base reward with a bonus when the rollout length is below the target length.The cited formulation also includes a variant that adds an exploration bonus for incorrect rollouts.
  • Evaluation scope: The additional training-dynamics results cover AIME’25, MATH-500, AMC, Minerva Math, Olympiad Bench, and LiveCodeBench.These results are presented in Figures 9 and 10.
  • Training dynamics: The training process follows length adaptation before reasoning refinement, while strategies exhibit distinct behavior across token budgets.At 2k, aggressive Kimi penalties can help, whereas at 32k Kimi may stagnate or collapse and truncation recovers more evenly.

B Detailed Results for Data Selection

Data difficulty strongly affects efficient-reasoning training: easier prompts provide denser positive rewards and produce more stable, transferable length adaptation.

  • Easy and hard prompts: Easier prompts let models reduce length without compromising performance because they provide sufficient and effective reward signals.The authors present this as the explanation for the observed stability of reasoning distillation.
  • Easy and hard prompts: Training exclusively on DeepScaleR-Hard causes instability, including violent fluctuations or complete collapse after initial length adaptation.The issue appears on benchmarks including AIME’25 and Olympiad Bench.
  • Easy and hard prompts: DeepScaleR-Easy matches or rivals full-data training and can perform comparably or better on difficult AIME’25 and LiveCodeBench tasks.This supports generalization of the learned length bias across domain and difficulty.

C Detailed Results for More Rollouts

More rollouts consistently improve convergence and efficient-reasoning stability, but their benefits depend on task difficulty and increase computational cost.

  • Consistent benefits across settings: Increasing the rollout count N benefits convergence and asymptotic performance across both long-context and tightly constrained settings.The authors interpret this as a robust way to increase reward-signal density.
  • Task-dependent sensitivity: On difficult AIME’25 and LiveCodeBench tasks, the performance gap between N = 8 and N = 32 is relatively narrow.The passage suggests that more rollouts are insufficient when the base model lacks the capability to solve the problem.
  • Task-dependent sensitivity: On AMC and MATH-500, larger N produces substantial performance separation on Mean@8.The stronger separation occurs on relatively easier or intermediate tasks.
  • Mean@8 vs. Pass@8: Larger N substantially improves Mean@8 while Pass@8 often remains stagnant or improves only marginally.The results indicate reduced policy variance rather than the ability to solve previously unsolvable problems.
  • Overall trade-off: Increasing N primarily stabilizes the policy on solvable problems rather than expanding its upper bound, while increasing computing overheads.The practical trade-off is improved stability at higher training cost.
  • Qualitative behavior: Hard prompts can produce reasoning collapse through overshort outputs, whereas the proposed strategy removes conversational fluff and yields a denser formal structure.The authors attribute hard-prompt collapse to sparse positive samples and describe the resulting outputs as more streamlined and expert-like.
  • Model-specific settings: The reported Qwen3 settings align rollout and target limits in most cases, while instruction models receive shorter targets to encourage further output reduction.Examples include LR = LT = 8k for Qwen3-0.6B and LR = LT = 16k for Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507.

F Results on Private Benchmark

The compressed models generalize their learned reasoning-length reduction across domains while preserving performance, but adaptive target lengths can become unstable and fail under generous budgets.

  • Cross-domain generalization: 15% CoT reduction is achieved on both AIME’25 and a private benchmark spanning 10 diverse domains.The model is trained on mathematical prompts, yet the same compression pattern transfers across domains.
  • Performance preservation: The proposed model substantially reduces token usage with minimal impact on Mean@4 and Pass@4 across private out-of-distribution domains.Table 3 reports Vanilla / Ours comparisons for diverse domains.
  • Adaptive target length: Adaptive target training shows many KL-divergence spikes, indicating rapidly changing gradients and potential instability.The authors leave exploration of additional settings for future work.
  • Training-data sensitivity: Training on DeepScaleR-Hard with truncation causes reasoning collapse, including skipping the double-check step at shorter lengths.The comparison contrasts DeepScaleR-Easy and DeepScaleR-Hard training.
  • Case comparisons: Both vanilla and trained models produce correct answers in the reported case comparisons.These examples illustrate correctness preservation in the selected cases.
Loading 2602.20945v3…