PaperlayerView exact revision on arXiv

Source-linked AI summary

When2Think: Learning Difficulty-Aware Length Control for Efficient Hybrid Reasoning Models

Jaejun Shim, HyunJin Kim, Young Jin Kim, JinYeong Bak

arXiv:2609.19671v1cs.AIcs.LG

TL;DR

Large reasoning models can overthink easy problems and underthink hard ones, while fixed compression or routing can trade token savings for hard-instance accuracy. When2Think uses difficulty-aware reward shaping and reference statistics to allocate reasoning depth. It improves the accuracy–efficiency trade-off across mathematical benchmarks, including on AIME24.

  • Problem

    Large reasoning models can overthink easy problems and underthink hard ones, while static token reduction may save computation on easy instances at the cost of hard-instance performance.

    Source: §1 · p. 1, §6.1 · p. 7

  • Method

    When2Think uses IDAC-based reward shaping with offline reference statistics to control whether to reason and how deeply within a hybrid reasoning policy.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Across diverse mathematical benchmarks, When2Think improves the accuracy–efficiency trade-off; on AIME24, it reduces token usage by 3,959 tokens (−27.9%) while improving accuracy from 46.0% to 56.0% (10.0%).

    Source: §6.1 · p. 7, §8 · p. 10

  • Takeaways & Limitations

    The paper frames efficient reasoning as allocating depth by instance rather than minimizing reasoning length uniformly.

    Source: §8 · p. 10

    The method relies on verifiable rewards and offline reference statistics, which may be less reliable when problem difficulty is ambiguous; open-ended tasks require additional reward modeling or human evaluation.

    Source: §7.4 · p. 9

Abstract

from arXiv · show

Large Reasoning Models (LRMs) achieve strong performance on complex tasks but exhibit systematic inefficiency: they often overthink easy problems and underthink hard ones. Existing approaches based on uniform length penalties or rigid routing incur an efficiency tax, trading reduced computation on easy instances for accuracy loss on hard instances. We formulate efficient reasoning as an instance-adaptive computation allocation problem and propose When2Think, a post-training framework for hybrid reasoning that dynamically allocates computation based on problem difficulty. Our method introduces Instance-level Difficulty-Aware Control (IDAC), a reward-shaping mechanism that leverages pre-computed reference statistics (accuracy and token usage) to regulate reasoning depth. Combined with verifier-based rewards and batch-wise standardized advantages, IDAC enables stable critic-free optimization without learned reward models or online reference-model queries. When2Think encourages direct answering on easy instances while preserving extended reasoning on hard instances, thereby learning when to use System 1 (NoThink) versus System 2 (Think). Experiments on mathematical benchmarks demonstrate improved accuracy-efficiency trade-offs: on AIME24, Pass@3 increases by 10.0% while token usage is reduced by 27.9% relative to the base model, and on AIME25, When2Think achieves 40.0% Pass@3, outperforming compression and routing-only baselines.

1 Introduction

Large reasoning models can waste tokens on easy problems yet stop too soon on hard ones. When2Think treats reasoning depth as an instance-level allocation problem and combines direct answering with deliberation.

1 Introduction

  • LRMs may generate redundant reasoning without accuracy gains on simple tasks, but terminate prematurely or produce fragmented reasoning on complex tasks.Static token-reduction strategies can save computation on easy instances while harming performance on harder ones.Source: §1 · pp. 1–2
  • Figure 1 compares Pass@1 accuracy and tokens per instance across MATH-500 difficulty levels for When2Think and LRM and LLM baselines.Its caption describes selective allocation across easy and hard instances; the introduction reports LLM-level efficiency on easy instances and LRM-level accuracy on harder ones.Source: Figure 1 · p. 2, §1 · p. 2
  • The paper reframes efficient reasoning as instance-level adaptive control, shifting the objective from token reduction to difficulty-aware computation allocation.Its analysis reports that token reduction can suppress overthinking on easy instances while inducing underthinking on hard ones.Source: §1 · p. 2
  • When2Think uses IDAC-based reward shaping to control whether the model reasons and how deeply it deliberates within one hybrid policy.The framework combines SYSTEM 1-style direct answering with SYSTEM 2-style reasoning.Source: §1 · p. 2

2 Related Work

Prior efficiency methods either shorten reasoning traces or select between direct and deliberative reasoning. The paper identifies limited instance-level control as a gap in hybrid approaches.

2 Related Work

  • The related-work survey organizes efficient multi-step reasoning into reasoning compression and hybrid reasoning.It frames both as post-training approaches for reducing reasoning cost in LRMs.Source: §2 · p. 2
  • Reasoning compression removes redundant steps or adds token budgets and length penalties, but static or globally tuned constraints can cause underthinking on hard instances.Examples include Skip-Thinking, LC-R1, ThinkPrune, LASER, and DLER.Source: §2 · p. 2
  • Hybrid reasoning methods switch between short and long trajectories or use controllers to skip reasoning on easy instances.The paper says these methods lack fine-grained instance-level control and presents When2Think as unifying SYSTEM 1 and SYSTEM 2 reasoning.Source: §2 · p. 3

3 Preliminaries

The preliminaries describe post-training and verifiable-reward reinforcement learning, then specify the importance-sampling setup and PPO-style optimization used for hybrid reasoning.

3 Preliminaries

  • Post-training modifies model parameters to encourage SYSTEM 2-style reasoning, while learning-to-reason approaches directly optimize reasoning policies for accuracy and efficiency.The paper focuses on reinforcement learning.Source: §3 · p. 3
  • RLVR uses deterministic outcome verification to reward task correctness without learned reward models or step-level supervision.The cited discussion contrasts outcome-based verification with process supervision, which requires step-level annotation.Source: §3 · p. 3
  • Importance sampling selects THINK or NOTHINK as the initial mode token and reweights sampled trajectories to keep optimization consistent with the target policy.The mode token controls whether explicit reasoning continues to an end-of-thinking token or the model directly generates an answer.Source: §3 · p. 3, Figure 2 · p. 4
  • A PPO-style clipped policy-gradient objective optimizes the target policy using importance-weighted trajectories collected from an auxiliary exploration policy.The clipping parameter is ϵ, and A_i,k denotes a trajectory-level advantage estimate.Source: §3 · p. 3

4 When2Think

When2Think uses offline reference statistics to adapt both whether a model reasons and how deeply it reasons for each instance. Its reward combines difficulty-aware length control with verifier-based correctness and batch-wise standardized advantages for critic-free optimization.

4 When2Think

  • When2Think learns whether to invoke explicit reasoning and how deeply to deliberate, using offline statistics to allocate computation per instance.It combines direct answering with multi-step reasoning in one hybrid policy.Source: §4 · p. 4

4.1 Reference Statistics

  • Reference accuracy α_i proxies instance difficulty, while average reference trajectory length τ_i supplies an instance-specific token budget.These statistics are precomputed for use during policy updates.Source: §4.1 · p. 4

4.2 Reward Formulation

  • IDAC modulates a correctness-gated efficiency bonus using trajectory length and reference statistics, applying stronger decay beyond reference length while allowing longer deliberation on harder instances.The final reward combines verifier correctness, the IDAC scaling factor, an efficiency bonus, and an instance-specific reference baseline.Source: §4.2 · p. 5

4.3 Batch-Wise Standardized Advantage

  • Batch-wise standardized advantages give every token in a trajectory the same advantage, supporting low-variance trajectory-level credit assignment without a learned critic.The standardized advantages feed the PPO-style clipped surrogate optimization.Source: §4.3 · p. 5

5 Experimental Setup

The experiments train When2Think on a competition-math dataset and assess accuracy and token usage across mathematical benchmarks. Comparisons include base language models, reasoning models, and efficiency-aware approaches.

5 Experimental Setup

  • The evaluation tests whether adaptive control maintains accuracy on difficult instances while reducing unnecessary computation on easier ones.Source: §5 · p. 6

5.1 Training

  • When2Think is trained on R1-distill-1.5B using approximately 40k competition-level mathematics problems from the DeepScaleR dataset.The backbone was distilled using approximately 800k curated samples via supervised fine-tuning.Source: §5.1 · p. 6

5.2 Evaluation

  • Across mathematical benchmarks, evaluation measures sampling-based pass@k accuracy and average generated tokens per instance, validating final answers with Math-Verify or MARIO Eval.Source: §5.2 · p. 6

5.3 Baselines

  • Baselines cover base language models, SFT- or RFT-trained reasoning models, and efficiency-aware reasoning models, including length-compression and hybrid-routing approaches.Methods share the same base model and comparable training conditions.Source: §5.3 · p. 6

6 Experimental Results & Analysis

The experiments test whether difficulty-aware allocation improves accuracy–efficiency trade-offs, adapts reasoning depth to problem difficulty, and benefits from IDAC, BWS, and importance sampling. Results report better trade-offs and instance-level allocation, with distinct contributions from the three mechanisms.

6.1 RQ1: Accuracy–Efficiency Trade-off under Adaptive Allocation

  • On AIME24, When2Think reduces token usage by 3,959 (−27.9%) while improving accuracy from 46.0% to 56.0% (10.0%).The authors frame this as mitigating the efficiency tax rather than merely shifting to a lower-token operating point.Source: §6.1 · p. 7

6.2 RQ2: Difficulty-aware Adaptive Allocation between System 1 and 2

  • On MATH-500 Level 1, When2Think uses 619 tokens versus R1-Distill’s 1,199 while maintaining 95.8% accuracy.The reported reduction is 580 tokens; the pattern also generalizes to easy GSM-Plus problems.Source: §6.2 · p. 7
  • On adversarially perturbed GSM-Plus instances, When2Think increases computation by 462 tokens, yielding a 6.3% absolute accuracy gain.On hard MATH-500 Level 5 problems it reduces tokens by 2,276 relative to R1-Distill while maintaining accuracy; on long-horizon AIME24/25, it maintains sufficient deliberation depth.Source: §6.2 · pp. 7–8
  • On MATH-500, the THINK-trajectory fraction rises from about 0.2 at Level 1 to over 0.7 at Level 5, unlike baselines’ weak or misaligned responses.Figure 4 reports Pass@1 accuracy and average token usage in its top panel and think ratio in its bottom panel, by difficulty level.Source: §6.2 · p. 8, Figure 4 · p. 8

6.3 RQ3: Mechanism Effects

  • On AIME24, When2Think reaches 56.0% Pass@3 while reducing token usage, whereas AdaptThink uses 5,806 tokens but reaches 44.7% Pass@3.The IS+BWS variant without IDAC has substantially worse validation accuracy, particularly on AIME24.Source: §6.3 · p. 8
  • BWS stabilizes critic-free optimization and maintains a think ratio of approximately 0.9, compared with 0.6 without BWS.It also improves convergence, stabilizes response length, and prevents degenerate reasoning behaviors.Source: §6.3 · p. 8
  • On GSM-Plus, full When2Think reduces token usage from 1,652 to 1,052 versus the variant without IS while maintaining competitive accuracy.The passage attributes the primary accuracy gains to IDAC and BWS, with IS primarily promoting effective NOTHINK use on easier instances.Source: §6.3 · p. 8

7 Discussion

The discussion frames reasoning efficiency as allocating depth across instances rather than simply compressing outputs. It describes continuous depth control and stable optimization, while noting scale- and reward-related scope boundaries.

7.1 The Efficiency Tax: From Compression to Control

  • Static or heuristic compression can suppress overthinking on easy problems while under-allocating reasoning on hard ones; When2Think instead reallocates reasoning depth by instance.The discussion characterizes inefficiency as misallocation rather than insufficient model capacity.Source: §7.1 · p. 9

7.2 Mechanisms: Continuous Control and Stability

  • IDAC uses instance-level statistics to scale reasoning incentives continuously, controlling how much to reason beyond discrete THINK routing.Instance-specific baselines and batch-wise standardization support critic-free, stable learning of length-sensitive policies.Source: §7.2 · p. 9

7.3 The Paradox of Scale: Overthinking and the Visibility of Efficiency Tax

  • Efficiency-tax effects are most visible at smaller scales; larger models show stronger inherent reasoning efficiency, making the effects less pronounced.The paper focuses its analysis on the 1.5B scale, where adaptive-control effects are most diagnostic.Source: §7.3 · p. 9

7.4 Limitations

  • The approach relies on verifiable rewards and is most directly applicable to mathematical reasoning; open-ended tasks need additional reward modeling or human evaluation.Offline reference statistics may also be less reliable when problem difficulty is ambiguous.Source: §7.4 · p. 9

8 Conclusion

The conclusion presents reasoning efficiency as an allocation problem, not a compression objective. When2Think uses IDAC and BWS to learn depth-aware policies and reports improved accuracy–efficiency trade-offs across mathematical benchmarks.

8 Conclusion

  • Static compression can under-allocate reasoning on difficult problems, so the conclusion frames inefficiency as misallocated reasoning depth across instances.The proposed alternative treats reasoning depth as an adaptive resource rather than a quantity to minimize.Source: §8 · pp. 9–10
  • IDAC teaches the model both when and how much to reason, while BWS supports stable, critic-free optimization of depth-aware policies..Source: §8 · p. 10
  • Across diverse mathematical benchmarks, When2Think improves the accuracy–efficiency trade-off by reducing unnecessary computation while preserving or expanding reasoning where needed.The conclusion presents allocation rather than compression as a direction for scalable reasoning systems.Source: §8 · p. 10

A.1 Reproducibility

The appendix describes reproducibility choices, computational infrastructure, a standardized inference and verification protocol, and a dual-verifier pipeline for mathematical answers.

A.1 Reproducibility

  • The experiments fix random seed 42 across data shuffling, model initialization, and sampling; hyperparameters follow prior reasoning-alignment studies.Table 2 is identified as the training-configuration hyperparameter table.Source: §A.1 · p. 17, Table 2 · p. 17

A.2 Infrastructure

  • Training uses about 70 GPU-hours for the 1.5B model and 120 GPU-hours for the 7B model; evaluation uses about 20 GPU-hours.Training uses verl with DDP and FlashAttention-2; inference and evaluation use vLLM.Source: §A.2 · p. 17, Table 3 · p. 17

B.1 Inference and Verification Protocol

  • Inference parameters and answer verification are standardized across models; Pass@k uses extracted final boxed answers and a unified binary correctness signal.The same correctness signal serves as the evaluation metric and terminal reward for policy optimization.Source: §B.1 · p. 18

B.2 Dual Math Verifier

  • The dual-verifier pipeline checks semantic equivalence using symbolic and numerical verification, with type-aware checks for diverse answer formats.Math-Verify handles extraction and symbolic equivalence; MARIO Eval adds type-specific checks and may use a language-model fallback when verification is inconclusive.Source: §B.2 · p. 18

B.3 Benchmarks

The evaluation uses mathematical benchmarks spanning foundational problems to undergraduate STEM. It also defines a trajectory structure for analysis and an efficiency metric for hybrid reasoning.

B.3 Benchmarks

  • The benchmark suite spans foundational arithmetic, competition problems, olympiad challenges, and undergraduate STEM tasks.This range supports analysis of accuracy and computational efficiency across problem complexities.Source: §B.3 · p. 19
  • GSM-Plus extends GSM-8K with perturbed variants, including paraphrases, numerical changes, distractors, and reasoning-structure changes.After excluding unanswerable or incomplete cases, the evaluation set contains approximately 9k problems.Source: §B.3 · p. 19
  • MATH-500 is a 500-problem held-out subset designed to avoid contamination from the PRM-800K training pipeline.It retains the original test split’s subject distribution and difficulty profile.Source: §B.3 · p. 19
  • Minerva comprises 272 automatically verifiable undergraduate STEM problems, while the OlympiadBench evaluation uses 674 open-ended, text-only English math problems.Minerva includes 191 numerically verifiable and 81 symbolically verifiable items.Source: §B.3 · p. 19
  • The AIME evaluation combines both 2024 and 2025 administrations to avoid overlap with DeepScaleR training material through 2023.The two administrations together contribute 30 problems.Source: §B.3 · p. 19

B.4 Thoughts & Steps

  • A reasoning trajectory is represented as ordered thoughts, each a coherent reasoning segment that can contain multiple intermediate steps.Thought boundaries are heuristically identified by discourse markers, while steps use formatting cues such as double newlines.Source: §B.4 · p. 20

B.5 Accuracy-Cost Efficiency

  • ACE evaluates hybrid-reasoning efficiency by reducing the score when correct answers are followed by redundant reasoning tokens.Incorrect predictions score zero; correct predictions receive maximal efficiency when no redundant reasoning occurs.Source: §B.5 · p. 21

C.1 Analysis of Token Distribution

When2Think’s token-use patterns differ by reasoning mode and correctness, and its benchmark comparisons generally show higher accuracy with fewer tokens. The analysis argues that batch-wise standardization preserves difficulty signals needed to reward long successful reasoning on hard problems.

C.1 Analysis of Token Distribution

  • Correct NOTHINK responses peak at low token counts, whereas incorrect THINK responses have a heavy-tailed token-use distribution reaching the context limit.Correct THINK responses also use more tokens than correct NOTHINK responses.Source: §C.1 · p. 22, Figure 5 · p. 22

C.2 Reasoning Mode Performance

  • The mode-level tables report accuracy, token use, and related reasoning measures for THINK and NOTHINK across five runs.The reported metrics include Pass@3 accuracy, average token usage, and step counts for THINK, plus accuracy, token usage, and selection ratio for NOTHINK.Source: Table 4 · p. 22, Table 5 · p. 22, Table 6 · p. 23

C.3 Performance Across Benchmarks

  • Across five benchmarks, When2Think generally occupies the higher-accuracy, lower-token region relative to R1-Distill-Qwen.The comparisons use Pass@3 accuracy and token usage, with the token axis reversed so farther right indicates fewer tokens.Source: Figure 6 · p. 23

D.1 Reward and Advantage Landscapes under IDAC and BWS

  • BWS preserves positive advantages for long successful trajectories on hard problems, while instance-local GRPO-style normalization can fragment those signals.The comparison uses simulated THINK trajectories built from offline R1-Distill-Qwen-1.5B statistics on DeepScaleR.Source: §D.1 · p. 26, Figure 7 · p. 26

D.2 Analysis of BWS Training Dynamics

The analyses attribute stable difficulty-aware reasoning depth chiefly to IDAC and batch-wise standardization, with importance sampling shifting the policy toward more efficient hybrid behavior on easier problems. They also examine verifier trade-offs, model-scale and supervision effects, illustrative reasoning cases, and broader deployment boundaries.

D.2 Analysis of BWS Training Dynamics

  • BWS reduces optimization instability, maintains reasoning depth without a value function, and lets When2Think reach peak AIME24 performance in about half the baseline’s training steps.Without BWS, policies can collapse to minimal reasoning or become excessively verbose; BWS also reduces response-length bloat without corresponding accuracy gains.Source: §D.2 · p. 27, Figure 8 · p. 27

D.3 Analysis of IS & IDAC Training Dynamics

  • IDAC with BWS remains competitive on AIME24 without importance sampling, while importance sampling lowers token use and Think Ratio on GSM8K at comparable accuracy.The ablation also finds that mode exploration alone does not provide IDAC’s control over reasoning depth across easy and hard instances.Source: §D.3 · p. 28, Figure 9 · p. 28

D.4 Ablation on Hybrid Bonus Design

  • Larger NOTHINK bonuses tend to reduce validation accuracy, while smaller unified efficiency bonuses yield more stable training behavior.The unified coefficient applies to both modes, with DADS controlling the bonus strength for THINK trajectories.Source: §D.4 · p. 29, Figure 10 · p. 29, Figure 11 · p. 29

D.5 Effectiveness of the Dual Math Verifier

  • The Dual Math Verifier improves robustness to symbolic and text-heavy answers in Geometry and Precalculus, but strict parsing can reject equivalent forms in Intermediate Algebra.The authors also note that IDAC’s efficiency constraints may limit reasoning steps needed for some algebraic transformations.Source: §D.5 · p. 30, Figure 12 · p. 30

E.1 Overthinking on Easy Problem

  • The examples label a 32,206-token GSM-Plus response from R1-Distill-Qwen-1.5B as correct overthinking and a 17,075-token AIME24 response from the same named model as incorrect underthinking.Figure 15 is captioned “When2Think correct NOTHINK”; another GSM-Plus example is an incorrect 8,837-token response tagged R1-Distill-Qwen-1-5B.Source: §E.1 · p. 31, §E.2 · p. 32, Figure 15 · p. 33

F.1 Impact of Model Scale and Supervision on Reasoning Efficiency

  • On GSM-Plus, 7B models use more tokens despite lower accuracy than 32B models, and RFT models use more tokens than SFT counterparts for modest accuracy gains.The paper associates capability-induced overthinking with smaller models and attributes RFT token inflation to objectives without explicit token constraints.Source: §F.1 · p. 35, Figure 17 · p. 35

G.1 Dual Process Theory in LLMs.

  • The related-work discussion contrasts external inference-time methods, which can add overhead, with post-training approaches that internalize reasoning but may require longer training and face generalization limits.It frames reasoning through System 1’s fast, intuitive mode and System 2’s slower, deliberate mode.Source: §G.1 · p. 36, §G.2 · p. 36, §G.3 · p. 36

H.1 Broader Impacts

  • The broader-impact discussion identifies possible deployment-cost reductions but warns that use beyond verifiable mathematics needs safeguards such as domain-specific validation and human oversight.The work uses public assets under their applicable licenses and terms.Source: §H.1 · p. 37, I Licenses and Terms · p. 37, Table 11 · p. 37
Loading 2609.19671v1…