Source-linked AI summary
Does Your Reasoning Model Implicitly Know When to Stop Thinking?
Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuanda Wang, Zhixia Zhang, Hongyan Xie, Songshi Liang, Zehao Chen, Xuefeng Xiao, Fuzhen Zhuang, Jianxin Li, Deqing Wang, Yikun Ban
TL;DR
Long-chain reasoning can be redundant and longer chains do not reliably improve correctness, raising whether LRMs implicitly know when to stop. The paper introduces SAGE, which uses self-confidence to discover concise effective chains, and integrates it into RLVR as SAGE-RL. The resulting approach improves reasoning accuracy and efficiency across multiple challenging mathematical benchmarks.
Problem
Longer reasoning chains frequently contain redundancy, are not reliably correlated with correctness, and can impair computational efficiency.
Method
SAGE uses model self-confidence to discover precise reasoning chains, while SAGE-RL integrates SAGE as mixed sampling into group-based reinforcement learning.
Results
SAGE-RL yields CoT length reduction and accuracy improvement, with lasting inference-time reasoning-efficiency gains across challenging mathematical benchmarks.
Takeaways & Limitations
LRMs’ efficient reasoning potential can be exposed through alternative sampling and incorporated into standard pass@1 inference.
Takeaways & Limitations
SAGE-RL adds computational overhead through candidate-sequence memory use and repeated generation calls, although sufficient parallelism can make wall-clock time comparable.
Abstract
from arXiv · showhide
Recent advancements in large reasoning models (LRMs) have greatly improved their capabilities on complex reasoning tasks through Long Chains of Thought (CoTs). However, this approach often results in substantial redundancy, impairing computational efficiency and causing significant delays in real-time applications. Recent studies show that longer reasoning chains are frequently uncorrelated with correctness and can even be detrimental to accuracy. In a further in-depth analysis of this phenomenon, we surprisingly uncover and empirically verify that LRMs implicitly know the appropriate time to stop thinking, while this capability is obscured by current sampling paradigms. Motivated by this, we introduce SAGE (Self-Aware Guided Efficient Reasoning), a novel sampling paradigm that unleashes this efficient reasoning potential. Furthermore, integrating SAGE as mixed sampling into group-based reinforcement learning (SAGE-RL) enables SAGE-RL to effectively incorporate SAGE-discovered efficient reasoning patterns into standard pass@1 inference, markedly enhancing both the reasoning accuracy and efficiency of LRMs across multiple challenging mathematical benchmarks.
1. Introduction
Longer reasoning chains can contain substantial redundancy, yet LRMs appear to recognize concise effective paths that current sampling often fails to select. The paper introduces SAGE and SAGE-RL to uncover and learn these efficient reasoning patterns.
- Motivation: Longer chains may be uncorrelated with correctness, while shorter responses can match or exceed accuracy with fewer tokens.DeepSeek-R1 responses were nearly 5× longer than Claude 3.7 Sonnet on AIME 2025 with comparable accuracy; QwQ-32B’s shortest responses used 31% fewer tokens and performed 2 percentage points better.
- Motivation: LRMs consistently assign high confidence to concise, effective reasoning paths during multi-chain exploration, but pass@1 paradigms often fail to select them.The paper reports convergence of this behavior as the exploration space expands.
- Contributions: The paper proposes SAGE, a sampling paradigm that uses model self-confidence to discover relatively precise reasoning chains.SAGE is designed to expose efficient reasoning potential obscured by existing sampling paradigms.
- Contributions: SAGE-RL integrates SAGE into RLVR rollouts so models learn concise, effective thinking patterns without altering their original reasoning paradigm.It is presented as a simple modification to group-based reinforcement learning frameworks.
- Results: SAGE-RL-tuned models achieve consistent gains across six challenging mathematical reasoning benchmarks.The listed benchmarks are MATH-500, AIME 2024, AIME 2025, AMC23, OlympiaBench, and Minerva.
2. Dilemmas of Reasoning Models under Current Sampling Paradigms
Evidence across multiple-sample analyses shows that increasing chain length does not reliably improve correctness and that pass@1 often preserves redundant reasoning after the answer is already obtained. The section motivates exploring a broader sampling space to retrieve concise effective chains.
- Pass@k: On AIME 2025, DeepSeek-R1 responses were nearly 5× longer than Claude 3.7 Sonnet with comparable accuracy, while QwQ-32B’s shortest responses used 31% fewer tokens and scored 2 percentage points higher.These comparisons challenge the assumption that longer chains are more likely to produce correct solutions.
- Pass@k: Once chain-of-thought length reaches a threshold, scaling it further does not correspondingly improve reasoning capability.The optimal response within the model’s capability is obscured by existing sampling paradigms and can currently be retrieved post hoc through test-time scaling.
- Pass@1: A response can derive the correct answer using 500 tokens yet continue for 452 redundant tokens under the current sampling strategy.The example illustrates delayed termination after the answer has already been produced.
- Pass@1: More than half of samples from all evaluated models contain ineffective reasoning steps, with no substantial RFCS improvement for stronger or more post-trained models.RFCS(<1) counts correct responses whose first correct step is not the final step, while RFCS(avg) averages the ratio across correct responses.
- Pass@1: Existing reasoning models struggle to terminate thinking at the appropriate moment under pass@1, motivating intentional exploration beyond standard sampling.The paper proposes using a larger exploration space to uncover precise reasoning chains hidden within the broader pass@k distribution.
3. Intentionally Exploring Shorter CoTs
SAGE intentionally explores shorter reasoning chains by maintaining multiple candidate sequences ranked with Φ, then terminating when enough candidates confidently reach </think>. It subsequently derives answers greedily from the retained reasoning chains.
- Token-Wise Reasoning Path Exploration: Φ ranks candidate reasoning sequences by their average cumulative log-probability during token-wise expansion.The method maintains the top-m sequences according to Φ and expands them until the maximum step budget is reached.
- Token-Wise Reasoning Path Exploration: Each retained sequence expands through its top 2m probable next tokens, producing a candidate group of size 2m2.The candidates are formed by appending probable tokens to each beam before ranking sequences for the next iteration.
- Token-Wise Reasoning Path Exploration: The algorithm retains the top-m highest-scoring candidate sequences for the next decoding iteration.Selection is based on the Φ scores of the expanded candidates.
- Exploration Termination: SAGE accepts a sequence when </think> appears within the top-h probable tokens and the required number of completions has not yet been reached.Candidates failing this termination criterion are discarded because the model’s confidence in ending the thinking process is low; the process stops when at least r completions are collected.
- Greedy Sampling of the Answers: After collecting r reasoning chains, SAGE greedily derives an answer from each chain and outputs the resulting r completions.Each completion pairs a reasoning chain with its greedily sampled answer.
4. Your Reasoning Model Implicitly Knows When to Stop Thinking
The experiments show that confidence-guided exploration uncovers shorter, high-quality reasoning paths and that LRMs increasingly approach this efficient stopping capability as exploration expands.
- TSearch w/ Φ retains candidate sequences using Φ, while TSearch w/ ϕ uses the most probable next token; EW = 0 corresponds to greedy sampling.
- Increasing m with Φ consistently reduces response length while improving accuracy, whereas removing Φ causes accuracy to degrade alongside length collapse.
- With Φ, varying TR has virtually no performance impact, while TR strongly affects TSearch w/ ϕ, indicating that Φ ranks termination consistently.
- With Φ, </think> consistently ranks first when it appears; without Φ, its rank ratio increases, indicating uncertainty and overlooked short, high-quality chains.
- As exploration width grows, pass@1 improves while response length decreases toward convergence across models and datasets, revealing increasingly precise and compact reasoning paths.
- Adequate exploration lets LRMs identify concise chains and terminate appropriately, whereas purely sampling-based strategies limit this capability through next-token probabilities.
5. Self-Aware Guided Efficient Reasoning
SAGE converts the observed stopping behavior into a sampling method that explores reasoning stepwise, retains high-confidence branches, and returns efficient completions.
- 5.1. Methodology: SAGE expands candidate sequences by independently sampling 2^m reasoning steps at each reasoning step until the maximum limit Tmax.
- 5.1. Methodology: SAGE terminates a branch when it ends with </think>, adding that completed sequence to the candidate output set.
- 5.1. Methodology: Degrade SAGE samples one reasoning step per iteration, while SAGE (2,1) is used as the representative algorithm to balance efficiency and performance.
- 5.2. SAGE Inference Scaling Trends with Step Budget: Under constrained step budgets, SAGE achieves higher pass@1 with similar sequence lengths by stopping earlier and producing more complete chains.
- 5.2. SAGE Inference Scaling Trends with Step Budget: SAGE favors accuracy gains on stronger models and harder datasets, but larger response-length reductions on weaker models and simpler datasets.
6. SAGE-RL: Integrating Efficient Reasoning Patterns into Current Inference Paradigms
SAGE-RL incorporates SAGE into RLVR rollouts through mixed sampling, combining SAGE-generated responses with standard randomly sampled responses.
- SAGE-RL modifies RLVR only in the rollout phase by using hybrid sampling for each question.
- SAGE generates r responses with SAGE(m,r), while standard random sampling generates the remaining G−r responses in each group.
7. Experiments
Experiments show that SAGE-RL improves reasoning capability and token efficiency across challenging benchmarks, while its rollout-only modification adds moderate computational overhead. The analyses also indicate that SAGE discovers shorter, high-confidence reasoning chains that standard sampling often misses.
- SAGE-RL improves reasoning capability and token efficiency, whereas AdaptThink reduces DS-1.5B tokens from 4,882 to 2,563 on MATH-500 but lowers pass@1 by 2.8%.
- SAGE-RL consistently achieves the best or second-best token efficiency across benchmarks while improving base-model capability through shorter, more precise reasoning chains.
- SAGE-RL learns shorter yet more accurate reasoning patterns even though SAGE changes only 2 of 8 samples per group.
- SAGE-RL produces larger entropy reductions and policy shifts than standard RLVR, indicating growing confidence in SAGE-identified reasoning chains during training.
- Replacing 2 of 8 rollouts with SAGE(2,2) adds 4× GPU memory overhead for those trajectories, although sufficient parallelism can make wall-clock time comparable.
- SAGE-GRPO adds only moderate wall-clock training cost under verl while maintaining nearly identical peak memory usage to GRPO.
- SAGE-GRPO-tuned models show fewer correct responses with RFCS<1 and higher RFCS(avg), indicating more frequent termination immediately after the correct answer.
8. Conclusion
The paper concludes that LRMs contain an implicit ability to stop reasoning at an appropriate point, but current sampling obscures it. SAGE exposes this ability, and SAGE-RL transfers the resulting efficient reasoning patterns into inference-time gains.
- SAGE uncovers precise reasoning chains that reduce CoT length and improve accuracy, while SAGE-RL integrates this capability into RLVR rollouts.
Impact Statement
The paper frames efficient reasoning as an alternative to redundancy-prone long-chain reasoning and positions SAGE as a training-free, self-confidence-guided approach. It contrasts this approach with offline training and objective-level reward modifications.
- Long CoTs can encourage advanced search and backtracking behaviors, but RLVR also biases models toward redundancy over the risk of error.
- Prior work identifies lengthy outputs that may neither improve accuracy nor introduce new solution strategies, motivating efficient-reasoning methods.
- SAGE is training-free and selects the currently optimal candidate sequence at each inference step using the model’s self-awareness.
- Offline methods face difficulty obtaining high-quality short CoTs and may limit exploration on difficult problems.
- The paper avoids offline distillation because reliance on a strong teacher could limit the model’s reasoning capability.
- Instead of modifying the optimization objective, the method changes sampling so RLVR advantage estimation teaches the policy efficient chains discovered by SAGE.
B. Significant Differences from Beam Search
The paper argues that step-wise SAGE differs from vanilla beam search by retaining candidates based on full-sequence confidence and accepting completed reasoning sequences directly. Experiments report comparable accuracy to token-wise TSearch with lower candidate-maintenance overhead.
- Vanilla beam search with exploration width m=4 achieves accuracy comparable to random and greedy sampling, whereas TSearch w/ Φ achieves markedly higher accuracy.
- Beam search can discard a sequence containing </think> when its full confidence score Φ is not highest, or prune it during later expansion.
- TSearch w/ Φ directly accepts a candidate when it detects </think>, avoiding the two beam-search failure modes described in the comparison.
- A reasoning step is a complete segment ending at a natural boundary or the end-of-thinking token, aligning token-wise and step-wise exploration for termination decisions.
- Random-position prefixes produce more similar candidate branches than complete-step prefixes, suggesting little diversity from token-level search inside a reasoning step.
- SAGE closely matches TSearch in accuracy while avoiding expensive token-level candidate maintenance.
- The GRPO and SAGE-GRPO objectives use group size G and token-level importance ratios and advantages in their training formulation.
- The GSPO and SAGE-GSPO objectives use group-based advantage estimation and sequence-likelihood importance ratios.
D.2. Experimental Setup
The experiments evaluate SAGE-RL across several reasoning models, curated mathematical training data, standardized training settings, and six challenging benchmarks.
- SAGE-RL is evaluated with DS-1.5B, DS-7B, DeepScaleR, and Qwen3-8B as base models.
- Training uses approximately 20,000 curated problems from English DAPO and MATH levels 3–5.
- SAGE-RL training uses verl with rule-based rewards and identical hyperparameters across each base model’s SAGE-RL and baseline runs.
- Each rollout group contains G = 8 completions, including two SAGE-generated precise chains and six default randomly sampled completions.
- Evaluation covers AIME24, AIME25, OlympiadBench, MATH-500, Minerva, and AMC23 with a maximum generation length of 32768 tokens.
E.1. Comparison with Extended Datasets and Additional Analysis
Across model scales and challenging mathematical benchmarks, SAGE-RL improves reasoning performance while reducing response length and overthinking. Sensitivity analyses indicate that exploration width matters more than increasing SAGE rollout quantity.
- Performance on DS-1.5B and DS-7B: SAGE-RL consistently achieves the best or second-best performance across six benchmarks with DS-1.5B, including pass@1 gains of 6.2% on AIME 2025 and 8.2% on AMC23 for SAGE-GSPO.
- Performance on DS-1.5B and DS-7B: With DS-7B, SAGE-GRPO improves AIME 2024 pass@1 by 2.2% over GRPO-LEAD while producing noticeably shorter responses.
- Performance on DS-1.5B and DS-7B: On distilled models, SAGE-RL alleviates overthinking while improving reasoning capability and achieving larger token-efficiency gains than competing approaches.
- Performance on DeepScaleR: SAGE-RL delivers significant token-efficiency improvements on DeepScaleR, particularly on OlympiaBench, MATH-500, and Minerva.
- Performance on Qwen3-8B: For Qwen3-8B, SAGE-GSPO raises Minerva pass@1 by 1.9% while reducing average response length to 45.7% of the original.
- Comparison of SAGE-GRPO and SAGE-GSPO: SAGE-RL improves pass@1 and reduces response length, while SAGE-GSPO particularly reduces response length and slightly exceeds SAGE-GRPO overall.
- Hyperparameter sensitivity: Increasing SAGE rollout quantity r from 1 to 2 has limited effect because additional similar trajectories provide little information and make advantage estimates less sharp.
- Hyperparameter sensitivity: Increasing exploration width m from 1 to 2 yields substantial gains, whereas limited width makes SAGE-RL behave similarly to standard GRPO.