Source-linked AI summary
AdaptThink: Reasoning Models Can Learn When to Think
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, Juanzi Li
TL;DR
Long thinking increases inference overhead and can be excessive for simple queries, while NoThinking can match or exceed Thinking on relatively simple problems with fewer tokens. AdaptThink trains models to select between modes according to problem difficulty, reducing response length and improving accuracy across three math datasets.
Problem
Long-thinking reasoning increases inference overhead and latency, while simple queries may receive unnecessarily detailed or redundant reasoning.
Method
AdaptThink uses a constrained objective and importance sampling to encourage NoThinking while preserving performance and balancing Thinking and NoThinking during on-policy training.
Results
AdaptThink reduces DeepSeek-R1-Distill-Qwen-1.5B response length by 50.9%, 63.5%, and 44.7% and improves accuracy by 4.1%, 1.4%, and 1.6% on GSM8K, MATH500, and AIME2024, respectively.
Takeaways & Limitations
Difficulty-adaptive thinking-mode selection substantially reduces inference costs while further improving model performance.
Takeaways & Limitations
Experiments cover only 1.5B and 7B models trained on mathematical datasets, although MMLU evaluation indicates out-of-distribution generalization.
Abstract
from arXiv · showhide
Recently, large reasoning models have achieved impressive performance on various tasks by employing human-like deep thinking. However, the lengthy thinking process substantially increases inference overhead, making efficiency a critical bottleneck. In this work, we first demonstrate that NoThinking, which prompts the reasoning model to skip thinking and directly generate the final solution, is a better choice for relatively simple tasks in terms of both performance and efficiency. Motivated by this, we propose AdaptThink, a novel RL algorithm to teach reasoning models to choose the optimal thinking mode adaptively based on problem difficulty. Specifically, AdaptThink features two core components: (1) a constrained optimization objective that encourages the model to choose NoThinking while maintaining the overall performance; (2) an importance sampling strategy that balances Thinking and NoThinking samples during on-policy training, thereby enabling cold start and allowing the model to explore and exploit both thinking modes throughout the training process. Our experiments indicate that AdaptThink significantly reduces the inference costs while further enhancing performance. Notably, on three math datasets, AdaptThink reduces the average response length of DeepSeek-R1-Distill-Qwen-1.5B by 53% and improves its accuracy by 2.4%, highlighting the promise of adaptive thinking-mode selection for optimizing the balance between reasoning quality and efficiency. Our codes and models are available at https://github.com/THU-KEG/AdaptThink.
1 Introduction
Long-thinking reasoning models improve capability but impose substantial overhead, especially on simple problems. AdaptThink teaches models to select Thinking or NoThinking according to difficulty, reducing inference cost while improving performance.
- Motivation: Long chain-of-thought reasoning increases inference overhead and latency, while producing redundant computation on some simple queries.These unnecessary steps can worsen the user experience when fast responses are expected.
- Motivation: NoThinking can match or exceed Thinking on relatively simple problems while using significantly fewer tokens.Thinking benefits become pronounced only when problems are sufficiently difficult.
- AdaptThink: AdaptThink uses a constrained objective to encourage NoThinking without degrading overall performance.The method is designed to select modes according to problem difficulty.
- AdaptThink: AdaptThink uses importance sampling to balance Thinking and NoThinking samples during on-policy training.This addresses cold start and supports exploration and exploitation of both modes throughout training.
- Results: 50.9%, 63.5%, and 44.7% reductions in average response length accompany 4.1%, 1.4%, and 1.6% accuracy improvements on GSM8K, MATH500, and AIME2024, respectively.These results use DeepSeek-R1-Distill-Qwen-1.5B.
- Contributions: The paper contributes a simplified NoThinking approach, the AdaptThink RL algorithm, and experiments validating its efficacy.The stated goal is to reduce inference costs while further improving performance.
2 Related Work
Efficient reasoning work largely reduces response length, but existing methods still apply long thinking to every problem. AdaptThink instead targets difficulty-adaptive selection between Thinking and NoThinking.
- Large Reasoning Models: Large reasoning models achieve advanced problem-solving ability through long chains of thought but incur substantial inference costs and latency.This capability is typically acquired through large-scale RL with verified rewards or distilled reasoning traces.
- Efficient Reasoning for LRMs: Existing efficiency methods use length-based RL rewards, response-length control, or preference fine-tuning to reduce token usage.These approaches primarily make responses more concise.
- Efficient Reasoning for LRMs: Despite these methods, long thinking remains applied to all problems regardless of whether it is necessary.The limitation motivates a mode-selection approach rather than universal response shortening.
- NoThinking and AdaptThink: NoThinking bypasses long thinking and directly outputs the final solution, matching Thinking in low-token-budget settings and outperforming it on simple problems with sufficient budget.This observation motivates AdaptThink as a new direction for efficient reasoning.
3 Motivation
The motivation is that reasoning models need not use long thinking uniformly: NoThinking is effective for simpler problems, whereas Thinking becomes useful as difficulty increases. AdaptThink builds on this difficulty-dependent pattern.
- Thinking and NoThinking: In Thinking mode, reasoning models generate a long chain of thought involving exploration, reflection, and self-verification before the final solution.The final solution contains the correct solving steps and answer.
- Thinking and NoThinking: NoThinking bypasses long thinking by prompting the model to directly generate the final solution.The original approach uses a fake thinking process, while this work further simplifies it.
- Thinking and NoThinking: This work enforces an empty thinking segment by requiring the first generated token to be </think>.The pilot study compares this NoThinking setup with Thinking using a sufficient 16K token budget.
- Pilot Study: The pilot study evaluates accuracy, response length, and instance-level pass rate across five MATH500 difficulty levels.DeepSeek-R1-Distill-Qwen-7B generates 16 responses in each mode for every problem.
- Pilot Study: NoThinking achieves comparable accuracy to Thinking on MATH500 Levels 1–3 and even slightly outperforms it on relatively simple problems.The provided passage introduces the difficulty-based comparison but truncates the later result description.
4 AdaptThink
AdaptThink trains reasoning models to select NoThinking or Thinking according to problem difficulty. It combines a constrained objective favoring NoThinking without reducing performance with importance sampling that supports both modes during on-policy training.
- 4.1 Constrained Optimization Objective: AdaptThink maximizes NoThinking selection while constraining training to preserve overall performance.The objective uses a penalty formulation and a PPO-style policy-gradient loss with an advantage function.
- 4.1 Constrained Optimization Objective: The reward function evaluates mathematical accuracy, while an indicator identifies responses whose first token ends thinking immediately.The reference model remains fixed, and its mean reward can be estimated by presampling K responses.
- 4.2 Importance Sampling: Importance sampling addresses cold start by supplying both Thinking and NoThinking responses when the initial policy rarely generates NoThinking.Training samples are drawn from πIS rather than πθold, with half the batch in each mode.
- 4.3 A New Perspective to Understand the Loss: AdaptThink ultimately guides the policy toward difficulty-adaptive mode selection rather than a fixed thinking strategy.The stated mechanism prioritizes performance when NoThinking substantially lags behind the alternatives.
- 4.2 Importance Sampling: Importance sampling preserves exploration and exploitation across both modes, preventing the policy from permanently collapsing into one mode.The resulting loss is optimized iteratively after sampling batches and estimating the PPO-style objective.
- 4.3 A New Perspective to Understand the Loss: The loss favors NoThinking when its accuracy, plus δ, exceeds both the reference and Thinking performance; otherwise it favors Thinking.Thus, simpler problems can trigger direct final-solution generation, while challenging problems receive more Thinking.
5 Experiments
Experiments evaluate AdaptThink on math benchmarks and analyze its adaptive mode selection, hyperparameter trade-offs, training dynamics, implicit thinking, and out-of-distribution performance. AdaptThink reduces response length while improving accuracy and selects NoThinking more often for easier problems and Thinking more often for harder ones.
- 5.3 Main Results: AdaptThink reduces average response length by 53.0% for the 1.5B model and 40.1% for the 7B model, while improving average accuracy by 2.4% and 2.3%, respectively.These results compare AdaptThink with the original models across GSM8K, MATH500, and AIME 2024.
- 5.3 Main Results: AdaptThink outperforms most baselines in both accuracy and length reduction, achieving the best average results among the compared methods.The baselines optimize response length within the Thinking mode, whereas AdaptThink selects between Thinking and NoThinking.
- 5.3 Main Results: AdaptThink selects NoThinking more often on GSM8K and MATH500, uses Thinking more frequently on AIME 2024, and increases Thinking as MATH500 difficulty rises.Across MATH500 levels, it predominantly selects NoThinking for Level 1 and progressively increases Thinking for harder levels while maintaining higher accuracy across most levels.
- 5.4 More Analyses: Increasing δ raises the proportion of NoThinking responses and lowers average response length, while the accuracy gain gradually decreases.The results identify δ as a control parameter for the trade-off between reasoning efficiency and accuracy improvement.
- 5.4 More Analyses: Importance sampling lets AdaptThink learn from Thinking and NoThinking samples at every training step, reducing MATH500 response length below 2,000 tokens.Naive GRPO initially learns only from Thinking samples and later reaches around 3,500 tokens before response length increases.
- 5.4 More Analyses: On MMLU, AdaptThink reduces average response length by more than 30% while producing NoThinking responses for about 16% of problems and achieving higher accuracy than the original models.MMLU contains 14K multiple-choice questions across 57 domains distinct from the training data in format and subjects.
6 Conclusion
The paper demonstrates that NoThinking can be advantageous for relatively simple tasks and introduces AdaptThink to select Thinking or NoThinking based on problem difficulty. Experiments show reduced inference costs alongside improved model performance.
- 6 Conclusion: AdaptThink teaches reasoning models to select the optimal thinking mode based on problem difficulty.The approach is designed to improve the balance between reasoning quality and efficiency.
- 6 Conclusion: Experiments show that AdaptThink significantly reduces inference costs while further improving model performance.The conclusion highlights adaptive thinking-mode selection as a promising paradigm for balancing reasoning quality and efficiency.
7 Limitation
The authors identify limited model-scale coverage and training-data scope as limitations, while noting that MMLU results indicate OOD generalization.
- Experiments cover only 1.5B and 7B models because of limited computational resources.
- Training uses mathematical datasets because they provide accessible, accurate, and verifiable rewards.
- More training datasets with verifiable rewards for general domains could improve results.
8 Ethical Considerations
The work uses publicly published models and datasets with permissible licenses.
- All models and datasets used are publicly published with permissible licenses.
A Case Study
Case studies show AdaptThink selecting NoThinking for relatively simple problems and Thinking for a challenging AIME problem. The examples also include an out-of-distribution MMLU question, illustrating adaptive mode selection across tasks and difficulty levels.
- AdaptThink-7B selects NoThinking for a simple GSM8K problem, avoiding about 3000 thinking tokens and producing a concise solution.
- AdaptThink-7B uses NoThinking on a simple MATH500 problem, producing a solution in 297 tokens instead of about 9000 thinking tokens.
- On an out-of-distribution MMLU question, AdaptThink-7B generates a NoThinking response despite differences in question format and subject.
- For a challenging AIME 2024 problem involving a complex number with |z| = 4, AdaptThink-7B employs Thinking rather than directly generating the final solution.