Source-linked AI summary
Thinkless: LLM Learns When to Think
Gongfan Fang, Xinyin Ma, Xinchao Wang
TL;DR
Reasoning models are effective on complex tasks but can waste computation by using extended reasoning for straightforward queries, raising whether they can learn when to think. Thinkless trains a hybrid model with control tokens and decoupled reinforcement learning to select reasoning modes, reporting 50% - 90% less long-form reasoning usage while preserving task performance.
Problem
Applying extended reasoning to every query creates computational inefficiency, motivating the question of whether LLMs can learn when to think.
Method
Thinkless uses reinforcement learning with <short>/<think> control tokens and DeGRPO, which separates mode selection from response accuracy improvement.
Results
50% - 90% reduction in long-form reasoning usage is reported on MATH-500, Minerva Algebra, and GSM8K.
Takeaways & Limitations
Adaptive reasoning reduces inference cost and user latency while preserving task performance.
Takeaways & Limitations
The warm-up phase validates only a simple SFT approach without extensive parameter tuning, causing a slight performance drop in the initial reinforcement-learning model.
Abstract
from arXiv · showhide
Reasoning Language Models, capable of extended chain-of-thought reasoning, have demonstrated remarkable performance on tasks requiring complex logical inference. However, applying elaborate reasoning for all queries often results in substantial computational inefficiencies, particularly when many problems admit straightforward solutions. This motivates an open question: Can LLMs learn when to think? To answer this, we propose Thinkless, a learnable framework that empowers an LLM to adaptively select between short-form and long-form reasoning, based on both task complexity and the model's ability. Thinkless is trained under a reinforcement learning paradigm and employs two control tokens, <short> for concise responses and <think> for detailed reasoning. At the core of our method is a Decoupled Group Relative Policy Optimization (DeGRPO) algorithm, which decomposes the learning objective of hybrid reasoning into two components: (1) a control token loss that governs the selection of the reasoning mode, and (2) a response loss that improves the accuracy of the generated answers. This decoupled formulation enables fine-grained control over the contributions of each objective, stabilizing training and effectively preventing collapse observed in vanilla GRPO. Empirically, on several benchmarks such as Minerva Algebra, MATH-500, and GSM8K, Thinkless is able to reduce the usage of long-chain thinking by 50% - 90%, significantly improving the efficiency of Reasoning Language Models. The code is available at https://github.com/VainF/Thinkless
1 Introduction
Thinkless addresses the challenge of deciding when extended reasoning is necessary by training a hybrid model to select short- or long-form responses. Its decoupled reinforcement-learning objective stabilizes this selection and reduces unnecessary long-form reasoning while preserving task performance.
- Motivation: Reasoning on every query can be inefficient because straightforward problems incur redundant token generation and increased memory use.
- Motivation: The central question is whether an LLM can decide when to think based on task complexity and its own capability.
- Thinkless: Thinkless trains a hybrid model to select short-form or long-form responses using reinforcement learning and <think>/<short> control tokens.The control token is generated first, and training uses supervised warm-up followed by reinforcement learning.
- Thinkless: DeGRPO separates mode selection on the control token from response accuracy improvement, balancing their contributions during training.This addresses the weak and biased gradients that vanilla GRPO can give the single control token when responses are much longer.
- Results: 50% - 90% reduction in long-form reasoning usage is reported on MATH-500, Minerva Algebra, and GSM8K.On more challenging AIME tasks, the model adopts a higher proportion of long-form reasoning.
- Results: Thinkless substantially reduces inference cost while preserving task performance through adaptive reasoning decisions.
2 Related Works
Prior work improves reasoning efficiency by reducing computational overhead or dynamically adapting inference behavior. Hybrid reasoning can switch between concise and extended responses, either across multiple models or within one model.
- Efficient Reasoning Models: Reasoning models generate intermediate chain-of-thought steps that benefit complex computations, logical deduction, and multi-step reasoning.Excessively long reasoning chains create substantial computational overhead.
- Hybrid Reasoning: Hybrid reasoning dynamically adapts inference behavior to task complexity by alternating between short-form responses and long-chain reasoning.
- Hybrid Reasoning: Hybrid reasoning can be implemented through collaborative systems involving multiple models or within a single model.
3 Method
Thinkless trains a hybrid reasoning model in two stages: distillation teaches short- and long-form responses controlled by <short> and <think>, while DeGRPO separately optimizes mode selection and response accuracy. This decoupling addresses token-imbalance problems that can destabilize reasoning-mode selection.
- Overview: Thinkless combines distillation warm-up with reinforcement learning using DeGRPO to train a model that selects between short- and long-form reasoning.The model is first trained to unify two reasoning styles, then optimized to choose the appropriate mode while improving response quality.
- Distillation for Warm-up: Distillation uses reasoning and instruction-following experts to generate paired long- and short-form responses conditioned on <think> and <short>.The paired construction supports balanced generation of both response styles before reinforcement learning.
- Learning When to Think via Decoupled GRPO: The policy factorizes mode selection and response generation, with the first control token determining the reasoning style and subsequent tokens forming the answer.The control token is generated first, followed by the response sequence conditioned on that token.
- Reward Design: The reward assigns 1.0 to correct short answers, 1.0 − γ to correct long answers, and −1.0 to incorrect answers, with 1 > γ > 0.This reward explicitly prefers a correct short response over a correct long response.
- Learning When to Think via Decoupled GRPO: Vanilla GRPO can under-optimize mode selection because one control token is averaged with many response tokens, potentially causing early mode collapse.The imbalance is amplified when short and long responses differ substantially in length.
- Learning When to Think via Decoupled GRPO: DeGRPO independently normalizes control-token and response-token contributions and introduces a length-independent coefficient α to balance their optimization.This gives the control token a consistent gradient scale across short and long sequences.
4 Experiments
Thinkless experiments show that hybrid reasoning can reduce thinking usage while preserving task-sensitive behavior. DeGRPO addresses vanilla GRPO’s mode-collapse dynamics, and training analyses clarify how control-token weighting shapes learning.
- Empirical Results: 50%–90%: Learned hybrid reasoning models reduce thinking usage while distinguishing complex from simple queries.This result is reported as Finding 1 across the evaluated benchmarks.
- Empirical Results: Reasoning models generate 5 to 20 times more tokens than standard models, while extended reasoning offers no clear advantage on GSM-8K.Reasoning models outperform on challenging datasets such as AIME and MATH-500, whereas Qwen2.5-Math-1.5B-Instruct performs better on GSM-8K.
- Training Dynamics: Vanilla GRPO can collapse toward excessive preference for either long or short outputs because control-token updates are imbalanced by response length.Long-chain samples receive slower updates on <think>, while short-oriented samples can be updated disproportionately.
- Training Dynamics: DeGRPO produces a U-shaped training pattern in which long-chain outputs initially increase and later short-chain outputs rise as short-response accuracy improves.Later declines in long-chain accuracy reflect harder queries being assigned to that mode rather than degraded reasoning ability.
- Training Dynamics: 0.5 versus 0.001: A higher control-token weight makes all-correct short samples emerge earlier but can assign queries prematurely to long-chain reasoning.The paper warns that excessively rapid policy updates can reduce collaborative learning of mode selection and accuracy improvement.
- Warm-up Distillation: Warm-up distillation equips the model to generate both long- and short-form answers, with larger datasets improving performance but yielding diminishing returns.Expanding the dataset from 114K to 1M improves long-chain accuracy on Math-500 by only 1%.
5 Limitations and Future Works
The authors identify limitations in the warm-up setup and evaluation scope, while suggesting stronger hybrid-model construction and broader datasets as future directions.
- Warm-up Distillation: Warm-up uses a simple SFT approach without extensive parameter tuning, causing a slight performance drop in the initial reinforcement-learning model.
- Hybrid Model Construction: Future hybrid-model improvements include merging techniques and lightweight LoRA fine-tuning to mitigate catastrophic forgetting.
- Evaluation Scope: The algorithm is validated on DeepScaleR’s 40K mathematical problems, leaving broader datasets and domains for future evaluation.
6 Conclusion
Thinkless trains a hybrid reasoning model to choose short or long responses according to problem complexity, using Decoupled GRPO to separate mode selection from answer accuracy. The framework reduces unnecessary long-form reasoning, lowering system cost and user latency.
- Thinkless autonomously selects short responses or long-form reasoning based on input complexity.
- Decoupled GRPO separates reinforcement learning into control-token mode selection and response-token accuracy improvement.
- The decoupled objectives balance learning contributions and reduce unnecessary long-form reasoning, lowering system cost and user latency.