PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Only When You Need with Large Hybrid-Reasoning Models

Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei

arXiv:2505.14631v2cs.CL

TL;DR

Existing LRMs gain reasoning ability from extended thinking but may impose unnecessary token and latency costs on simple queries. This paper introduces LHRMs, trained with HFT and HGPO to select thinking modes adaptively, and reports stronger reasoning and general capabilities with improved efficiency.

  • Problem

    Existing LRMs can overthink simple queries, spending computational resources with negligible performance gains and creating efficiency concerns.

  • Method

    LHRMs use Hybrid Fine-Tuning followed by Hybrid Group Policy Optimization to learn adaptive selection between thinking modes, with Hybrid Accuracy evaluating hybrid thinking.

  • Results

    LHRMs adapt hybrid thinking to query difficulty and type and outperform existing LRMs and LLMs in reasoning and general capabilities while improving efficiency.

  • Takeaways & Limitations

    The results support reducing unnecessary reasoning on simple queries while mitigating insufficient reasoning capabilities in conventional LLMs.

Abstract

from arXiv · show

Recent Large Reasoning Models (LRMs) have shown substantially improved reasoning capabilities over traditional Large Language Models (LLMs) by incorporating extended thinking processes prior to producing final responses. However, excessively lengthy thinking introduces substantial overhead in terms of token consumption and latency, which is particularly unnecessary for simple queries. In this work, we introduce Large Hybrid-Reasoning Models (LHRMs), the first kind of model capable of adaptively determining whether to perform thinking based on the contextual information of user queries. To achieve this, we propose a two-stage training pipeline comprising Hybrid Fine-Tuning (HFT) as a cold start, followed by online reinforcement learning with the proposed Hybrid Group Policy Optimization (HGPO) to implicitly learn to select the appropriate thinking mode. Furthermore, we introduce a metric called Hybrid Accuracy to quantitatively assess the model's capability for hybrid thinking. Extensive experimental results show that LHRMs can adaptively perform hybrid thinking on queries of varying difficulty and type. It outperforms existing LRMs and LLMs in reasoning and general capabilities while significantly improving efficiency. Together, our work advocates for a reconsideration of the appropriate use of extended thinking processes and provides a solid starting point for building hybrid thinking systems.

1 Introduction

Existing LRMs improve reasoning through extended thinking but can overthink simple queries, creating efficiency and user-experience problems. LHRMs address this by selecting thinking adaptively and combining hybrid training with evaluation of hybrid-thinking ability.

  • Motivation: LRMs improve reasoning and generalization through extended thinking traces, but their real-world implications remain underexplored.The supplied passages associate these improvements with programming, mathematics, and commonsense reasoning.
  • Motivation: Overthinking expends computational resources on simple queries with negligible performance gains, including trivial inputs such as “Hello”.This motivates making reasoning more efficient and better aligned with human usage requirements.
  • Approach: LHRMs dynamically decide whether to invoke extended thinking from the semantic and contextual characteristics of user queries.The model class is intended to preserve reasoning ability while improving efficiency and user experience across task complexity.
  • Approach: Hybrid Fine-Tuning provides a cold start for two thinking modes, followed by Hybrid Group Policy Optimization to learn when to engage in thinking.The pipeline is designed to avoid mode collapse and simultaneously improve helpfulness and harmlessness.
  • Evaluation: Hybrid Accuracy is introduced to quantitatively evaluate a model’s capability for hybrid thinking.The metric is reported as correlating strongly with human expert judgment.
  • Results: LHRMs outperform existing LRMs and LLMs in reasoning and general capabilities while significantly improving efficiency across diverse model scales and domains.The experiments cover Qwen-2.5 models from 1.5B to 7B parameters and include mathematics, programming, and general tasks.

2 Large Hybrid-Reasoning Models

LHRMs combine Thinking and No-Thinking modes and learn to select between them according to each query. The approach uses Hybrid Fine-Tuning followed by HGPO, with Hybrid Accuracy evaluating mode selection.

  • Problem formulation: LHRMs select between Thinking and No-Thinking modes to maximize query-specific utility.The model learns a policy that maps queries to the reasoning mode expected to produce the greatest utility.
  • Training pipeline: The two-stage training pipeline uses Hybrid Fine-Tuning for initialization and HGPO to learn adaptive mode selection.HFT combines reasoning-intensive and direct-answer data, while HGPO applies online reinforcement learning after the cold start.
  • Stage I: Hybrid Fine-Tuning: Hybrid Fine-Tuning integrates reasoning-intensive and direct-answer examples to stabilize cold-start training and support both response styles.The resulting dataset contains 1.7M hybrid-formatted training examples after deduplication and benchmark-overlap removal.
  • Stage II: Hybrid Group Policy Optimization: HGPO samples outputs under both modes, scores them, assigns inter-group and intra-group rewards, estimates advantages, and updates the policy.The reward design captures relative quality across modes and answer quality within each mode; GRPO is the default advantage estimator.
  • Stage II: Hybrid Group Policy Optimization: The margin δ controls the trade-off between reasoning modes, with larger values encouraging No-Thinking behavior.Thus, δ can adjust the balance between responsiveness and reasoning quality for different application needs.
  • Evaluation: Hybrid Accuracy measures whether LHRMs correctly select the preferred reasoning pattern for each prompt.The preferred mode is determined from average reward, with shorter responses selected when the mode scores are equal or sufficiently close.

3 Experimental Results

Across benchmarks and analyses, LHRMs outperform comparable baselines while adaptively selecting Thinking or No-Thinking modes according to query difficulty, domain, and model capacity.

  • Overall Performance: LHRMs outperform comparable baselines across reasoning and general benchmarks at both 1.5B and 7B scales.They achieve average improvements of 9.2% and 7.1% over HFT-DPO, and 18.3% and 11.5% over HFT-RFT, respectively.
  • Overall Performance: On 7B models, LHRMs exceed HFT-DPO by 50.2% on Alpaca and 93.4% on Arena-Hard.AIME24 and Arena-Hard are identified as the most challenging math and general-domain benchmarks, respectively.
  • Overall Performance: LHRMs achieve the strongest hybrid-thinking performance, with Hacc improvements over HFT-DPO and RFT of 93.8% and 44.7%.The results are presented as evidence that HGPO enables correct hybrid-thinking behavior.
  • Training Ablations: HFT improves reasoning and general capabilities while providing stable cold-start hybrid reasoning without failure or collapse.This comparison is made against the Qwen2.5 and DeepSeek-R1-Distill series.
  • Training Ablations: HGPO further improves reasoning, general capabilities, and mode selection, producing Hacc gains above 31.4% at 1.5B and 110.2% at 7B.Comparisons with HFT-DPO and HFT-RFT also show stronger downstream performance and reasoning-mode selection accuracy.
  • Cross-Domain and Behavioral Analysis: LHRMs generalize hybrid-thinking patterns across domains: 1.5B models gain 11.1% on MBPP and 13.9% on MBPP+, unlike DPO and RFT baselines.RL uses math and general-domain data, while the reported code-task gains occur outside those training domains.
  • Cross-Domain and Behavioral Analysis: Larger models increasingly favor No-Thinking on solvable problems, whereas smaller models invoke Thinking more often to compensate for limited capabilities.During RL, the 7B think ratio decreases while the 1.5B think ratio increases on MATH500.
  • Cross-Domain and Behavioral Analysis: LHRMs select No-Thinking for simple queries and Thinking for complex problems, combining faster responses with deeper reasoning where needed.The reported behavior covers everyday questions, basic mathematics, and more demanding reasoning problems.

4 Related Works

The related work reviews test-time scaling for enhancing model performance and the development of Large Reasoning Models through reinforcement learning or distillation. It distinguishes parallel and sequential test-time scaling approaches.

  • Test-Time Scaling: Test-time scaling enhances model performance beyond scaling model size.The passage identifies test-time scaling as a validated approach for improving model performance.
  • Test-Time Scaling: Parallel test-time scaling samples multiple solutions and selects one using a selector such as an outcome reward model.Best-of-N and Monte-Carlo Tree Search are cited as examples.
  • Large Reasoning Models: Large Reasoning Models generate long reasoning chains with reflection before producing final answers.This distinguishes LRMs from general LLMs in the related-work discussion.
  • Large Reasoning Models: LRMs are commonly developed using reinforcement learning methods such as GRPO and REINFORCE++, or distilled from stronger models.The passage lists both reinforcement-learning and distillation-based development routes.

5 Conclusion

The paper proposes Hacc and a two-stage HFT–HGPO pipeline to balance reasoning and general-purpose performance. Experiments indicate improved reasoning, downstream-task performance, and hybrid thinking efficiency.

  • Contributions: Hacc is introduced to evaluate hybrid-thinking ability consistently across diverse tasks.The metric is designed to assess whether models appropriately perform hybrid thinking.
  • Contributions: The proposed training pipeline combines Hybrid Fine-Tuning and Hybrid Group Policy Optimization.The two stages are presented as the core training approach.
  • Results: The pipeline improves performance on both reasoning-centric and general downstream tasks.The conclusion reports gains across both capability categories.
  • Implications: Hybrid thinking reduces unnecessary reasoning on simple queries while mitigating insufficient reasoning in conventional LLMs.The reported benefit addresses inefficiency in LRMs and limited reasoning in conventional LLMs.

A Implement Details for DPO and RFT

The DPO and RFT implementation constructs training data from responses generated in both thinking modes, scores those responses, and optimizes models initialized from HFT. Implementations match HGPO training settings for fair comparison.

  • Data Construction: For each query, two responses are sampled in each of the two thinking modes, producing four responses per query.The modes are used to construct offline-sampling data for DPO and RFT.
  • Data Construction: The reward function Rϕ4 scores the sampled responses before subsequent training-data construction.The passage places reward scoring after sampling.
  • DPO: DPO constructs win–lose training pairs from the scored responses.The DPO dataset is described as containing winning and losing samples.
  • DPO: DPO optimizes πθ initialized from πθHFT using its training objective.The initialization comes from the HFT model.
  • RFT: RFT constructs its own training dataset and applies the stated RFT training objectives.The passages separately identify RFT data construction and optimization objectives.
  • Implementation: DPO and RFT use LLaMA-Factory while matching HGPO's learning rate, batch size, and total training-sample count.These settings are maintained to ensure a fair comparison.

B Implement Details for RLOO and Reinforce++

The implementation adapts REINFORCE++ and RLOO to HGPO's intra-group and inter-group rewards, using rule-based or parametric rewards according to query type. The resulting framework supports multiple advantage estimators.

  • REINFORCE++: REINFORCE++ is adapted to combine intra-group and inter-group advantages within HGPO.The formulation labels separate REINFORCE++ terms for Aintra and Ainter.
  • Reward Design: Rule-based rewards are used for queries with definitive answers, while other queries use a trained parametric reward model.This reward choice applies to the implementation of the compared estimators.
  • RLOO: The decay-factor hyperparameter is set to 0.99 in the experiments.The passage identifies this value for the RLOO implementation.
  • RLOO: RLOO is adapted with separate intra-group and inter-group advantage terms.The implementation labels both RLOO components explicitly.
  • Generality: HGPO functions as a general plug-in for almost any advantage estimator, extending its use cases.The implementation includes RLOO and REINFORCE++ on VeRL.

C Dataset Statistics

The paper describes datasets for the two training stages, covering reasoning-intensive and general-purpose tasks. Stage I includes distinct Thinking and No-Thinking data distributions with substantially different token lengths.

  • Thinking data averages 575 tokens, while No-Thinking data averages 4,897 tokens.The 25th–75th percentile ranges are 362–672 and 1,719–6,738 tokens, respectively.
  • Thinking data reaches 9,148 tokens at maximum, compared with 23,997 tokens for No-Thinking data.
  • Stage I HFT data covers mathematics, code, and general-purpose question answering across diverse domains.
  • Figure 8 visualizes token length distributions for the Thinking and No-Thinking datasets constructed in Stage I.

D Evaluation Settings

The evaluation uses established frameworks and benchmark-specific baselines across reasoning and general-purpose tasks. Inference employs fixed sampling and generation settings, with evaluation tools summarized in Table 4.

  • Table 3 describes Stage II data distributions and sources, noting that experiments use subsets rather than full dataset versions.
  • Inference uses temperature 0.6, top-p 0.95, and a maximum generation length of 35,000 tokens.
  • OpenR1 evaluates MATH500, AIME24, GPQA Diamond, and LiveCodeBench, while EvalPlus evaluates MBPP and MBPP+ with Chain-of-Thought prompting.
  • Pass@1 is reported for all reasoning-related benchmarks.
  • GPT-4-Turbo is the AlpacaEval 2.0 baseline, and GPT-4o-mini is the Arena-Hard baseline.
  • Table 4 summarizes the evaluation tools used in the experiments.

E Example Outputs

The examples show LHRMs using No-Thinking for an easy arithmetic problem and an easy array problem, while Figure 9 describes No-Thinking use on easy math problems. Figure 10 describes Thinking use for hard code problems.

  • The arithmetic example verifies the answer by direct calculation, with both methods yielding 26000.
  • LHRMs-7B uses No-Thinking to solve the difference-of-squares arithmetic example.The displayed calculation obtains 513 − 487 = 26, and the solution then multiplies by 1000 to reach 26000.
  • Figure 9 describes LHRMs-7B using No-Thinking for easy math problems.
  • HFT-7B applies the No-Thinking mode to an array problem involving the power of fixed-size subarrays.
  • For the array task, a subarray’s power is its maximum element only when elements are consecutive and ascending; otherwise it is -1.
  • Figure 10 describes LHRMs-7B selecting Thinking mode to solve hard code problems.
Loading 2505.14631v2…