PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Only When You Need with Large Hybrid-Reasoning Models

Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, Furu Wei

arXiv:2505.14631v2cs.CL

TL;DR

Overthinking can expend substantial resources on simple queries with negligible gains, motivating models that decide when to think. The paper trains LHRMs with HFT and HGPO to select a thinking mode, reporting improved reasoning and general capabilities alongside improved efficiency compared with existing LRMs and LLMs.

  • Problem

    Current LRMs may expend substantial computational resources on simple queries with negligible performance gains, raising questions about their real-world use.

    Source: §1 · p. 1

  • Method

    The paper trains LHRMs through Hybrid Fine-Tuning as a cold start followed by Hybrid Group Policy Optimization to learn when to engage in thinking.

    Source: §1 · p. 3

  • Results

    Across reasoning and general capabilities, LHRMs outperform existing LRMs and LLMs while improving efficiency; Hacc exceeds HFT-DPO by 93.8%.

    Source: §1 · pp. 2–3, §3.2 · p. 7

  • Takeaways & Limitations

    The findings support adaptive thinking that reduces unnecessary reasoning on simple queries while mitigating insufficient reasoning in conventional LLMs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent Large Reasoning Models (LRMs) have shown substantially improved reasoning capabilities over traditional Large Language Models (LLMs) by incorporating extended thinking processes prior to producing final responses. However, excessively lengthy thinking introduces substantial overhead in terms of token consumption and latency, which is particularly unnecessary for simple queries. In this work, we introduce Large Hybrid-Reasoning Models (LHRMs), the first kind of model capable of adaptively determining whether to perform thinking based on the contextual information of user queries. To achieve this, we propose a two-stage training pipeline comprising Hybrid Fine-Tuning (HFT) as a cold start, followed by online reinforcement learning with the proposed Hybrid Group Policy Optimization (HGPO) to implicitly learn to select the appropriate thinking mode. Furthermore, we introduce a metric called Hybrid Accuracy to quantitatively assess the model's capability for hybrid thinking. Extensive experimental results show that LHRMs can adaptively perform hybrid thinking on queries of varying difficulty and type. It outperforms existing LRMs and LLMs in reasoning and general capabilities while significantly improving efficiency. Together, our work advocates for a reconsideration of the appropriate use of extended thinking processes and provides a solid starting point for building hybrid thinking systems.

1 Introduction

LRMs gain reasoning ability through extended thinking but can overthink simple queries, motivating systems that adapt their thinking to query needs. LHRMs use a two-stage training pipeline and are reported to improve reasoning, general capabilities, and efficiency over existing LRMs and LLMs.

1 Introduction

  • Existing LRMs’ extended thinking improves reasoning and generalization but can waste computation on simple queries with negligible performance gains, including a single-word greeting.The introduction frames the challenge as balancing system 2 reasoning with system 1 ability, drawing on the idea that complex problems call for deliberate thought while simple ones can be handled intuitively.Source: §1 · pp. 1–2
  • LHRMs are introduced as models that decide whether to use extended thinking from query semantics and context, aiming to balance computational efficiency with task complexity.The authors describe them as the first models to explicitly address existing LRMs’ limitations in this way.Source: §1 · pp. 2–3
  • The proposed two-stage pipeline uses Hybrid Fine-Tuning for a cold start and Hybrid Group Policy Optimization to learn when to think while generating more helpful and harmless responses.Hybrid Fine-Tuning teaches two thinking modes on the same query without mode collapse; Hybrid Accuracy measures hybrid-thinking capability and correlates strongly with human expert judgment.Source: §1 · pp. 2–3
  • Experiments and human studies across Qwen-2.5 models from 1.5B to 7B parameters show LHRMs adapt thinking to query difficulty and type, outperforming existing LRMs and LLMs in reasoning and general capabilities while improving efficiency.The evaluation spans mathematics, programming, and general tasks.Source: §1 · pp. 2–3

2 Large Hybrid-Reasoning Models

LHRMs select between Thinking and No-Thinking for each query to maximize expected task utility. The section introduces hybrid fine-tuning, reinforcement learning for mode selection, and Hybrid Accuracy for evaluating that capability.

2.1 Problem Formulation

  • LHRMs choose between Thinking and No-Thinking per query to maximize expected task utility across query distributions.The model learns a policy that selects the mode with the highest expected utility for the task.Source: §2.1 · p. 3

2.2 Stage I: Hybrid Fine-Tuning

  • Hybrid Fine-Tuning combines reasoning-intensive and direct-answer examples, creating a robust cold-start model from 1.7M curated, hybrid-formatted examples.The examples use think or no-think tags, and the collection removes duplicates and evaluation-benchmark overlaps.Source: §2.2 · pp. 3–4

2.3 Stage II: Hybrid Group Policy Optimization

  • HGPO teaches adaptive mode selection by sampling responses in both modes and assigning inter-group and intra-group rewards, with GRPO as its default advantage estimator.It omits a critic model, using multiple samples per prompt to estimate values while also improving helpfulness and harmlessness.Source: §2.3 · pp. 4–5, Figure 2 · p. 4

2.4 Evaluating Hybrid Thinking Capability

  • Hybrid Accuracy measures whether the model selects the reasoning mode judged most appropriate for each prompt.The ground-truth mode is based on average reward scores; if scores tie or differ by less than a threshold, the shorter response’s mode is preferred.Source: §2.4 · p. 5

3 Experimental Results

Across 1.5B and 7B scales, LHRMs outperform comparable baselines on reasoning, general, and hybrid-thinking measures. Analyses show mode selection varies with query difficulty and model capacity, and can transfer across domains.

3.2 Main Results

  • Across 1.5B and 7B scales, LHRMs consistently outperform comparable baselines, averaging 9.2% and 7.1% gains over HFT-DPO and 18.3% and 11.5% over HFT-RFT, respectively.LHRMs surpass HFT-DPO by 10.3% and 13.6% on AIME24 at 1.5B and 7B, and at 7B by 50.2% on Alpaca and 93.4% on Arena-Hard. On Hacc, LHRMs exceed HFT-DPO by 93.8% and RFT by 44.7%.Source: §3.2 · pp. 6–7
  • HFT provides a stable hybrid-reasoning cold start, and HGPO further improves task performance and mode selection.LHRMs improves Hacc over HFT by more than 31.4% at 1.5B and 110.2% at 7B; it also outperforms HFT-DPO and HFT-RFT on reasoning, general tasks, and reasoning-mode selection.Source: §3.2 · p. 7
  • LHRMs-1.5B improves on code tasks after math and general-domain RL, while HFT-DPO-1.5B and HFT-RFT-1.5B drop on both.LHRMs gains 11.1% on MBPP and 13.9% on MBPP+; the respective baselines show performance drops of 7.8% and 9.3% on those tasks.Source: §3.2 · p. 7

3.3 Analysis

  • HGPO is robust across advantage estimators, while increasing margin δ favors No-Thinking and can tune the trade-off between responsiveness and reasoning quality.The paper reports that all tested estimators yield competitive performance; higher δ may suit real-time responsiveness, while lower δ may suit applications prioritizing reasoning quality.Source: §3.3 · pp. 7–8, Figure 3 · p. 8
  • On MATH500, LHRMs-7B reduces its thinking ratio on easier queries while retaining full thinking for harder ones, reducing inference cost without sacrificing accuracy.HFT-7B maintains a high thinking ratio across difficulty levels; after HGPO, LHRMs-7B uses No-Thinking more for simpler queries and full thinking for more challenging ones.Source: Figure 4 · p. 8, §3.3 · p. 8
  • After math-domain RL, the model’s thinking ratio rises on unseen code tasks alongside performance, indicating transfer of hybrid-thinking patterns across domains.The HFT model initially has a relatively low think ratio in code, which gradually increases during RL on math and general-domain data.Source: Figure 5 · p. 8, §3.3 · p. 8
  • During RL, the 7B model’s thinking ratio decreases while the 1.5B model’s increases, linking hybrid-thinking behavior to model capacity and knowledge storage.The analysis attributes the larger model’s shift toward No-Thinking on solvable problems to stronger knowledge reserves, while the smaller model uses Thinking more often to compensate for limited capabilities.Source: §3.3 · pp. 8–9, Figure 7 · p. 9
  • Case examples show LHRMs selects No-Thinking for simple questions and switches to Thinking for complex problems requiring deeper reasoning.The paper describes simple everyday and basic math questions as examples of No-Thinking, and complex problems as examples of Thinking.Source: Figure 6 · p. 9, §3.3 · p. 9

4 Related Works

Test-time scaling enhances performance through parallel sampling or longer sequential reasoning, while large reasoning models generate extended reasoning chains and are commonly developed using reinforcement learning or distillation.

Test-Time Scaling (TTS)

  • Test-time scaling uses parallel selection methods such as Best-of-N and MCTS, or sequential longer outputs.It has also improved test-time performance in image and video generation [LWC+25] and multimodal learning [FGL+25] [HJZ+25].Source: §4 · p. 9

Large Reasoning Models

  • Large reasoning models, including DeepSeek-R1 and OpenAI o1/o3, generate extended reasoning chains with reflection before final answers.They are typically developed through reinforcement learning such as GRPO [GYZ+25] and REINFORCE++ [Hu25], or distilled from stronger models.Source: §4 · p. 9

5 Conclusion

The work introduces Hacc to assess hybrid thinking and a two-stage HFT–HGPO training pipeline. Experiments show improved reasoning and general-task performance alongside reduced unnecessary reasoning on simple queries and mitigation of insufficient reasoning in conventional LLMs.

5 Conclusion

  • The HFT–HGPO pipeline improves reasoning-centric and general downstream performance while enhancing hybrid thinking capabilities.The authors report that it reduces unnecessary reasoning on simple queries and mitigates insufficient reasoning in conventional LLMs.Source: §5 · p. 10
  • Hacc is a proposed evaluation metric for consistently assessing hybrid-thinking ability across diverse tasks.Source: §5 · p. 10
  • The proposed two-stage training pipeline consists of Hybrid Fine-Tuning (HFT) followed by Hybrid Group Policy Optimization (HGPO).Source: §5 · p. 10

A Implement Details for DPO and RFT

The DPO and RFT data pipeline reuses HGPO queries and samples responses in both thinking modes; the section also describes their training objectives and shared implementation settings.

A Implement Details for DPO and RFT

  • For each HGPO query, the pipeline samples two responses per thinking mode, yielding four responses that are scored and used to construct DPO and RFT datasets.The section states that per-mode average rewards are computed, but the supplied passages do not show the formulas or dataset selection rules.Source: A Implement Details for DPO and RFT · p. 16

DPO

  • DPO training uses a dataset of win/lose samples and optimizes a model initialized from πθHFT after obtaining DDPO.The supplied passage does not include the training objective formula.Source: A Implement Details for DPO and RFT · p. 16

RFT

  • RFT uses a constructed training dataset and has specified training objectives, although their details are not visible in the supplied passages.The passages provide no further information about the dataset contents or objective.Source: A Implement Details for DPO and RFT · p. 16

Implementation

B Implement Details for RLOO and Reinforce++

This section adapts REINFORCE++ and RLOO advantage estimators to intra-group and inter-group rewards, and describes the reward and implementation choices used in experiments.

B Implement Details for RLOO and Reinforce++

  • The experiments compare REINFORCE++ [Hu25], GRPO [SWZ+24], and RLOO, with and without HGPO, adapting the estimators to combine intra-group and inter-group rewards.For REINFORCE++, the provided formula fragment labels separate intra-group and inter-group advantages and shows an α-weighted inter-group reward term.Source: B Implement Details for RLOO and… · p. 16
  • For queries with definitive answers, rewards are rule-based; otherwise, the experiments use a trained parametric reward model.The cited reward approaches are rule-based rewards [SWZ+24] [GYZ+25] and a trained parametric reward model, respectively.Source: B Implement Details for RLOO and… · p. 16
  • The decay factor is set to 0.99 in the experiments, and the section presents separate intra-group and inter-group advantage components for RLOO.The supplied fragments identify the RLOO components but do not show their full formulas.Source: B Implement Details for RLOO and… · pp. 16–17
  • The authors describe HGPO as a general plug-in for almost any advantage estimator and implement RLOO and REINFORCE++ on VeRL [SZY+24].They state that this broadens HGPO's use cases.Source: B Implement Details for RLOO and… · p. 17

C Dataset Statistics

Stage I HFT data spans reasoning-intensive and general-purpose tasks, while Thinking and No-Thinking datasets have sharply different token-length distributions.

Stage I

  • Thinking data averaged 575 tokens (25th/75th percentiles: 362/672; maximum: 9,148), versus 4,897 tokens (1,719/6,738; maximum: 23,997) for No-Thinking data.Figure 8 presents the Stage I token-length distributions.Source: C Dataset Statistics · p. 17, Figure 8 · p. 17
  • The Stage I HFT dataset covers diverse domains, primarily mathematics, code, and general-purpose question answering.Table 2 presents the dataset's scores, distribution, and sources.Source: C Dataset Statistics · p. 17, Table 2 · p. 17

D Evaluation Settings

The evaluation settings specify inference parameters and task-specific evaluation frameworks, metrics, prompts, and judge baselines.

  • Inference uses temperature 0.6, top-p 0.95, and a maximum generation length of 35,000 tokens.Source: D Evaluation Settings · p. 17
  • Reasoning benchmarks use OpenR1 for MATH500, AIME24, GPQA Diamond, and LiveCodeBench, EvalPlus with CoT prompting for MBPP and MBPP+, and pass@1 throughout.The evaluations use OpenR1, EvalPlus [LXWZ23] [LXW+24], and CoT [WWS+22].Source: D Evaluation Settings · p. 18
  • AlpacaEval 2.0 uses GPT-4-Turbo as baseline; Arena-Hard uses Arena-Hard-v2.0 queries and GPT-4o-mini baseline, with only the final think-mode summary submitted to the judge.The final-summary restriction is used to reduce length-related bias.Source: D Evaluation Settings · p. 18

E Example Outputs

The examples show LHRMs-7B using No-Thinking for easy math problems and selecting Thinking for hard code problems.

  • LHRMs-7B uses No-Thinking mode to solve easy math problems.Source: Figure 9 · p. 18
  • LHRMs-7B learned to select Thinking mode to solve hard code problems.Source: Figure 10 · p. 19

Related papers

Loading 2505.14631v2…