Source-linked AI summary
Efficient Inference for Large Reasoning Models: A Survey
Yue Liu, Jiaying Wu, Yufei He, Ruihan Gong, Jun Xia, Liang Li, Hongcheng Gao, Hongyu Chen, Baolong Bi, Jiaheng Zhang, Zhiqi Huang, Bryan Hooi, Stan Z. Li, Keqin Li
TL;DR
LRMs improve complex reasoning but incur substantial token, memory, and inference-time costs, motivating efficient inference that preserves reasoning quality. This survey organizes and evaluates recent methods, identifies open challenges, and highlights improvement directions including model merging, new architectures, and agent routers.
Problem
LRM deliberative reasoning causes significant token consumption, memory overhead, and increased inference time, while token-efficient inference remains an underexplored deployment challenge.
Method
The survey reviews LRM-specific efficient inference methods using a taxonomy of explicit compact CoT and implicit latent CoT, alongside empirical analyses of scenarios, objectives, performance, and efficiency.
Results
The survey identifies four open challenges and highlights model merging, new architectures, and agent routers as potential techniques for further improving efficient inference.
Takeaways & Limitations
Efficient LRM inference must address user control, interpretability, safety, and broader applications while balancing reasoning efficiency with quality.
Takeaways & Limitations
Efficient reasoning methods may compromise safety alignment, increasing risks of jailbreaking and privacy leakage.
Abstract
from arXiv · showhide
Large Reasoning Models (LRMs) significantly improve the reasoning ability of Large Language Models (LLMs) by learning to reason, exhibiting promising performance in solving complex tasks. However, their deliberative reasoning process leads to inefficiencies in token usage, memory consumption, and inference time. Thus, this survey provides a review of efficient inference methods designed specifically for LRMs, focusing on mitigating token inefficiency while preserving the reasoning quality. The overview structure of this paper is shown in Figure~\ref{fig:paper_structure}. First, we introduce a taxonomy to group the recent methods into two main categories: (a) explicit compact Chain-of-Thought (CoT), which reduces tokens while keeping the explicit reasoning structure, and (b) implicit latent CoT, which encodes reasoning steps within hidden representations instead of explicit tokens. Meanwhile, we discuss their strengths and weaknesses. Then, we conduct empirical analyses on existing methods from reasoning scenarios, object functions, and performance \& efficiency aspects. Besides, we present open challenges in this field, including human-centric controllable reasoning, trade-off between interpretability and efficiency of reasoning, ensuring the safety of efficient reasoning, and broader applications of efficient reasoning. In addition, we highlight key insights for enhancing LRMs' inference efficiency via techniques such as model merging, new architectures, and agent routers. We hope this work serves as a valuable guide, helping researchers overcome challenges in this vibrant field. A collection of efficient reasoning methods for LRMs (papers and codes) is provided at this link: https://github.com/yueliu1999/Awesome-Efficient-Inference-for-LRMs.
1 INTRODUCTION
LRMs improve complex reasoning through deliberative inference, but their intermediate reasoning is costly in tokens, memory, and time. This survey organizes efficient inference methods, evaluates them, identifies challenges, and highlights directions for further improvement.
- LRMs address slow-thinking tasks by learning to conduct advanced analytical and deliberative reasoning.
- Their intermediate reasoning creates significant token consumption, high memory overhead, and increased inference time.These inefficiencies increase service inference costs and degrade user experience.
- Because general LLM efficiency methods primarily address memory and inference time, the survey focuses on LRM-specific token inefficiency while preserving reasoning quality.
- Taxonomy: The survey classifies methods into explicit compact CoT, which preserves explicit reasoning structure, and implicit latent CoT, which encodes reasoning in hidden representations.Explicit compact CoT further includes CoT compression, CoT preference optimization, and reward-based CoT conciseness.
- Survey analyses: It reviews methods hierarchically, empirically analyzes reasoning scenarios, objective functions, performance, and efficiency, and discusses their strengths and weaknesses.
- Challenges and improvements: The survey identifies challenges in user control, interpretability, safety, and broader applications, while highlighting model merging, new architectures, and agent routers as improvement directions.
2 BACKGROUND
LRMs extend LLMs with explicit reasoning processes and inference-time compute scaling, enabling difficult tasks but creating overthinking and deployment-efficiency challenges. The survey situates these challenges within the field’s recent development and examines token-efficient approaches.
- Background: LRMs extend LLMs by representing reasoning processes with intermediate tokens, supporting structured logical reasoning and complex problem-solving.
- Efficiency challenge: Scaling inference-time compute enables complex reasoning through long CoTs, but can cause overthinking with excessive tokens and marginal gains on simple problems.
- Efficiency challenge: Overthinking can negatively correlate with issue resolution rates in software engineering agents and expose retrieval-augmented systems to computationally intensive attacks.
- Efficiency challenge: Optimizing LRM token efficiency without compromising overall effectiveness remains an underexplored challenge for practical, scalable deployment.
3 LANDSCAPE OF EFFICIENT REASONING
The survey organizes efficient LRM inference around explicit compact CoT and implicit latent CoT, with explicit methods further divided by compression, fine-tuning, and reward-based strategies. These approaches reduce reasoning tokens through compression, internalization, adaptive control, and pruning, while introducing interpretability, data, or reward-design trade-offs.
- Taxonomy: Efficient LRM inference is categorized into explicit compact CoT and implicit latent CoT.Explicit compact CoT preserves an explicit reasoning structure, whereas implicit latent CoT encodes reasoning in hidden representations.
- Explicit Compact CoT: Explicit compact CoT includes CoT compression, fine-tuning for compact reasoning, and reward-based incentivization.These categories target shorter reasoning chains through compression procedures, compact reasoning data, or incentives for conciseness.
- CoT Compression: Inference-time methods adapt reasoning by early stopping, verifier-guided pruning, importance scoring, token budgets, and representation-level cache pruning.Examples include confidence-guided stopping, verifier-based exits, perplexity-based refinement, adaptive budgeting, and KV-cache pruning.
- CoT Compression: RPC achieves up to 4× memory reduction and 1.6× speedup by periodically pruning the KV cache using semantic sparsity.This illustrates compression at the memory and representation level rather than only shortening emitted reasoning.
- Takeaways: CoT compression can improve scalability but may omit essential intermediate logic, undermining transparency and interpretability.The survey also notes that supervised fine-tuning depends on curated condensed datasets and heavy preprocessing, limiting adaptability to open-ended domains.
- Implicit Latent CoT: Implicit latent CoT internalizes reasoning and can match or surpass explicit CoT accuracy while significantly reducing generation costs.The survey identifies reduced interpretability and more difficult verification as a corresponding limitation.
4 EMPIRICAL ANALYSES
The survey evaluates efficient reasoning methods across diverse reasoning scenarios, benchmark datasets, objectives, and accuracy–token-cost comparisons. Its empirical framework covers prompt-based, supervised, preference-optimization, and reinforcement-learning objectives, alongside model and deployment trade-offs.
- Reasoning Scenarios: Benchmarks span general, mathematical, causal, code, logical, symbolic, commonsense, visual, agent, and task-specific reasoning scenarios.The survey maps datasets to these scenarios to assess method applicability across diverse reasoning settings.
- Reasoning Scenarios: Mathematical benchmarks range from grade-school arithmetic and competition mathematics to graduate-level STEM reasoning.Examples include GSM8K, MATH, AIME24, OlympiadBench, and GPQA.
- Performance and Token Efficiency: The GSM8K analysis compares accuracy and token costs across multiple methods, models, and experimental settings.Results are organized in Tables 6 and 7 to support comparative evaluation of performance and efficiency.
- Performance and Token Efficiency: Distilled models retain a substantial portion of teacher-model reasoning accuracy while significantly lowering latency and token consumption.The survey identifies these models as suitable for constrained-compute or strict response-time environments.
- Objective Functions: The surveyed objectives include prompting, supervised fine-tuning, preference optimization, reinforcement learning, self-training, and inference-time intervention.Their formulations are summarized in the main text and accompanying tables.
- Objective Functions: Prompting modifies decoding without parameter updates, SFT fits curated reasoning trajectories, preference optimization ranks outputs, and RL optimizes explicit rewards.The objectives respectively use decoding constraints, cross-entropy, pairwise losses, and policy-gradient reward maximization.
5 LIMITATIONS & CHALLENGES
Efficient reasoning methods face challenges in controllability, interpretability, safety, and broader applicability. The section outlines future directions for balancing reasoning efficiency with transparency, reliability, and usability.
- 5.1 Human-centric Controllable Reasoning: Adaptive inference, external verification, and user-configurable reasoning could help balance reasoning depth, speed, transparency, and human trust.Suggested mechanisms include interactive controls, personalized reasoning profiles, symbolic reasoning, and retrieval-based justifications.
- 5.3 Ensuring Safety of Efficient Reasoning: Token-efficient methods may compromise safety alignment, increasing risks such as jailbreaking and privacy leakage.Shorter generations may not inherit the safety properties of the original long reasoning traces, and models may hide harmful intent.
- 5.4 Broader Applications: Current LRMs are concentrated in math, code, and research because these domains have clearer objectives and tolerate higher computational costs.Open-ended and time-sensitive domains remain more difficult, although efficient reasoning may improve feasibility for real-time and open-ended applications.
- 5.2 Trade-off Between Interpretability and Efficiency: Efficient reasoning methods may obscure crucial reasoning processes, creating a trade-off between token efficiency and interpretability.Implicit reasoning can be harder to understand, while explicit reasoning remains important in healthcare and legal applications.
- 5.5 Takeaways of Limitations Challenges: Future work should combine safety constraints during training with stronger reasoning-based safeguards.The survey also identifies interactive and personalized reasoning, adaptive interpretability controls, and broader applications as open directions.
6 FURTHER IMPROVEMENT
The survey proposes new architectures, model merging, and agent routing as directions for improving inference efficiency while preserving reasoning quality. Each direction offers potential gains but introduces technical or operational trade-offs.
- 6.4 Takeaways of Further Improvement: The survey identifies new architectures, model merging, and agent routing as key future directions for efficient reasoning.These approaches are presented as alternatives for further improving inference efficiency while maintaining reasoning quality.
- 6.1 New Architectures: Hybrid autoregressive-diffusion models could parallelize sequence generation, but controlling generation remains challenging.Diffusion models may optimize global reasoning structure rather than generate strictly token by token.
- 6.1 New Architectures: Linear-attention and state-space architectures could reduce memory consumption and improve inference speed, but may struggle with long-range dependencies.Their value depends on whether resource reductions preserve reasoning quality on high-fidelity tasks.
- 6.2 Model Merge: Model merging aims to combine conventional LLM responsiveness with LRM deliberative reasoning through parameter-level integration.Open questions include module selection, neuron depth, weight merging, and compatibility across architectures.
- 6.3 Agent Router: Agent routing can direct queries to specialized agents, reducing computation and resource usage while maintaining performance.Its advantage depends on accurate specialization and routing, while multiple models increase system complexity and operational costs.
7 FINAL REMARKS
The survey reviews efficient inference techniques for large reasoning models, organizing their trade-offs and practical implications. It positions inference cost as a major bottleneck as reasoning models scale.
- 7 FINAL REMARKS: The survey organizes efficient inference techniques for large reasoning models and discusses their trade-offs and practical implications.It presents the work as a foundation for further research into efficient reasoning.
APPENDIX
The appendix analyzes the mathematical objective functions used in efficient reasoning methods. This analysis is presented in Tables 9 and 10.
- APPENDIX: Tables 9 and 10 analyze mathematical objective functions in efficient reasoning methods.The appendix identifies these tables as the location of the objective-function analysis.