PaperlayerView exact revision on arXiv

Source-linked AI summary

From System 1 to System 2: A Survey of Reasoning Large Language Models

Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhiwei Li, Bao-Long Bi, Ling-Rui Mei, Junfeng Fang, Xiao Liang, Zhijiang Guo, Le Song, Cheng-Lin Liu

arXiv:2502.17419v6cs.AI

TL;DR

Foundational LLMs are fast and capable but remain limited on complex, step-by-step reasoning. This survey synthesizes the foundations, construction methods, characteristics, benchmarks, evolution, and future directions of reasoning LLMs, which show strong performance across text and multimodal tasks, especially mathematics and coding. It also identifies unresolved issues in balancing deliberate and fast thinking and in explaining how reasoning capabilities arise.

  • Problem

    Foundational LLMs struggle with deep logical analysis, while existing studies lack systematic organization across reasoning methods, training approaches, and slow-thinking techniques.

  • Method

    The survey synthesizes foundational LLMs, early System 2 technologies, reasoning-LLM features and construction methods, evolutionary stages, benchmarks, performance comparisons, and future directions.

  • Results

    Reasoning LLMs show strong performance across text and multimodal benchmarks, particularly in mathematics and coding, where they outperform foundational LLMs by a large margin.

  • Takeaways & Limitations

    The survey provides a comprehensive reference for reasoning-LLM concepts, methods, challenges, benchmark evidence, and ongoing developments.

  • Takeaways & Limitations

    Current reasoning LLMs struggle to balance fast and slow thinking, while LongCoT mechanisms and reinforcement-learning origins remain unclear, especially in multimodal settings.

Abstract

from arXiv · show

Achieving human-level intelligence requires refining the transition from the fast, intuitive System 1 to the slower, more deliberate System 2 reasoning. While System 1 excels in quick, heuristic decisions, System 2 relies on logical reasoning for more accurate judgments and reduced biases. Foundational Large Language Models (LLMs) excel at fast decision-making but lack the depth for complex reasoning, as they have not yet fully embraced the step-by-step analysis characteristic of true System 2 thinking. Recently, reasoning LLMs like OpenAI's o1/o3 and DeepSeek's R1 have demonstrated expert-level performance in fields such as mathematics and coding, closely mimicking the deliberate reasoning of System 2 and showcasing human-like cognitive abilities. This survey begins with a brief overview of the progress in foundational LLMs and the early development of System 2 technologies, exploring how their combination has paved the way for reasoning LLMs. Next, we discuss how to construct reasoning LLMs, analyzing their features, the core methods enabling advanced reasoning, and the evolution of various reasoning LLMs. Additionally, we provide an overview of reasoning benchmarks, offering an in-depth comparison of the performance of representative reasoning LLMs. Finally, we explore promising directions for advancing reasoning LLMs and maintain a real-time \href{https://github.com/zzli2022/Awesome-Slow-Reason-System}{GitHub Repository} to track the latest developments. We hope this survey will serve as a valuable resource to inspire innovation and drive progress in this rapidly evolving field.

1 INTRODUCTION

Reasoning LLMs address foundational LLMs’ weakness in deep, step-by-step analysis by emulating slower System 2 reasoning. The survey organizes prior foundations, construction methods, model evolution, benchmarks, and future directions.

  • Foundational LLMs excel at rapid, heuristic-driven responses but struggle with deep logical analysis and precision on complex reasoning tasks.
  • Reasoning LLMs use step-by-step processing to tackle advanced mathematics, logical reasoning, and multimodal reasoning with expert-level performance.
  • The survey reviews foundational LLMs and early System 2 technologies, then analyzes reasoning-LLM features, construction methods, evolution, benchmarks, and future directions.
  • Existing work includes specific technical analyses and replications, but lacks systematic organization across slow-thinking methods, training, and reinforcement-learning perspectives.
  • The survey contributes a comprehensive, multi-approach account of reasoning-LLM concepts, challenges, and construction methods beyond reinforcement learning.

2 FOUNDATIONS OF REASONING LLMS

Reasoning LLMs build on foundational language and multimodal models together with symbolic logic, search, and reinforcement learning. These foundations provide structured reasoning, planning, and reward-driven optimization while retaining broad learned knowledge.

  • Foundational LLMs evolved from pretrained transformers to multimodal systems, improving language understanding, text generation, image processing, and human-computer interaction.
  • Despite broad capabilities, foundational LLMs resemble System 1 reasoning and lack the step-by-step analysis characteristic of System 2.
  • Symbolic logic provides precise rule-based knowledge representation and inference in structured domains, while exposing limitations associated with rigid systems.
  • Macro actions combine symbolic templates or rules with foundational LLMs, enabling hierarchical planning before detailed problem solving.
  • MCTS searches reasoning trees through selection, expansion, simulation, and backpropagation to identify promising paths using estimated rewards.
  • Reinforcement learning trains agents through environmental interaction and rewards, and its self-play lineage supports self-improvement and reasoning-LLM optimization.

3 BLUEPRINTING REASONING LLMS

The survey blueprints reasoning LLMs by examining their behavioral and training characteristics, core enabling methods, and evolutionary stages, while comparing them with traditional reasoning models.

  • The survey analyzes reasoning-LLM features from output behavior and training dynamics perspectives before reviewing construction methods and evolutionary stages.

3.1 Analysis of the Features of Reasoning LLMs

Reasoning LLMs exhibit exploratory, longer-form problem solving and distinctive training dynamics. Their development also emphasizes hard-sample data, sparse training, and larger-model scalability.

  • Output Behavior: Reasoning LLMs explore novel hypotheses and alternative solution paths more than conventional Chain-of-Thought models.
  • Output Behavior: Reasoning LLMs often generate outputs exceeding 2000 tokens for complex coding and mathematics problems, though this can produce overthinking.
  • Training Dynamics: Hard-sample Slow-thinking CoT datasets can improve generalization in medicine and mathematics compared with uniformly difficult instruction sets.
  • Training Dynamics: LongCoT training produces relatively uniform gradient norms across layers, unlike simplified CoT training’s larger and more variable early-layer gradients.
  • Training Dynamics: Models exceeding 30 billion parameters are more compatible with reasoning-LLM training, while data-scaling effects are stronger in larger models.

3.2 Core Method

The survey presents core methods for building reasoning LLMs, including structure search, reward modeling, self-improvement, macro actions, and reinforcement fine-tuning. These methods support iterative reasoning, step-level evaluation, scalable training, and improved performance, while introducing computational, trustworthiness, stability, and interpretability challenges.

  • Structure Search: Structure search uses MCTS to construct reasoning trees, simulate future states, backpropagate rewards, and identify high-reward reasoning paths.Applications include code generation, mathematical reasoning, instructional alignment, and response refinement.
  • Structure Search: MCTS extends single-pass CoT and Tree-of-Thought methods by revisiting, critiquing, and dynamically refining multiple reasoning paths.This supports long-term planning and tasks with large decision spaces.
  • Reward Modeling: Outcome supervision evaluates final-answer correctness through ORM, whereas process supervision evaluates each reasoning step through PRM.PRM provides fine-grained error identification and supports reinforcement learning, automated correction, and interpretable reasoning.
  • Reward Modeling: PRMs remain constrained by insufficient explanations, biased training data, and early-step bias, limiting reliable identification of erroneous reasoning steps.These trustworthiness challenges affect interpretability and error detection during test-time reasoning.
  • Reinforcement Fine-Tuning: Reinforcement fine-tuning improves domain-specific reasoning with minimal high-quality data, reward models, and stable long-context optimization.DeepSeek-R1 illustrates simplified RL supervision, scalable online training, and emergent Long-CoT capabilities compared with traditional approaches.
  • Reinforcement Fine-Tuning: RFT still faces unclear reasoning mechanisms, reward saturation, exploration collapse, and unstable long-CoT generation, including context overflow and failure to return final answers.Future work targets stable RL, scalable prompt and reward data, adaptive length control, hierarchical RL, and clearer mechanisms.

3.3 Evolutionary of Reasoning LLMs

Reasoning LLMs evolved from externally orchestrated search toward internalized reasoning and RL-scaled behaviors. This progression expanded exploration, extended reasoning chains, and enabled reflection and planning.

  • Early methods augmented frozen LLMs with external search algorithms, representing reasoning as tree or graph traversal.
  • Externalized approaches were constrained by narrow exploration spaces and limited sharing of experience across reasoning paths.
  • Richer action spaces broadened exploration and enabled more comprehensive structured search, but required careful design of the action space.
  • Models such as o1 and QwQ internalized exploratory planning and alternative-path exploration within the LLM context, producing extended reasoning chains.
  • RL-based models such as DeepSeek-R1 and Kimi-k1.5 exhibited long reasoning chains, reflective reasoning, and advanced planning through simple RL scaling.

4 BENCHMARKING REASONING LLMS

The survey organizes reasoning benchmarks by task, metrics, and model comparisons, covering text and multimodal reasoning. Results show strong gains in math and coding, while benchmark saturation, efficiency, leakage, and multimodal evaluation remain concerns.

  • Benchmark categories: Reasoning benchmarks span math, code, scientific, agent, medical, and multimodal task categories.
  • Benchmark limitations: Simple benchmarks such as GSM8K, MATH-500, and ScienceQA are approaching saturation, motivating harder soft-reasoning benchmarks and stronger leakage controls.
  • Evaluation metrics: Pass@k and Cons@k evaluate mathematical success and consistency, while code tasks use metrics including exact match and Accuracy.
  • Evaluation metrics: Outcome Efficiency and Process Efficiency assess whether later reasoning improves accuracy and solution diversity, exposing overthinking on simple problems.
  • Benchmark limitations: Existing evaluation frameworks focus mainly on final answers and inadequately measure the computational and temporal efficiency of reasoning processes.
  • Performance comparisons: Reasoning LLMs outperform foundational LLMs broadly, especially on math and coding benchmarks; OpenAI-o1 exceeds GPT-4o by 69.9% and 73% on the cited tasks.
  • Performance comparisons: OpenAI-o1 achieves 78.2% on MMMU and exceeds GPT-4o by 7.2% on MathVista, although multimodal gains are less pronounced than text-only gains.

5 EXTENDED TECHNIQUES

Extended techniques describe RLHF infrastructure for reasoning and agentic systems, from rollout and preprocessing through optimization. The section also highlights scalability trade-offs, implementation complexity, and challenges in long-text and networked environments.

  • RL infrastructure: RLHF workflows combine generative rollouts, auxiliary-information preprocessing, and iterative actor-critic optimization.
  • RL infrastructure: High-throughput engines support rollout, while distributed frameworks use data, pipeline, and tensor parallelism to improve training scalability and efficiency.
  • Architectural paradigms: Single-controller systems incur dispatch overhead at scale, whereas multi-controller systems offer scalability and lower scheduling overhead.
  • Architectural paradigms: Multi-controller RLHF requires tightly coordinated communication, computation, and data-transfer components, creating substantial development, maintenance, and optimization challenges.
  • Prospective challenges: Long-text generation causes GPU under-utilization through variable response lengths, while agentic RL adds adaptive interaction with increasingly complex environments.
  • Prospective challenges: Agentic RL depends on reliable distributed HTTP environments, and standardized protocols such as MCP can reduce bespoke interface engineering and improve interoperability.

5.2 Trustworthiness of Reasoning LLMs

Trustworthiness in reasoning LLMs requires evaluating reasoning traces, resolving knowledge conflicts, addressing safety vulnerabilities, and grounding outputs in evidence. Their explicit multi-step reasoning and tool use create distinctive transparency and safety challenges.

  • Trust challenges: Reasoning LLMs create trust challenges because their multi-step traces, external tools, and complex decisions complicate assessment of genuine reasoning.
  • Interpretability: Faithfulness metrics and mechanistic analyses assess whether reasoning explanations align with model behavior and how reasoning is encoded.
  • Knowledge conflicts: Knowledge conflicts between model memory and external context can undermine reasoning accuracy, motivating detection, resolution, and model-editing methods.
  • Safety and robustness: Safety alignment can reduce reasoning performance, while prompt attacks, input slowdown, and backdoor jailbreaks expose vulnerabilities in reasoning flows.
  • Grounding and truthfulness: External evidence grounding is presented as a way to mitigate hallucination and improve truthfulness in reasoning.

5.3 Reasoning LLM as Agent

Reasoning LLM agents are evolving from passive responders into systems that search, use tools, learn interaction strategies, and undergo more realistic evaluation.

  • Agent capabilities: Reasoning LLM agents increasingly interact with external environments by seeking information, using diverse tools, and dynamically refining their reasoning.Key developments include search augmentation, broader tool integration, new training methods, and evolving evaluation frameworks.
  • Search augmentation: Reinforcement learning trains LLMs to generate relevant search queries and invoke search proactively during reasoning.These approaches optimize search interaction based on outcomes rather than predefined processes or distillation.
  • Tool integration: Self-learning frameworks teach models when and how to use external tools such as code interpreters and specialized solvers.Hint-based learning and rejection-sampling fine-tuning support acquisition of tool-using skills.
  • Training methodologies: Reinforcement learning improves reasoning in smaller LLMs, while bootstrapping reduces reliance on advanced teacher models and extensive human annotations.Together, these methods suggest more scalable routes to training agentic reasoning capabilities.
  • Evaluation: New benchmarks and interactive arenas evaluate multi-step reasoning and tool use through systematic comparisons, user-defined tasks, and community feedback.These settings broaden evaluation beyond static task performance.

5.4 Efficient Reasoning LLM

Efficient reasoning research targets the high computational cost of long thought chains through budget control, shorter or hidden reasoning, non-autoregressive models, and efficient architectures.

  • Motivation: Long autoregressive reasoning imposes substantial computational burdens, especially for agent-based and edge applications.Models may also produce redundant inference in their thought chains.
  • Budget control: Reasoning-budget-sensitive methods constrain inference cost through explicit budgets or length penalties.Examples include CoD, TALE-EP, L1, TOPS, O1-Pruner, and Kimi-k1.5.
  • Reasoning representations: Hidden-space and mixed reasoning methods seek more efficient strategies while balancing interpretability with efficiency.ICoT-KD, CCoT, Token Assorted, and Heima represent variants across textual, hidden-space, and multimodal reasoning.
  • Architectural efficiency: Non-autoregressive diffusion approaches reduce autoregressive prediction steps, while linear-attention and state-space architectures address ultra-long-context costs.Diffusion-LM, LLaDA, Diffusion-of-Thoughts, Mamba, and RWKV exemplify these directions.

6 CHALLENGES & FUTURE DIRECTIONS

Reasoning LLMs face efficiency, adaptability, interpretability, language-coverage, and safety challenges; the survey proposes tools, adaptive switching, neural-symbolic integration, and broader language support as directions forward.

  • 6.1 Efficient Reasoning LLMs: Extended autoregressive inference creates high latency, with some Codeforces problems requiring over 10,000 reasoning tokens.Smaller models also struggle to match larger models on slow-thinking tasks, while correct solutions may undergo lengthy verification.
  • 6.1 Efficient Reasoning LLMs: External reasoning tools could support early stopping and verification, while research also seeks slow-thinking abilities in smaller language models.These directions target both inference efficiency and model-size constraints.
  • 6.2 Collaborative Slow & Fast-thinking Systems: Current reasoning LLMs may apply unnecessary deep reasoning to simple tasks because they do not reliably switch between fast and slow thinking.Adaptive switching, joint training, and coevolution are proposed to harmonize efficiency with precision.
  • 6.3 Reasoning LLM for Scientific Research: Reasoning LLMs support structured analysis in scientific domains such as medicine and mathematics, including differential diagnosis and treatment planning.Inference-time scaling is identified as a way to enhance step-by-step reasoning in medical applications.
  • 6.4 Deep Integration of Neural and Symbolic Systems: Neural-symbolic integration combines adaptability with interpretability and can outperform reasoning-only systems on geometry problems.AlphaGeometry2 solved 84% of IMO geometry problems from 2000–2024, while OpenAI-o1 solved none.
  • 6.5 Multilingual Reasoning: Reasoning LLMs remain limited in low-resource languages because of data sparsity, stability, safety, and performance challenges.Suggested responses include parameter sharing and incremental injection of domain-specific knowledge.
  • 6.6 Safety and Control: Rapidly advancing reasoning models raise safety and control concerns, including reward hacking, generalization failures, and language mixing under reinforcement learning.The survey emphasizes safe, responsible, and transparent use alongside methods for controlling model actions.

7 CONCLUSION

The survey synthesizes the development, construction, capabilities, methods, benchmarks, and future directions of reasoning LLMs to organize progress in this rapidly evolving field.

  • Survey scope: The survey connects foundational LLMs with symbolic logic, MCTS, and reinforcement learning to explain the path toward reasoning LLMs.It also analyzes recent models, their features, enabling methods, and representative evolution.
  • Contribution: The paper provides a structured resource for understanding reasoning LLM techniques and ongoing research directions.Its coverage includes development, evaluation, and future advancement of the field.
Loading 2502.17419v6…