From System 1 to System 2: A Survey of Reasoning Large Language Models

1 INTRODUCTION

• Foundational LLMs such as GPT-4o and DeepSeek-v3 [17] provide rapid responses but often lack deep logical analysis and precision on complex reasoning tasks.The paper defines reasoning as answering questions involving complex, multi-step processes with intermediate steps; foundational models handle simple or single-step tasks.Source: §1 · pp. 1–2
• Reasoning LLMs such as OpenAI’s o1/o3, and DeepSeek’s R1 [31] use step-by-step processing to tackle complex math, logic, and multimodal tasks at expert level.The survey characterizes this as emulating the slower, more deliberate System 2 mode.Source: §1 · p. 2

1.1 Structure of the Survey

• The survey spans foundational LLMs and early System 2 technologies, reasoning-LLM construction and evaluation, related technical areas, future directions, and a real-time repository.Its coverage includes symbolic logic, MCTS, and RL; reasoning-model characteristics, construction methods, and evolution; benchmarks, metrics, and performance comparisons; large-scale RL, safety, agents, adaptive and efficient reasoning; and future directions.Source: §1.1 · p. 2

1.2 Contribution of the Survey

• The survey addresses fragmented prior analyses by offering a comprehensive account of reasoning LLM concepts, methods, and challenges, including early System 2 foundations and construction methods beyond RL.Earlier work examined specific approaches, test-time slow thinking, or reasoning-model training, often from an RL perspective.Source: §1.2 · p. 2

2.1 Foundational LLMs

• Foundational LLMs evolved from pretrained Transformers to large multimodal models, gaining broad language and reasoning capabilities while still relying largely on fast System 1 heuristics.BERT emphasized bidirectional context, while GPT focused on generation; GPT-3 showed few-shot strength, and DeepSeek-V3 [17] offered benchmark performance with efficiency and speed improvements. These models also exhibit in-context learning, prompt engineering, and Chain-of-Thought reasoning.Source: §2.1 · pp. 2–4

2.2 Symbolic Logic Systems

• Symbolic logic uses formal rules to reason precisely in structured domains, but its rigid rule-based systems are limited; symbolic templates also inform hierarchical macro actions in reasoning LLMs.Prolog – exemplifies logic programming with facts, rules, and queries, while macro actions combine symbolic templates or rules with LLMs to plan at a high level before addressing details.Source: §2.2 · p. 4

2.3 Monte Carlo Tree Search

• MCTS searches decision trees through selection, expansion, simulation, and backpropagation, helping reasoning LLMs explore promising future paths and avoid suboptimal reasoning trajectories.Selection uses UCB1 to balance exploration and exploitation; the method informs Structural Search and supports strategic or multi-step reasoning.Source: §2.3 · pp. 4–5

2.4 Reinforcement Learning

• RL uses reward feedback to optimize decisions, and its deep-RL advances and techniques inform self-improvement, stepwise supervision, and direct training of reasoning LLMs.Q-learning and DQNs enabled learning in complex state spaces; AlphaZero and AlphaStar demonstrated planning in strategic and dynamic environments. Reward shaping motivates process reward models, while RL directly guides reasoning-LLM training, as in DeepSeek-R1 [31].Source: §2.4 · pp. 5–6

• The section analyzes reasoning LLM features from the perspectives of output behavior and training dynamics.Source: §3 · p. 6
• It surveys the core methods that enable reasoning LLMs’ advanced reasoning capabilities.Source: §3 · p. 6
• It summarizes the evolution of reasoning LLMs and identifies Figure 3 as a comparison with traditional reasoning models.Source: §3 · p. 6

3.1.1 Output Behaviour Perspective

• Reasoning LLMs explore novel hypotheses and alternative paths, unlike conventional CoT approaches such as WizardMath [109] and DeepSeekMath [110].Studies describe latent generation during next-token prediction [49], behaviors emerging during RL scaling [31], and Quiet-STaR’s auxiliary next-token pre-training [111].Source: §3.1.1 · p. 6
• OpenAI’s o1 and o3 combine long-term planning with micro-actions for verification, a pattern reflected in Marco-o1 [112] and Huatuo-o1 [113].Marco-o1 assigns MCTS tree nodes a reflective “Wait!” state, while Huatuo-o1 uses “Backtracking” and “Correction” prompts to address incorrect CoT during validation.Source: §3.1.1 · p. 6
• Over 2,000-token outputs can support complex coding and mathematics, but may cause overthinking without improving solutions.Slow-thinking is described as useful for multimodal observation and comparison [116], [117], external knowledge or conflicts [118], and complex medical problems using test-time scaling [52].Source: §3.1.1 · p. 6
• Reasoning LLMs perform strongly in mathematics, coding, medical QA, and translation, yet may generate unnecessary solution rounds or uncertainty on simple questions such as “2+3=?” [124].The passage identifies this as a potential inefficiency, contrasting unnecessary paths on easy math problems with limited exploratory actions for simpler questions.Source: §3.1.1 · p. 6

3.1.2 Training Dynamic Perspective

• 5,000 distilled samples for STILL2 [51], 17,000 LongCoT samples for Sky-T1, and 4,000 core LongCoT samples for RedStar [54] illustrate sparse-data training.Hard-sample-focused Slow-thinking CoT datasets improve generalization in medicine and mathematics [52], [54]; such SFT data can achieve comparable results with 1/100th the sample size of simple CoT, while non-dense or rule-based RL rewards may suffice.Source: §3.1.2 · p. 6
• 30-billion-plus models are described as more compatible with reasoning training, while LongCoT yields more uniform layerwise gradient norms than simplified CoT.RedStar [54] reports stronger data-scaling effects in larger models, and Deepseek-R1 [31] reports that a 670-billion-parameter model approaches o1 benchmark performance.Source: §3.1.2 · p. 7

3.2.1 Structure Search

• MCTS searches reasoning trees by simulating future states and backpropagating rewards, enabling iterative refinement across complex tasks and specialized domains.RAP adds a world model, while Forest-of-Thought explores multiple trajectories; applications include code, mathematics, alignment, and medical reasoning. Actions range from reasoning steps and tokens to task-specific structures and self-correction; rewards can be outcome-based, stepwise, self-evaluated, domain-specific, or preference-derived. MCTS can incur high simulation costs and depends on domain-appropriate actions and rewards, limiting real-time use and generalizability.Source: §3.2.1 · pp. 7–8

3.2.2 Reward Modeling

• Process Reward Models evaluate reasoning steps, providing finer-grained error detection and more robust, interpretable supervision than outcome-only models.PRMs have been applied to code generation, long-chain mathematics, and multimodal reasoning; costly process labels motivate automated annotation and ORM-based alternatives, while trust concerns include weak explanations, training-data bias, and lower accuracy on early steps.Source: §3.2.2 · pp. 8–9

3.2.3 Self Improvement

• Training-based self-improvement uses model exploration for self-supervision, while test-time methods refine answers using prompts, external tools, or logits.Exploration ranges from few-shot examples and sampled trajectories to token-level methods and MCTS-generated data; improvement strategies include filtering with SFT, reward-based RL, and external feedback with distillation. rStar-Math [143] shows small models gaining reflective capabilities through self-evolving training.Source: §3.2.3 · pp. 9–10

3.2.4 Macro Action

• Macro-action frameworks organize reasoning into hierarchical phases such as planning, verification, and refinement, supporting test-time scaling and more diverse reasoning data.HiICL-MCTS [222] builds action-chain templates, while ReasonFlux [224] iteratively updates thought templates; applications include multimodal CoT synthesis, long-horizon reasoning, collaborative planning, and self-improvement.Source: §3.2.4 · p. 10

3.2.5 Reinforcement Fine-Tuning

• Reinforcement Fine-Tuning optimizes domain-specific reasoning with rewards and limited high-quality data; DeepSeek-R1 [31] demonstrates Long-CoT reasoning and a simplified, scalable training pipeline.R1-inspired multimodal systems report state-of-the-art performance across mathematics, medical imaging, and segmentation; strategies include structured rule-based rewards, modality-agnostic training, and explicit reasoning paths.Source: §3.2.5 · p. 11
• RFT remains challenged by unclear causes of emergent reasoning, reward saturation, and unstable long chains, motivating more stable RL, prompt synthesis, and theoretical analysis.Long-CoT can overflow context or fail to return answers, and may reflect latent base-model capabilities rather than RL alone; proposed priorities include adaptive reward shaping and length control.Source: §3.2.5 · pp. 11–13
• Reward-free RL variants use internal confidence, self-consistency, or voting as reward signals, but remain inferior to golden-answer rewards and need further study.Methods also explore entropy controls to preserve training diversity, though the role of entropy in RL remains largely empirical.Source: §3.2.5 · p. 12

Evolution of Reasoning LLMs

• Early approaches used LLM-driven BFS, DFS, or MCTS to traverse reasoning trees or graphs, selecting final paths through voting or Monte Carlo value estimation.Examples include Tree of Thoughts, [395] and Reasoning via Planning; paths were represented as intermediate-state nodes, with search strategies generating alternatives.Source: §3.3 · p. 13
• Search-based reasoning faced narrow, predefined exploration spaces, insufficient diversity across branches, limited cross-path experience sharing, and substantial computational overhead.Reward models and self-consistency were used to assess paths, while PRM/ORM pruning and speculative decoding addressed overhead.Source: §3.3 · p. 13
• Models including rSTaR [225] and Mulberry [228] introduced richer action spaces with high-level planning cues to broaden exploration, but required careful action-space design.Other examples include LLaMAV-o1 [230], HiICL-MCTS [222], g1, and Thinking-Claude.Source: §3.3 · p. 13
• o1 and QwQ internalized reasoning in context through exploratory macro-planning, alternative-path exploration, and extended chains using mechanisms such as “Rethink” and “Verification”.STILL-1 [396] linearized tree-search outputs into long chains with “Rethink”, “Wait”, and “Explore New Path” attributes, while STILL-2 [53] and sky-T1 used distillation to synthesize chains; linearized search chains struggled to match distilled-chain quality.Source: §3.3 · p. 13
• Distilling text-based slow thinking into multimodal LLMs has yielded marginal gains on MathVision [365], while fine-grained visual perception remains a challenge.Virgo [398] attempted this transfer, but extending slow reasoning to multimodal contexts remains difficult, especially for tasks requiring detailed perception [117], [399]–[401].Source: §3.3 · p. 13
• DeepSeek-R1 [31] and Kimi-k1.5 [258] show that simple RL scaling can elicit long reasoning chains, reflection, and advanced planning, while multimodal RL remains challenging.SimpleRL explored a streamlined replication pipeline, and R1V explored multimodal reasoning; challenges include consistent slow thinking on complex visual inputs and comparable training benefits to unimodal RL.Source: §3.3 · p. 13

4.1 Benchmark Categories

• Reasoning benchmarks span math, code, scientific, agent, medical, and multimodal tasks.Examples include AIME 2024 and MATH-500 for math; Codeforces and LiveCodeBench for code; GPQA Diamond and MMLU-Pro for science; WebShop and WebArena for agents; MedQA for medicine; and MMMU and MathVista for multimodal reasoning. Visual-centered benchmarks also test multimodal reasoning with text and images.Source: §4.1 · p. 14, §4.1.1 · pp. 14–15

4.1.2 Summary

• Simple benchmarks such as GSM8K, MATH-500, and ScienceQA are nearing saturation, motivating new benchmarks and soft reasoning evaluations.Soft reasoning lacks explicitly defined correct answers and can capture more nuanced human-like reasoning; benchmark data confidentiality and neutrality are also needed to address leakage risks.Source: §4.1.2 · pp. 15–16

4.3.1 Performance on Plain Text Benchmarks

• 69.9% on AIME 2024 and 73% on Codeforces are OpenAI-o1’s respective margins over GPT-4o, while reasoning LLMs generally excel in math and coding.DeepSeek-R1 also surpasses its DeepSeek-V3 predecessor across all benchmarks shown.Source: Table 8 · p. 16, §4.3.1 · p. 17

4.2.1 Task Types

• Math benchmarks use Pass@k and Cons@k, code uses Elo and Percentile, and scientific tasks use Exact Match and Accuracy.Pass@k measures success within k attempts, while Cons@k captures consistent correctness or logical coherence; Exact Match and Accuracy apply to fill-in-the-blank and multiple-choice questions, respectively.Source: §4.2.1 · p. 16

4.2.2 Technical Proposals

• ORM or PRM approaches use RM@k and Best-of-N, self-consistency uses decoding and majority-selection metrics, and RL uses reward and sample-efficiency measures.RM@k assesses reward-model ranking among top candidates; Best-of-N selects the highest-scored trajectory, while RL examples include Cumulative Reward and Sample Efficiency.Source: §4.2.2 · p. 16

4.2.3 Reasoning Paradigms

• Outcome Efficiency and Process Efficiency assess whether later solutions improve accuracy and diversify reasoning, respectively, and expose overthinking on simple problems.They are ratios of efficient tokens contributing to a correct answer and tokens from distinct solutions, respectively, to all output or solution tokens.Source: §4.2.3 · pp. 16–17

4.2.4 Summary

• Current evaluation frameworks emphasize final answers and inadequately measure reasoning efficiency and coherence despite high inference-computation costs.The survey recommends efficient proxy tasks to assess the accuracy and efficiency of long reasoning chains more comprehensively.Source: §4.2.4 · p. 17

4.3.2 Performance on Multimodal Benchmarks

• 78.2% on MMMU is OpenAI-o1’s highest vision-task score, and it exceeds GPT-4o by 7.2% on MathVista.Multimodal gains are less pronounced than text-only gains, in part because of technique limitations and insufficient evaluation datasets.Source: Table 9 · p. 17, §4.3.2 · p. 17

5.1 Advanced Infrastructure

• RL is used across LLM, MLLM, and agentic applications, including Agent-R1 and UI-R1 [426], with frameworks such as VeRL, OpenRLHF, and ReaLHF [429] supporting RLHF workflows.Despite architectural differences, these frameworks follow a canonical three-stage process.Source: §5.1 · p. 17
• RLHF infrastructure generates responses, derives training signals, then iteratively updates actor parameters to align decisions with task objectives and environmental feedback.The stages include rollout, preprocessing with reference-policy log probabilities, critic value estimates and advantage estimates, and actor–critic optimization.Source: §5.1 · pp. 17–18
• vLLM and SGLang support high-throughput rollout, while Megatron-LM [432] and MegaScale [433] use 3D parallelism for scalable distributed optimization.The optimization paradigms include data, pipeline, and tensor parallelism.Source: §5.1 · p. 18

5.1.1 Taxonomy of RL Infrastructure

• Single-controller systems such as VeRL and ReaLHF offer integrated workflow coordination but incur dispatch overhead across large-scale clusters.A central hub coordinates distributed programs, and messages propagated to worker nodes can hinder scalability.Source: §5.1.1 · p. 18
• Multi-controller designs scale with minimal scheduling overhead, but coordinating communication, computation, and data transfer across workers creates intricate code that is difficult to maintain and optimize.OpenRLHF exemplifies this architecture’s interplay of device autonomy and centralized coordination.Source: §5.1.1 · p. 18

5.1.2 Prospective Challenges of RL Infrastructure

• LongCoT generation can underuse GPUs when prompt responses vary in length, while agentic rollouts must also adapt to lengthy outputs and dynamic environments.Kimi-1.5 [258] incrementally stores partial rollouts and excludes non-critical segments from loss computation; Seed-Thinking-1.5 decouples model evolution from runtime to adjust on/off-policy sampling.Source: §5.1.2 · p. 18
• Agent–environment systems face heterogeneous response latencies and distributed HTTP-server stability demands, motivating asynchronous producer–consumer queues and scalable environmental design.The passage identifies agentic applications including DeepResearch [437], DeepSearch [438], and OpenManus.Source: §5.1.2 · pp. 18–19
• The Model-Computer Protocol (MCP) [440] can reduce bespoke tool-interface work and improve interoperability, but third-party servers create security risks requiring audits [441], authentication, and runtime monitoring [443].Potential threats include unauthorized access, code injection, and systemic sabotage.Source: §5.1.2 · p. 19

Interpretability of Reasoning Processes

• Interpretability research evaluates explanation faithfulness and analyzes how reasoning is encoded and learned, aiming to improve transparency without sacrificing performance, [445], [446], [447].Approaches include faithfulness metrics, mechanistic analyses of neurons and attention heads, and studies of gradient dynamics during training for deliberate reasoning.Source: §5.2 · p. 19

Knowledge Conflicts

• Methods detect and resolve conflicts between internal memory and external context through prompting, decoding, alignment, and fact editing without full retraining, [449],, [451], [452],, [454], [455], [456].Knowledge inconsistencies can undermine reasoning accuracy; the cited surveys categorize these conflicts and their impact.Source: §5.2 · p. 19

Safety & Robustness

• Alignment procedures often reduce reasoning performance, a trade-off termed “safety tax”; chain-of-thought attacks such as prompt hijacking and input slowdown expose structural vulnerabilities [457], [458], [459].Source: §5.2 · p. 19

External Grounding and Truthfulness

• External grounding aims to mitigate hallucination through retrieval-adapted context, evidence-first prompting, rewards for verifiable reasoning, and symbolic consistency checks, [466], [467], [468], [173].These methods connect reasoning paths to external evidence or retrieved data.Source: §5.2 · p. 19

Consistency and Controllability

• Self-reflection, plan-separated prompting, and step-length controls target flawed reasoning and coherence, since excessively long chains can reduce accuracy [469], [470], [471], [472], [473].Open challenges include consistency on open-domain inputs and aligning long reasoning chains with verified knowledge while balancing reasoning ability and safety.Source: §5.2 · p. 19

5.3 Reasoning LLM as Agent

• LLM agents combine information seeking, external-tool use, and dynamic refinement of reasoning to interact with complex environments.Recent work develops enhanced search, broader tool integration, new training methods, and evolving evaluation frameworks.Source: §5.3 · p. 19

5.3.1 Search-Augmented Reasoning

• Reinforcement learning trains LLMs to generate search queries within reasoning chains and proactively invoke search based on outcomes.Combining open-source reasoning models with specialized search tools can be competitive with larger proprietary systems, while collaborative agents and MCTS address complex retrieval tasks.Source: §5.3.1 · pp. 19–20

5.3.2 Tool Integration Beyond Search

• Agent tool use extends beyond search to code interpreters, specialized solvers, and integrated reasoning and memory capabilities.Self-learning approaches use hint-based learning and rejection-sampling fine-tuning to teach when and how to use tools; OR systems translate natural-language problems into mathematical models and executable solver code. Integrated frameworks combine web search [482], [483], code execution [484], and structured memory [485]–[487].Source: §5.3.2 · p. 20

5.3.3 Training Methodologies

• Training advances use reinforcement learning and bootstrapping to build strategic, specialized, and long-chain reasoning capabilities.Approaches include small-model RL [488], format tuning followed by RL for autoregressive search and self-correction, rule-based rewards for software engineering [490], multi-turn policy optimization without initial supervised fine-tuning [491], and bootstrapping with limited teacher-model or human-annotation reliance.Source: §5.3.3 · p. 20

5.3.4 Evaluation Benchmarks and Metrics

• Evaluation combines benchmarks for multi-step reasoning and tool use with interactive agent arenas and established reasoning and coding tests.Arenas compare agents on user-defined tasks and may incorporate community voting and feedback [494],, [495], [496]; foundational yardsticks include AIME, GPQA, LiveCodeBench [327], and SWE-Bench.Source: §5.3.4 · p. 20

Budget-sensitive LLMs

• Budget-sensitive methods constrain reasoning costs or penalize output length, including CoD [498], TALE-EP [499], L1 [500], TOPS [501], O1-Pruner [296], and Kimi-k1.5 [258].These methods impose reasoning-budget constraints or add length penalties.Source: §5.4 · p. 20

Diverse reasoning-length data

• CoT-Valve [502], TOPS, and C3oT [503] support post-training with diverse reasoning lengths through data interpolation, budget-sensitive sampling, or jointly training on compressed Short-CoT outputs.TOPS samples corresponding versions using a budget-sensitive data model.Source: §5.4 · p. 20

External models and switching mechanisms

• Routing and reward-model methods adjust reasoning scope: Routellm [505] selects a suitable model, Self-REF [506] uses confidence to route difficulty, and JudgeLRM [507]–[509] and DeepSeek-GRM [166] remove irrelevant paths.These approaches use external models or switching mechanisms to intervene in reasoning scope and support efficient test-time scaling.Source: §5.4 · p. 20

Efficient representations

• Efficient representations include TokenSkip’s token-importance selection, latent-space reasoning in COCONUT [510], hidden-space strategies in ICoT-KD [511] and CCoT [511], and hybrid or multimodal approaches.Token Assorted combines hidden-space and text-based reasoning to balance interpretability and efficiency, while Heima targets hidden-space reasoning for multimodal models.Source: §5.4 · pp. 20–21

Non-autoregressive reasoning models

• Non-autoregressive approaches aim to reduce prediction steps: Diffusion-LM uses non-autoregressive generation, LLaDA [512] scales a diffusion-based LM, and Diffusion-of-Thoughts [513] uses post-training to denoise token predictions.The passage describes Diffusion-of-Thoughts as significantly improving reasoning efficiency.Source: §5.4 · p. 21

New architectures for efficient reasoning

• Mamba [514] and RWKV [515] use linear attention or state-space techniques to improve efficiency by addressing the computational cost of ultra-long context in long thought-chain prediction.Source: §5.4 · p. 21

6.1 Efficient Reasoning LLMs

• Long autoregressive reasoning chains create latency, while smaller models struggle with slow-thinking tasks; future work targets reasoning tools, early stopping, verification, and capable SLMs.Some Codeforces problems require over 10,000 reasoning tokens [516], [517], and models may continue verifying after finding a correct answer [124]. Self-improvement through RL appears more effective in larger models, while 3B and 7B models struggle [31],,, [290].Source: §6.1 · p. 21

6.2 Collaborative Slow & Fast-thinking Systems

• Current reasoning LLMs struggle to balance fast responses with deliberate reasoning; adaptive switching, joint training, and coevolution could combine fast systems’ efficiency with reasoning models’ precision.Existing efforts include the System 1-2 switcher [116], speculative decoding –[520], and interactive continual learning –[523], but integrating both modes remains challenging and can leave strengths underused in domain-specific and complex tasks.Source: §6.2 · p. 21

6.3 Reasoning LLMs For Science

• Reasoning LLMs support scientific work [524], [525], with applications in medicine and mathematics, and potential in physics, engineering, and computational biology.Inference-time scaling can improve diagnostic accuracy in medical differential diagnosis and treatment planning [52]; FunSearch uses slow-thinking principles and AI-human collaboration to advance mathematical discoveries. These systems may also support more reliable and interpretable scientific discoveries.Source: §6.3 · p. 21

6.4 Deep Integration of Neural and Symbolic Systems

• Reasoning LLMs’ limited transparency and unclear reasoning pathways hinder complex tasks; integrating neural models with symbolic systems may improve explainability and reliability.Earlier symbolic logic systems offered clearer reasoning steps but were less adaptable; AlphaGeometry and AlphaGeometry2 [528] illustrate the neural-symbolic approach.Source: §6.4 · p. 21
• 84% of IMO geometry problems (2000–2024) were solved by AlphaGeometry2 [528], surpassing gold medalists’ averages, while OpenAI-o1 solved none.AlphaGeometry2 combines a Gemini-based model, [529], [530] with a more efficient symbolic engine, a new search algorithm, and knowledge sharing; neural-symbolic integration may improve adaptability and interpretability beyond geometry.Source: §6.4 · p. 21

6.5 Multilingual Reasoning LLMs

• Reasoning LLMs perform strongly in high-resource languages but remain limited in low-resource ones by data scarcity, stability, safety, and performance challenges.Research directions include addressing data scarcity and cultural biases through parameter sharing and incremental injection of domain-specific knowledge, enabling faster adaptation across languages.Source: §6.5 · p. 22

6.6 Safe Reasoning LLMs

• RL can introduce reward hacking, generalization failures, and language mixing; combining RL with SFT is proposed to reduce harmful outputs while preserving model effectiveness [539], [540].SFT addresses some safety issues but is not a complete solution; continued research is needed to control and guide models’ actions [536]–[538].Source: §6.6 · p. 22

• The survey connects foundational LLM progress with early System 2 technologies—symbolic logic, MCTS, and RL—as groundwork for reasoning LLMs.Source: §7 · p. 22
• It analyzes the features and core methods of recent reasoning LLMs and highlights representative models.Source: §7 · p. 22
• It reviews mainstream reasoning benchmarks and performance comparisons, identifies promising research directions, and tracks developments in a real-time GitHub repository.Source: §7 · p. 22