PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models

Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, Yong Li

arXiv:2501.09686v3cs.AIcs.CL

TL;DR

LLM reasoning research seeks scalable ways to produce and train intermediate reasoning processes for complex tasks. This survey reviews automated data construction, learning-to-reason methods, test-time scaling, and related model developments; it concludes that these advances establish a path toward large reasoning models, exemplified by OpenAI’s o1 series, while theoretical understanding of slow thinking remains limited.

  • Problem

    Step-by-step reasoning requires substantial training data, but human annotation of reasoning trajectories is costly and difficult to scale.

  • Method

    The survey synthesizes automated reasoning-data construction, supervised and reinforcement-based learning, preference learning, prompting, search, test-time scaling, benchmarks, and model projects.

  • Results

    Train-time reinforcement learning and search-based test-time scaling form a path toward large reasoning models, with OpenAI’s o1 series identified as a key milestone.

  • Takeaways & Limitations

    High-quality process-reward data and deliberate reasoning computation are central themes in current efforts to expand LLM reasoning capabilities.

  • Takeaways & Limitations

    Theoretical analysis of LLM slow-thinking remains underdeveloped, and current human-brain imitation is largely limited to system-level design.

Abstract

from arXiv · show

Language has long been conceived as an essential tool for human reasoning. The breakthrough of Large Language Models (LLMs) has sparked significant research interest in leveraging these models to tackle complex reasoning tasks. Researchers have moved beyond simple autoregressive token generation by introducing the concept of "thought" -- a sequence of tokens representing intermediate steps in the reasoning process. This innovative paradigm enables LLMs' to mimic complex human reasoning processes, such as tree search and reflective thinking. Recently, an emerging trend of learning to reason has applied reinforcement learning (RL) to train LLMs to master reasoning processes. This approach enables the automatic generation of high-quality reasoning trajectories through trial-and-error search algorithms, significantly expanding LLMs' reasoning capacity by providing substantially more training data. Furthermore, recent studies demonstrate that encouraging LLMs to "think" with more tokens during test-time inference can further significantly boost reasoning accuracy. Therefore, the train-time and test-time scaling combined to show a new research frontier -- a path toward Large Reasoning Model. The introduction of OpenAI's o1 series marks a significant milestone in this research direction. In this survey, we present a comprehensive review of recent progress in LLM reasoning. We begin by introducing the foundational background of LLMs and then explore the key technical components driving the development of large reasoning models, with a focus on automated data construction, learning-to-reason techniques, and test-time scaling. We also analyze popular open-source projects at building large reasoning models, and conclude with open challenges and future research directions.

1 Introduction

LLM reasoning has progressed from prompting intermediate “thoughts” to combining reinforcement-learning train-time scaling with search-based test-time scaling. This survey reviews that progression toward large reasoning models, including OpenAI’s o1 series and open-source efforts.

  • “Thought” tokens represent intermediate reasoning steps, enabling techniques such as chain-of-thought and tree-of-thought prompting.These approaches support more sophisticated reasoning architectures, including tree search and reflective reasoning.
  • Human annotation is costly for step-by-step reasoning trajectories, motivating LLM-driven search with external verification and process reward models.The resulting reinforced cycle combines automated search with learning and reduces reliance on human annotation data.
  • Test-time compute scaling can improve reasoning accuracy by using process reward models to evaluate and search through intermediate thoughts.The survey describes deliberate reasoning with more tokens as a test-time scaling law.
  • Train-time reinforcement learning and search-based test-time scaling together define a path toward large reasoning models.OpenAI’s o1 series is identified as a key milestone, while projects such as OpenR, LLaMA-Berry, and Journey Learning pursue open-source reproduction.
  • The survey reviews LLM reasoning foundations, automated data construction, reinforcement learning, test-time scaling, open-source projects, enhancement techniques, benchmarks, and open problems.

2 Background

LLM reasoning builds on pretraining, fine-tuning, prompting, and agentic workflows. Structured intermediate reasoning, multi-path exploration, and problem decomposition provide increasingly deliberate ways to improve performance on complex tasks.

  • Pretraining provides linguistic and world knowledge that supports the emergence of advanced capabilities and value alignment.
  • Supervised fine-tuning adapts general-purpose LLMs to task-specific requirements and can improve zero-shot and reasoning capabilities.It refines outputs using labeled datasets after pretraining.
  • Alignment: Reinforcement learning and preference learning train reasoning through feedback, reducing reliance on explicit reward models in approaches such as DPO.Alignment-based post-training decomposes reasoning and gradually feeds signals back to the model.
  • Prompting: Chain-of-thought prompting improves problem solving by explicitly eliciting intermediate reasoning steps.Self-Consistency extends this approach by generating multiple reasoning paths for more reliable conclusions.
  • Prompting: Tree- and graph-structured methods explore multiple reasoning paths, while ReAct interleaves reasoning with actions in external environments.
  • Prompting: Decomposition methods break complex problems into manageable components for tasks requiring multiple steps or levels of analysis.Examples include Least-to-Most Prompting, Algorithm of Thoughts, and Plan-and-Solve.
  • Prompting: Structured prompting strategies can unlock LLM reasoning abilities for complex social-scientific tasks requiring careful and systematic analysis.
  • Agentic Workflow: Agentic workflows program LLM thinking patterns without additional training, but often require more test-time compute.In-context learning helps models generalize to unseen problems from a few demonstrations.

3 Data Construction: from Human Annotation to LLM Automation

Reasoning-data construction is shifting from expensive human annotation toward LLM automation and search-based process annotation. The survey presents a progression from human–LLM collaboration to stronger-model labeling, simulation, tree search, and self-refinement.

  • Human Annotation: Human annotation is high-quality but prohibitively expensive and difficult to scale for large reasoning datasets.LLM automation is more cost-effective but faces limited validation, especially for step-by-step reasoning.
  • Human Annotation: Process supervision labels intermediate reasoning decisions, but producing the required large-scale human annotations is resource-intensive and increasingly impractical.
  • Human Annotation: Human–LLM collaboration uses LLMs for pre-annotation and humans to review and correct only poor-quality annotations.
  • LLM Automation: LLMs can automate annotation across tasks from question-answer extraction to adding target information, using reasoning and in-context learning for complex cases.
  • LLM Automation: Search with feedback enables LLMs to self-correct erroneous samples and iteratively generate refined training data.
  • LLM Automation: Automated process annotation can use stronger external LLMs to evaluate intermediate generation steps instead of relying on manual labels.
  • LLM Automation: Monte Carlo simulation assesses intermediate steps by repeatedly extending reasoning and averaging the resulting outcomes.
  • LLM Automation: Monte Carlo Tree Search improves simulation efficiency, while PRM-based self-refinement repeatedly upgrades the model and annotation quality.

4 Learning to Reason: from Supervised to Reinforcement Fine-tuning

This section reviews the progression from supervised fine-tuning to reinforcement-based methods for improving LLM reasoning. It emphasizes automated feedback, process-level rewards, and the trade-offs between data requirements, reasoning diversity, and computational cost.

  • 4.1 Optimizing Pre-trained LLM: Supervised Fine-tuning: Supervised fine-tuning adapts pretrained LLMs to task-specific requirements using labeled datasets, improving reasoning for downstream applications.CoT-based fine-tuning additionally trains models to generate intermediate reasoning steps, improving final answers and interpretability.
  • 4.1 Optimizing Pre-trained LLM: Supervised Fine-tuning: SFT depends on expensive labeled data, can cause catastrophic forgetting, and may impose prohibitive computational costs.These limitations motivate alternatives such as reinforcement learning and other parameter-efficient or multitask approaches.
  • 4.2 Optimizing Pre-trained LLM: Reinforcement Learning: Reinforcement learning trains LLMs through trial-and-error reward signals, updating model parameters according to feedback on actions.RLHF and Constitutional AI illustrate how reward or principle-based feedback can align models with human preferences while reducing reliance on human-labeled data.
  • 4.2 Optimizing Pre-trained LLM: Reinforcement Learning: RL and DPO are described as straightforward approaches that foster reasoning ability and align models with human preferences through short-term feedback.The section links this feedback structure to simpler learning and suitability for real-time applications and concise reasoning tasks.
  • 4.3 Enhancing Multi-step Reasoning with Outcome Reward Model: Outcome reward models provide feedback only after complete reasoning solutions, making intermediate-step credit assignment a central challenge.Monte Carlo sampling and Monte Carlo Tree Search address this challenge, but their search costs remain high and existing applications emphasize easily verifiable tasks.
  • 4.4 Enhancing Multi-step Reasoning with Process Reward Model: Process reward models distribute rewards across reasoning steps, replacing sparse outcome feedback with more detailed process-oriented supervision.This step-level feedback is presented as a major direction for improving multi-step reasoning and expanding RL methods across task domains.
  • 4.5 Reinforcement Fine-tuning: Reinforcement fine-tuning uses limited user preference data and a grader model to iteratively optimize domain-specific, multi-step reasoning.The survey reports that OpenAI discussions suggested learning in new domains with as few as dozens of user preference data, while noting that RFT technical details remain incompletely released.

5 Test-time Scaling: from CoTs to PRM Guided Search

This section surveys test-time scaling methods that improve LLM reasoning by allocating more computation to explicit thought generation and search. It covers majority voting, tree and beam search, and lookahead search, alongside PRM-guided inference.

  • 5 Test-time Scaling: from CoTs to PRM Guided Search: Test-time prompting methods such as Chain-of-Thought and Tree-of-Thoughts improve LLM performance by guiding models through explicit reasoning processes.The section frames this as a complement to train-time optimization through reinforcement learning.
  • 5.2 PRM Guided Search: PRMs guide test-time reasoning by evaluating intermediate thoughts, while test-time scaling laws associate increased inference compute with improved reasoning capability.Figure 3 distinguishes discarded paths, adopted paths, and endpoints during inference exploration.
  • 5.1 Test-time Scaling Methods: Majority vote generates multiple inference traces and selects the answer receiving the most predictions.It aggregates outputs across traces rather than selecting from a single reasoning path.
  • 5.1 Test-time Scaling Methods: Tree Search recursively explores choices in a search tree, while Tree-of-Thoughts structures alternative reasoning paths and uses self-evaluation to select subsequent steps.MCTS implements selection, expansion, simulation, and backpropagation to progressively expand the search space.
  • 5.1 Test-time Scaling Methods: Beam Search retains the top-K highest-scoring paths at each step, maintaining multiple candidates instead of following only a greedy path.This broader candidate maintenance is used to improve generation quality.
  • 5.2 PRM Guided Search: Lookahead Search modifies Beam Search by rolling out up to k steps ahead and stopping early when a solution endpoint is reached.Its scoring incorporates forward simulations rather than relying only on current-step scores.

6 Path toward Large Reasoning Model

The survey presents OpenAI’s o1 series as demonstrating structured reasoning, adaptive problem solving, and scaling with additional computation. It also reviews four open-source efforts that pursue related reasoning capabilities through distinct training and inference strategies.

  • 6.1 Development of OpenAI o1 Series: OpenAI’s o1 series demonstrates effective knowledge integration across structured analytical and computational reasoning tasks.Reported evaluations include 83.3% success in competitive programming and 150% of human-level performance in structured analytical thinking and computational reasoning tasks.
  • 6.1 Development of OpenAI o1 Series: o1 systematically decomposes increasingly difficult problems into structured solution steps.Studies report near-perfect scores on the Dutch Mathematics B exam and high accuracy across 105 science and math problems of increasing difficulty.
  • 6.1 Development of OpenAI o1 Series: o1 maintains reliable and coherent reasoning across diverse complex tasks, including planning with incomplete information and long-horizon dependencies.PlanBench evaluations highlight constraint satisfaction, state management, temporal reasoning, and causal understanding across varied task formulations.
  • 6.1 Development of OpenAI o1 Series: o1’s reasoning improves with additional computation during both training and inference, including more time spent thinking at test time.In programming tasks, allowing 10,000 submissions per problem reportedly pushes performance above the gold medal threshold without test-time selection strategies.
  • 6.2 Open-source Attempts of Large Reasoning Models: Four open-source frameworks explore distinct implementations for reinforced reasoning and help researchers understand o1-like capabilities.OpenR uses process-supervised search data and PRM training; Rest-MCTS* jointly updates policy and PRM; the o1 Replication Journey emphasizes trial-and-error, reflection, and backtracking; LLaMA-Berry focuses on inference-time MCTS and Self-Refine.

7 Other Test-time Enhancing Techniques

The survey reviews training-free test-time techniques that improve reasoning without modifying the underlying LLM. These methods use iterative feedback, memory, or searches over agentic-system designs, while facing efficiency and automation challenges.

  • 7 Other Test-time Enhancing Techniques: Training-free test-time methods improve reasoning with off-the-shelf LLMs through verbal reinforcement, memory-based reinforcement, and agentic system search.The methods dynamically refine reasoning without changing model parameters and are organized around the three categories shown in Figure 4.
  • Verbal Reinforcement Search: Verbal Reinforcement Search iteratively uses feedback and the LLM’s semantic capabilities to explore and refine solution spaces without additional training.The survey describes applications spanning mathematical optimization, symbolic reasoning, hypothesis discovery, multi-agent collaboration, and embodied experimentation.
  • Memory-based Reinforcement: Memory-based reinforcement addresses open-ended tasks by storing observations and experiences for later retrieval, reflection, or abstraction.The reviewed approaches include experiential learning from observation-action pairs, reflective learning about successes and failures, and concept learning that generalizes across tasks.
  • Agentic System Search: Agentic System Search optimizes agentic systems at the prompt, module, or agent level rather than directly searching the solution space.Module-level search can recombine planning, reasoning, tool-use, and memory blocks through standard input-output interfaces.
  • Agentic System Search: Agentic System Search enables self-improvement without changing the LLM structure, but must improve search efficiency, reduce costs, and preserve automation and rationality.The broad search spaces create a shared challenge across prompt-, module-, and agent-level approaches.

8 Evaluation Benchmarks

The survey organizes LLM reasoning benchmarks across mathematical, logical, commonsense, and code-generation capabilities. These benchmarks span difficulty levels, reasoning types, modalities, and practical task settings.

  • 8 Evaluation Benchmarks: Reasoning benchmarks are systematically reviewed through a taxonomy covering mathematical, logical, commonsense, and code-generation tasks.The survey presents the taxonomy as a framework for documenting capability improvements and selecting promising research directions.
  • Mathematical Reasoning: Mathematical benchmarks range from elementary arithmetic and word problems to competition mathematics, geometry, theorem proving, and cross-modal reasoning.Examples include MATH-401, MultiArith, CHAMP, FIMO, Geometry3K, MINIF2F, MATHVISTA, and SciBench.
  • Logical Reasoning: Logical reasoning benchmarks assess deductive, inductive, and abductive reasoning as distinct forms of drawing conclusions, recognizing patterns, and explaining observations.ProofWriter targets premise-based derivations, BigBench and CLUTTR examine pattern inference, and αNLI evaluates likely explanations under incomplete information.
  • Commonsense Reasoning: Commonsense benchmarks evaluate everyday, social, physical, adversarial, counter-commonsense, and factual reasoning.Representative datasets include CommonsenseQA, SocialIQA, HellaSwag, PIQA, OBQA, MMLU, and FactCC.
  • Code Generation: Code-generation benchmarks test accurate, efficient, and reliable programming across execution-driven, software-engineering, data-science, and competitive-programming settings.ODEX verifies generated code through execution, while SWE-bench, DS-1000, and APPS represent practical and challenge-oriented programming tasks.

9 Discussion

The discussion identifies high-quality reasoning data and slow-thinking mechanisms as central to advancing large reasoning models, while emphasizing substantial theoretical and practical gaps. It also points toward broader, verification-friendly applications beyond standard benchmarks.

  • 9.1 Inspirations from the recent advances: High-quality process-reward data is increasingly treated as more important than general pre-training data size for large reasoning models.Approaches include Monte Carlo Tree Search and LLM generation for producing chain-of-thought data.
  • 9.2 Slow-thinking and reasoning: OpenAI o1’s apparent breakthrough remains technically opaque and seems mainly associated with post-training on slow-thinking data.The paper characterizes current human-brain imitation as system-level design rather than detailed implementation.
  • 9.2 Slow-thinking and reasoning: Slow-thinking knowledge should inform reasoning-data generation, reward functions, and learning processes.The discussion presents these areas as related design targets for developing stronger reasoning abilities.
  • 9.2 Slow-thinking and reasoning: The field still lacks significant representative theoretical analyses of LLM slow-thinking.Comparisons between o1 Preview and o1 Mini suggest that cost and chain-of-thought depth matter across text, code, and mathematics tasks.
  • 9 Discussion: Verification-friendly tasks beyond popular benchmarks may benefit from reasoning-enhancement methods, including domains such as urban planning and logistics scheduling.FunSearch is cited as an example of general ability on tasks where solutions are difficult to provide but fast to verify.

10 Conclusion

Recent LLM advances combine intermediate thought, reinforcement learning for train-time scaling, and search algorithms for test-time scaling. Together, these developments establish the groundwork for large reasoning models capable of increasingly complex cognitive tasks.

  • 10 Conclusion: Intermediate thoughts, train-time reinforcement learning, and test-time search have laid the groundwork for large reasoning models.The conclusion connects these innovations with addressing increasingly complex cognitive tasks, exemplified by OpenAI’s o1 series.
Loading 2501.09686v3…