PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems

Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, Caiming Xiong, Shafiq Joty

arXiv:2504.09037v4cs.AIcs.CL

TL;DR

LLM reasoning research lacks a unified view spanning when reasoning is achieved and which system components perform it. This survey organizes methods across regimes and architectures, analyzes input and output perspectives, and synthesizes algorithms and agentic workflows. It highlights a shift from inference scaling toward learning to reason and from standalone models toward agentic systems, while noting unresolved generalization and reliability challenges.

  • Problem

    LLM reasoning research spans diverse regimes, architectures, algorithms, and workflows, but existing surveys often cover only limited dimensions.

  • Method

    The survey categorizes reasoning by inference scaling versus learning to reason and standalone versus agentic architectures, using input and output perspectives to organize techniques.

  • Results

    The survey synthesizes major milestones, learning algorithms, reasoner and verifier training, and agentic-system designs while identifying shifts toward learned reasoning and agentic systems.

  • Takeaways & Limitations

    The framework provides a structured perspective for analyzing LLM reasoning and highlights future directions including broader reasoning systems and automated workflow design.

  • Takeaways & Limitations

    Whether a truly generalizable reasoning system is possible remains open, as many state-of-the-art models focus mainly on mathematics and coding.

Abstract

from arXiv · show

Reasoning is a fundamental cognitive process that enables logical inference, problem-solving, and decision-making. With the rapid advancement of large language models (LLMs), reasoning has emerged as a key capability that distinguishes advanced AI systems from conventional models that empower chatbots. In this survey, we categorize existing methods along two orthogonal dimensions: (1) Regimes, which define the stage at which reasoning is achieved (either at inference time or through dedicated training); and (2) Architectures, which determine the components involved in the reasoning process, distinguishing between standalone LLMs and agentic compound systems that incorporate external tools, and multi-agent collaborations. Within each dimension, we analyze two key perspectives: (1) Input level, which focuses on techniques that construct high-quality prompts that the LLM condition on; and (2) Output level, which methods that refine multiple sampled candidates to enhance reasoning quality. This categorization provides a systematic understanding of the evolving landscape of LLM reasoning, highlighting emerging trends such as the shift from inference-scaling to learning-to-reason (e.g., DeepSeek-R1), and the transition to agentic workflows (e.g., OpenAI Deep Research, Manus Agent). Additionally, we cover a broad spectrum of learning algorithms, from supervised fine-tuning to reinforcement learning such as PPO and GRPO, and the training of reasoners and verifiers. We also examine key designs of agentic workflows, from established patterns like generator-evaluator and LLM debate to recent innovations. ...

1 Introduction

The survey organizes LLM reasoning across orthogonal regimes and architectures, while tracing a field-wide movement from inference-time scaling toward learned reasoning and from standalone models toward agentic systems.

  • Survey framework: The survey categorizes reasoning by regime— inference-time scaling versus learning to reason—and architecture—standalone LLMs versus interactive agentic systems.These dimensions are orthogonal, so either regime can apply to either architecture.
  • Reasoning regimes: Inference scaling improves reasoning by modifying prompts, searching or planning over candidates, and selecting better trajectories without updating the LLM.These methods trade additional computation, cost, and latency against effectiveness.
  • Emerging trends: The survey identifies milestones including OpenAI’s o1 for inference-time scaling and DeepSeek-R1 for learned reasoning, with reflection and alternative-solution exploration emerging as test-time computation increases.DeepSeek-R1 is described as achieving performance comparable to o1 while requiring fewer computational resources.
  • Reasoning regimes: Learning to reason updates the LLM through dedicated training to reduce reliance on costly inference-time computation, using supervised fine-tuning, preference learning, or reinforcement learning.The survey highlights training-data scarcity and automatic trajectory generation as central challenges.
  • Architectures: Agentic reasoning extends standalone LLMs through single-agent tool interaction or multi-agent communication, introducing workflow, coordination, and consensus challenges.Examples include external knowledge bases, verifiers, browser and Python tools, and collaborative or adversarial communication.
  • Survey contribution: The survey provides a comprehensive synthesis of reasoning milestones, algorithms, reasoners, verifiers, and architectural developments across these dimensions.It aims to clarify the rapidly expanding research landscape and support future work.

2 Background

The survey introduces foundational concepts for framing LLM reasoning as sequential decision-making, including reasoning steps, trajectories, rewards, and policies. It distinguishes training-based optimization from inference scaling with a frozen model.

  • LLM reasoning is framed as a sequential decision-making process within a Markov Decision Process framework.
  • A reasoning step is a token sequence representing a coherent deduction or intermediate conclusion, although its granularity can range from an entire response to a single token.
  • The survey does not analyze reasoning benchmarks, directing readers to existing benchmark surveys instead.
  • A reasoning trajectory records successive states, actions, and rewards as the model progresses from a question to a final answer.
  • RL-based reasoning seeks to maximize cumulative reward under a reasoning policy and reward function.
  • Training methods optimize policy parameters through preference learning or reinforcement learning, whereas inference scaling searches with a frozen model, often using a reward model.

2.2 Key Components of LLM Reasoning Systems

LLM reasoning systems can combine a reasoner, verifier, and refiner, whose complementary roles generate, evaluate, and improve reasoning responses. These components may be implemented differently depending on the regime and architecture.

  • A reasoner generates reasoning steps and serves as the policy model driving the reasoning process.
  • The three components are complementary but need not be separate models or modules.
  • A verifier evaluates final answers or intermediate steps and returns feedback ranging from scalar rewards to natural-language explanations.
  • A refiner revises flawed reasoning trajectories using verifier feedback, supporting inference-time improvement and implicit search.

2.3 System Architectures

The survey organizes reasoning systems into standalone LLM, single-agent, and multi-agent architectures. These architectures differ in whether reasoning is self-contained, interacts with an environment, or additionally involves agent-agent communication.

  • Standalone LLM Systems: Standalone LLM systems use one model for one or more reasoning components and generate outputs, often with rationales, without external interaction.
  • Standalone LLM Systems: Self-contained reasoning incorporates prior reasoning steps into prompts and samples from the same model.
  • Agentic Systems: Agenticness is treated as a spectrum increasing with an LLM’s interactiveness and autonomy.
  • Single-agent Systems: Single-agent systems interact with an external environment through actions such as tool use, retrieval, or answer refinement, receiving perceptions such as verifier or compiler feedback.
  • Single-agent Systems: The generator-critic-refiner workflow uses feedback to revise flawed actions but can suffer from over-refinement and diminishing or degraded performance.
  • Multi-agent Systems: Multi-agent systems add an agent-agent communication loop in which role-differentiated agents exchange messages and coordinate actions in a shared environment.
  • Multi-agent Systems: Debate and reconcile patterns reach consensus through mechanisms including judges, voting, or confidence-weighted decisions.

2.4 Reasoning Regimes

Reasoning systems operate under inference-time or training-time regimes. Inference scaling spends more computation during testing, while learning-to-reason shifts computation into training to reduce inference costs, with newer methods combining both.

  • Inference Scaling: Inference-time systems refine or search for solutions through repeated sampling and iterative computation, balancing computational cost against effectiveness.
  • Inference Scaling: Inference scaling includes prompt engineering, search and planning, and methods for evaluating candidate solutions.
  • Learning-to-Reason: Learning-to-reason trains models before deployment by generating reasoning trajectories and applying supervised or reinforcement learning.
  • Learning-to-Reason: Learning-to-reason reduces inference computation but incurs higher training costs and faces limited human-annotated reasoning data.
  • Hybrid Strategies: Recent approaches use knowledge of training and testing methods to optimize reasoners for known inference techniques or distribute computation adaptively between training and testing.

3 Improving Reasoning with Inference Scaling

Inference scaling improves reasoning by eliciting, structuring, exploring, and evaluating additional reasoning at test time. The survey organizes these methods across prompts, demonstrations, decomposition, search, planning, and agentic tool use.

  • Overview: Inference-time methods improve reasoning without updating the LLM by allocating additional computation to selecting or refining reasoning trajectories.These methods build on prompting, search, and planning to improve reasoning before producing an answer.
  • Input-level techniques: Instruction engineering improves reasoning by designing structured, template-based, or human-curated prompts.Prompt quality strongly affects whether LLMs reason effectively.
  • Input-level techniques: Demonstration engineering selects or generates examples relevant to each query instead of applying one fixed set to all queries.Approaches emphasize analogy, similarity, diversity, and scaling demonstrations to many-shot quantities.
  • Input-level techniques: Prompt optimization systematically searches or iteratively refines instructions using genetic, evolutionary, or related optimization methods.These methods target improved reasoning performance through strategic prompt modification.
  • Output-level techniques: Task decomposition and adaptive reasoning structures break complex problems into subtasks and organize intermediate reasoning steps.Frameworks such as Self-Discover and Auto-Evolve identify or adapt reasoning modules for different tasks.
  • Output-level techniques: Exploration-based inference scaling generates multiple reasoning paths whose quality is improved through aggregation or verification.Ensemble methods aggregate candidates, while verifier-based methods evaluate their correctness.
  • Agentic systems: Agentic reasoning augments LLMs with external verifiers, retrieval, tools, and interleaved reasoning-and-acting workflows.ReAct interleaves reasoning traces with task-specific actions, while LATS and RAFA extend reasoning-and-acting orchestration.

4 Learning Algorithms

The survey introduces the learning algorithms used to train reasoning policies and verifiers before discussing data curation. It organizes these algorithms into supervised fine-tuning, reinforcement learning, and preference learning.

  • Organization: Learning algorithms are presented through their loss functions and treated separately from the data curation process.The section first covers algorithms for training reasoning models, then discusses verifier training.
  • Organization: The learning-algorithm discussion comprises imitation learning through supervised fine-tuning, reinforcement learning, and preference learning.These form the section’s three main parts.

4.1 Learning of Reasoner

The survey explains how reasoners are trained with supervised fine-tuning, reinforcement learning, and preference learning. It formalizes language-model reasoning as sequential decision-making and compares policy-gradient and preference-optimization approaches.

  • 4.1.1 Imitation Learning - Supervised Fine-tuning: Supervised fine-tuning maximizes next-token log probabilities for reasoning outputs given prompts and previously generated tokens.Its dataset contains inputs and human-written or AI-generated reasoning processes and answers, with prompt tokens masked from the loss.
  • 4.1.2 Reinforcement Learning for Reasoning: Reinforcement learning models LLM text generation as a Markov decision process with states, actions, transitions, rewards, and a policy.At each step, the model selects an action, transitions to a new state, and receives an environmental reward.
  • 4.1.2 Reinforcement Learning for Reasoning: LLM reinforcement-learning actions may be individual tokens, token chunks representing reasoning steps, or full responses.These choices determine the action space and whether rewards, values, and advantages are assigned at token, step, or response level.
  • 4.1.2 Reinforcement Learning for Reasoning: PPO trains a policy with a reference model and value model, while clipping and KL regularization constrain policy updates and exploration.The value model estimates state values, and both policy and value model are updated iteratively.
  • 4.1.2 Reinforcement Learning for Reasoning: REINFORCE and RLOO optimize responses using observed rewards without a parameterized value function, with RLOO replacing the baseline by leave-one-out reward averages.RLOO uses Monte Carlo samples to compute its leave-one-out baseline.
  • 4.1.2 Reinforcement Learning for Reasoning: GRPO uses PPO’s clipped surrogate objective but estimates advantages from groups of sampled outputs rather than a parameterized value model.Variants such as DAPO address length bias and penalties for responses exceeding the context length.
  • 4.1.3 Preference Learning: Preference learning trains models to generate outputs aligned with human preferences using chosen and rejected responses.DPO is described as simpler and more stable than PPO-based techniques, while remaining subject to response-length and extended-training limitations.

4.2 Learning of Verifiers and Reward Models

The survey organizes verifiers by whether they assign rewards to complete outcomes, individual reasoning steps, or generate explanatory feedback. These verifiers support both training-time credit assignment and inference-time verification.

  • Verifier roles: Verifiers improve reasoning through both training-time credit assignment and inference-time scaling verification.Reasoning reward models focus on correctness of reasoning chains rather than general criteria such as helpfulness or safety.
  • Discriminative verifiers: Outcome reward models assign a scalar reward to a full trajectory using correctness labels or correct–incorrect output pairs.Pairwise training commonly uses the Bradley-Terry loss, analogous to DPO.
  • Discriminative verifiers: Bradley-Terry loss has been reported as the best-performing option among evaluated pairwise losses for outcome reward models.Alternatives include hinge, margin-based, focal, and Bradley-Terry variants.
  • Discriminative verifiers: Process reward models evaluate correctness at each reasoning step, requiring finer-grained supervision than outcome reward models.Monte Carlo Tree Search variants can generate step labels by rolling out intermediate responses and using outcome accuracy as a correctness proxy.
  • Generative verifiers: Generative verifiers produce natural-language evaluations that provide actionable feedback and explanations alongside assessment.They include critique models, LLM-as-judge models, and generative reward models.
  • Generative verifiers: Generative reward models combine language-generation objectives with an answer-token loss to unify explanation generation and outcome classification.The answer-token loss is added to standard SFT or DPO-style generation losses.

5 Learning to Reason

The survey frames learning to reason as an alternative to inference-time scaling, focusing on training data and learning algorithms. It distinguishes offline methods that reuse collected trajectories from online reinforcement learning that learns directly from questions and rewards.

  • Motivation: Inference-time reasoning can improve performance but leaves model parameters unchanged and requires substantial computation during inference.The survey presents learning to reason as a response to these limitations.
  • Data recipe: Trajectory collection constructs reasoning data in which generators produce steps, potentially using external tools, for synthetic or real-world inputs.The central challenge is making the simulation realistic and diverse.
  • Model recipe: Offline learning uses collected trajectories for methods such as SFT and DPO, whereas online learning uses algorithms such as GRPO and PPO directly on questions and rewards.The distinction concerns whether trajectories are collected beforehand or generated during learning.

5.1 Learning to Reason with Standalone LLM

Standalone LLMs learn reasoning from collected trajectories through offline training or directly from sampled chains and rewards. The survey covers prompt construction, output filtering and synthesis, distillation, supervised fine-tuning, preference learning, reinforcement learning, and latent reasoning.

  • Training framework: Offline standalone-LLM methods collect trajectories with correct and incorrect outcomes, while online methods learn from sampled reasoning chains and their rewards.The section organizes both approaches around constructing inputs, obtaining outputs, and training the LLM.
  • Input construction: Diverse, high-quality prompts are important because narrow or homogeneous prompt sets limit exploration and weaken distillation and reinforcement learning.Question augmentation and knowledge-graph synthesis address breadth and coverage in input construction.
  • Output construction: Output collection methods improve training trajectories by sampling targeted reasoning patterns and retaining outputs that satisfy ground-truth quality criteria.The section separates ground-truth-based methods from learned-verifier methods discussed elsewhere.
  • Output construction: Reasoning trajectories can encode self-reflection, planning, failed branches, and distilled behaviors from stronger reasoning models.Examples include Reasoning-as-Planning, rule-based synthesis from MCTS trajectories, and distillation from models such as OpenAI-o1 and DeepSeek-R1.
  • Training algorithms: SFT, preference learning, and reinforcement learning provide successive training routes for collected reasoning data, including DPO and verifiable-answer rewards.DeepSeek-R1 is cited as evidence for the potential of pure reinforcement learning with verifiable answers.
  • Latent reasoning: Latent reasoning reduces inference-time reasoning traces by omitting intermediate tokens or compressing them into specialized tokens or continuous representations.This addresses the inefficiency of long natural-language chains, which may contain many tokens that contribute little directly to reasoning.

5.2 Learning to Reason with Single-agent Systems

Single-agent systems extend standalone reasoning through interactions with external environments, tools, verifiers, and feedback. Training methods use filtered or corrected trajectories, search, teacher-agent demonstrations, SFT, reinforcement learning, and refiners.

  • Agent trajectories: Single-agent reasoning uses environment interactions to construct trajectories and train agents through perceptions, actions, and feedback.The survey includes execution feedback, external models, verifier-guided search, and teacher-agent trajectory distillation.
  • Trajectory construction: Execution feedback from unit tests, theorem provers, and proof assistants helps filter or improve trajectories for training.These interactions provide correctness signals for program repair and formal theorem proving.
  • Trajectory construction: External critic and refiner models guide exploration, identify easier problems, correct wrong outputs, and support deeper proof searches.Refiners can use environmental feedback to iteratively improve agent responses.
  • Trajectory construction: Verifier-guided search uses value functions, outcome reward models, process preference models, or LLM judges to improve difficult reasoning trajectories.MCTS-based methods represent sentences or tokens as tree nodes and use evaluators to guide search.
  • Trajectory construction: Teacher agents generate interactive tool-use and diverse solutions, while self-sampled correct and teacher-corrected trajectories broaden the training distribution.Examples include GPT-4-derived mathematical trajectories and multi-perspective data augmentation.
  • Agent training: SFT adapts agents to unfamiliar environments using collected, filtered, refiner-corrected, or self-training trajectories.The cited applications include program repair and other agentic tasks.
  • Agent training: Reinforcement learning enhances agent reasoning through verifiable outcome rewards and feedback from external tools or proof assistants.The survey highlights GRPO, PPO, Expert Iteration, RLEF, and RLPAF among the approaches used.

5.3 Learning to Reason with Multi-agent System

Multi-agent learning-to-reason systems improve reasoning by coordinating agents through communication, centralized control, structured coordination, and debate. These methods address the added complexity of aligning actions and interests across agents.

  • Communication and coordination: Multi-agent systems collect reasoning trajectories by coordinating agent-agent communication and actions across different agents.Communication functions as an input or perception mechanism for trajectory collection.
  • Communication and coordination: Centralized controllers and centralized-training methods help agents share information, improve sample efficiency, and direct exploration toward uncertain areas.MARCO uses centralized training with decentralized execution and a shared model across agents’ policies.
  • Communication and coordination: Learned protocols, shared message pools, and broadcast private intentions provide alternative mechanisms for agents to exchange task-relevant information.These mechanisms tailor communication to tasks, agent profiles, or current goals and subtasks.
  • Action coordination: Expert constraints, graph-based methods, and hierarchical policies reduce coordination difficulty by narrowing exploration and separating strategy from execution.HAVEN divides policies into strategy and execution levels to improve inter-agent and inter-level synchronization.
  • Action coordination: Multi-agent training addresses higher communication and coordination complexity through iterative debate and gradient modification for stable cooperation.DEBATUNE generates refined, diverse training data through multi-round debate, while another method balances individual and collective interests.

5.4 Toward Cost-aware and Inference-aware Training

Cost-aware and inference-aware training jointly address the computational expense of reasoning by allocating resources selectively and incorporating inference procedures into training. The survey highlights efficient data use and training objectives aligned with multi-attempt inference.

  • Motivation: Inference-time scaling and learning-to-reason are complementary, but both incur costs from additional generation or large trajectory datasets.Cost-aware methods allocate resources during training and inference, while inference-aware methods target time and cost efficiency.
  • Cost-aware methods: Dynamic resource allocation predicts prompt difficulty or output requirements to assign inference budgets according to input complexity.Learnable models estimate batched-query difficulty and seek efficient inference configurations.
  • Cost-aware methods: 1,000 samples enabled Qwen2.5-32B-Instruct to surpass o1-preview on competition math benchmarks, while 817 curated samples produced superior performance across broader math benchmarks.These gains depended on the strong Qwen2.5-32B-Instruct base model and did not occur with Qwen1.5-32B-Instruct.
  • Inference-aware training: Training models to explore diverse strategies can outperform selecting candidates from a single best-attempt distribution when multiple problem-solving attempts are available.This motivates explicitly incorporating inference procedures into training rather than treating inference computation as post-hoc.
  • Inference-aware training: Inference-aware objectives align training with Best-of-N by distilling its candidate distribution, calibrating rewards, or directly optimizing selection.These approaches address the mismatch between Best-of-N’s non-differentiable argmax and conventional fine-tuning objectives.

6 Discussion: Trends and Open Challenges

The survey identifies a shift from inference scaling to learning-to-reason and from standalone models to agentic systems, while emphasizing unresolved questions about generalization, autonomy, evaluation, and reasoning mechanisms. It also highlights domain specialization and efficiency constraints as continuing challenges.

  • Observed trends: Reasoning research is trending from inference scaling toward learning to reason and from standalone LLMs toward agentic systems.The survey organizes these trends across its regime and architecture dimensions.
  • Observed trends: Inference scaling complements reasoning-specific training, and combining self-consistency with such training has yielded further improvements.The survey presents the two approaches as complementary rather than mutually exclusive.
  • Observed trends: Supervised fine-tuning and knowledge distillation are effective but depend on strong teacher models, while outcome-based rejection sampling can converge quickly.These limitations motivated development of additional learning-to-reason methods.
  • Agentic systems: Agentic reasoning introduces workflow and coordination complexity, and current autonomy remains largely limited to planning and lacks proactive clarification or information requests.The survey also notes ongoing debate about whether agentic reasoning benefits straightforward tasks.
  • Domain-specific challenges: LLM reasoning in tabular tasks remains constrained by heterogeneous features, dependence on prompt design and preprocessing, hallucinations, and fairness concerns.Poor or out-of-distribution preprocessing can cause information loss, misinterpretation, multicollinearity, and interpretability problems.
  • Generalization and domain scope: Whether reasoning skills generalize and transfer beyond pretraining tasks remains open, with many state-of-the-art models concentrated in mathematics and coding.The survey notes growing interest in domain-specific reasoning models alongside the unresolved goal of a general-purpose reasoner.
  • Understanding reasoning: Empirical studies report nontrivial performance across reasoning forms, but counterfactual performance can degrade relative to default conditions, suggesting narrow procedures.The survey covers abstractive, compositional, logical, commonsense, mathematical, and social reasoning evaluations.

7 Conclusion

The survey organizes LLM reasoning across regimes, architectures, and input/output perspectives while reviewing learning algorithms, reasoners, verifiers, and agentic workflows. It highlights shifts toward learning-to-reason and agentic systems, alongside unresolved evaluation, mechanism-understanding, and data challenges.

  • Survey scope: The survey categorizes LLM reasoning techniques along regimes and architectures, analyzing both input-level and output-level perspectives.It covers inference-time and training-based regimes alongside standalone and agentic architectures.
  • Survey scope: The review compares supervised fine-tuning, reinforcement learning, reasoner and verifier training, and emerging agentic reasoning approaches.It includes trends toward inference-aware learning-to-reason and automated multi-agent design.
  • Open challenges: Open challenges remain in evaluating reasoning, understanding real reasoning mechanisms, and addressing data limitations.The survey encourages future work on inference-aware learning-to-reason and automated multi-agent design.
Loading 2504.09037v4…