PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Reinforcement Learning for Large Reasoning Models

Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, Fangfu Liu, Xiang Xu, Jiaze Ma, Xuekai Zhu, Ermo Hua, Yihao Liu, Zonglin Li, Huayu Chen, Xiaoye Qu, Yafu Li, Weize Chen, Zhenzhao Yuan, Junqi Gao, Dong Li, Zhiyuan Ma, Ganqu Cui, Zhiyuan Liu, Biqing Qi, Ning Ding, Bowen Zhou

arXiv:2509.08827v3cs.CLcs.AIcs.LG

TL;DR

Scaling reinforcement learning for reasoning models faces challenges in rewards, algorithms, data, and infrastructure. This survey synthesizes RLVR components, debated issues, resources, and applications, highlighting research directions for broader reasoning models.

  • Problem

    Scaling RL for reasoning models remains challenging because progress depends on computational resources, algorithm design, training data, infrastructure, and reliable reward signals.

  • Method

    The paper surveys RLVR for LLM reasoning, covering reward design, policy optimization, sampling, debated training issues, resources, infrastructure, and applications.

  • Results

    The survey organizes recent RL-for-LRM research around foundational components, controversial problems, training resources, downstream applications, and promising research directions.

  • Takeaways & Limitations

    RLVR is presented as a framework for enhancing LLM reasoning through direct outcome-level rewards, while its broader development remains an active research area.

  • Takeaways & Limitations

    RL remains sensitive to initial data distributions and has limited ability to recover and generalize after substantial representation drift from SFT.

Abstract

from arXiv · show

In this paper, we survey recent advances in Reinforcement Learning (RL) for reasoning with Large Language Models (LLMs). RL has achieved remarkable success in advancing the frontier of LLM capabilities, particularly in addressing complex logical tasks such as mathematics and coding. As a result, RL has emerged as a foundational methodology for transforming LLMs into LRMs. With the rapid progress of the field, further scaling of RL for LRMs now faces foundational challenges not only in computational resources but also in algorithm design, training data, and infrastructure. To this end, it is timely to revisit the development of this domain, reassess its trajectory, and explore strategies to enhance the scalability of RL toward Artificial SuperIntelligence (ASI). In particular, we examine research applying RL to LLMs and LRMs for reasoning abilities, especially since the release of DeepSeek-R1, including foundational components, core problems, training resources, and downstream applications, to identify future opportunities and directions for this rapidly evolving area. We hope this review will promote future research on RL for broader reasoning models. Github: https://github.com/TsinghuaC3I/Awesome-RL-for-LRMs

1. Introduction

RL with verifiable rewards has shifted LLM development toward explicitly training and scaling long-form reasoning, as demonstrated by OpenAI o1 and DeepSeek-R1. This survey reviews the field’s methods, unresolved scaling challenges, resources, applications, and future directions toward artificial superintelligence.

  • Emergence of RL for LRMs: RLVR rewards such as mathematical correctness and code unit-test pass rates enable long-form reasoning, including planning, reflection, and self-correction.OpenAI reports that o1 improves with both additional RL train-time compute and increased inference-time thinking compute.
  • Emergence of RL for LRMs: LRMs treat reasoning as an explicitly trainable capability by allocating test-time compute to generate, evaluate, and revise intermediate chain-of-thought.Performance rises as the test-time compute budget increases, providing a path complementary to data and parameter scaling during pre-training.
  • Scaling challenges: Scaling RL for LRMs remains unresolved because it introduces constraints in computational resources, algorithm design, training data, and infrastructure.The survey frames enhanced RL scalability toward artificial superintelligence as a timely research objective.
  • Survey scope: The survey reviews RL foundations for LRMs, including reward design, policy optimization, sampling strategies, and controversial questions about RL, SFT, priors, recipes, and rewards.It also outlines preliminary RL modeling definitions and the development of frontier reasoning models since OpenAI o1.
  • Survey scope: The survey covers RL training resources, applications, and future directions, spanning corpora, environments, infrastructure, coding, agents, multimodality, multi-agent systems, robotics, and medicine.It notes that resources are reusable in research and production but require further standardization and development.

2. Preliminaries

This section formalizes reinforcement learning for language models by mapping standard MDP components to prompts, generated tokens, deterministic state transitions, and rewards. It also situates RL-trained reasoning models across language, agentic, and multimodal directions while framing the survey’s focus on RL throughout the LLM training lifecycle.

  • RL fundamentals: RL models sequential decision making as an agent interacting with an environment to maximize cumulative reward.The standard MDP is defined by state and action spaces, transition dynamics, rewards, and a discount factor.
  • RL fundamentals: In language-model RL, the prompt is the initial context, the policy generates a token sequence, and each state combines the prompt with tokens generated so far.The policy is parameterized as π_θ, and the state is s_t = (x, a_1:t−1).
  • RL fundamentals: Actions may represent whole sequences, individual tokens, or segments, while language-model transitions append the selected action deterministically until an EOS terminal state.This formulation supports sequence-level, token-level, and step-level reward assignment.
  • RL fundamentals: Training maximizes expected cumulative reward over the data distribution and commonly regularizes the learned policy toward a reference policy with KL constraints.The regularization is intended to stabilize training and maintain language quality.
  • Reasoning-model landscape: RL-trained reasoning models are organized into three directions: large reasoning models, agentic reasoning models, and multimodal reasoning models.The survey describes RL as expanding reasoning-model capabilities and applications in mathematics, coding, science, agentic tasks, and multimodal domains.
  • Survey scope: The survey places RL at the center of the LLM training lifecycle, covering reward design, policy optimization, and sampling strategies to identify paths for scaling LRMs toward ASI.Its stated focus includes long-term interactions and evolution.

3. Foundational Components

This section reviews the foundational components of RL for LRMs: reward design, policy optimization algorithms, and sampling strategies, organized in a taxonomy shown in Figure 5.

  • Reward design: Reward design is reviewed as one of the foundational components of RL for LRMs.
  • Policy optimization algorithms: Policy optimization algorithms are reviewed as a foundational component of RL for LRMs.
  • Sampling strategies: Sampling strategies are reviewed as a foundational component of RL for LRMs within the taxonomy presented in Figure 5.

3.1. Reward Design

Reward design for reinforcement learning in large reasoning models centers on reliable rule-based signals for verifiable tasks and generative signals for subjective or difficult-to-verify tasks. Dense, turn-level, and unsupervised rewards further improve credit assignment, optimization, and scalability while reducing reliance on human annotations.

  • Rule-based Rewards: Rule-based rewards provide scalable, reliable signals for mathematical and coding reasoning through accuracy and format checks.Accuracy rewards compare delimited mathematical answers with ground truth or use unit tests and compilers for code; format rewards enforce structured reasoning and answer fields.
  • Rule-based Rewards: Efficient RL is best supported by tasks with clear ground truth, rapid automated verification, scalable candidate evaluation, and rewards aligned with correctness.Mathematical problem solving and code generation satisfy these criteria because they are difficult to solve but comparatively easy to verify.
  • Generative Rewards: Generative reward models extend RL to subjective or non-verifiable domains by producing structured critiques, rationales, preferences, and more nuanced feedback.They also improve robustness for difficult-to-parse verifiable tasks, support hybrid reward schemes, and can provide process-level feedback for credit assignment.
  • Dense Rewards: Dense rewards provide fine-grained credit assignment that improves training efficiency and optimization stability, while turn-level rewards bridge process- and outcome-based supervision in multi-turn agent RL.Turn-level rewards may be assigned directly at each interaction or derived through outcome decomposition.
  • Unsupervised Rewards: Unsupervised rewards eliminate the human-annotation bottleneck by generating reward signals at the scale of computation and data, though both model-specific and model-agnostic approaches remain susceptible to reward hacking.Model-specific methods leverage internal knowledge, whereas model-agnostic methods use external automated feedback.

3.2. Policy Optimization

Policy optimization for reasoning LLMs primarily uses first-order policy-gradient methods to maximize expected cumulative reward, with PPO commonly used for improved stability. The survey contrasts critic-based and critic-free approaches, highlighting critic-free scalability in rule-based settings and KL-based stability controls.

  • Policy-gradient formulation: RL policy optimization for LLMs mostly uses first-order gradient-based algorithms to maximize expected cumulative reward.The context is treated as the environment, while next-token prediction probabilities define the policy.
  • Policy-gradient formulation: The advantage function measures an action’s improvement over expected reward, while vanilla REINFORCE uses the full trajectory reward as its estimate.A(s, a) = Q(s, a) − V(s), and with trajectory-level rewards, A_t = R(x, y).
  • Policy-gradient formulation: PPO is commonly used instead of vanilla policy gradients because long-chain reasoning creates stability challenges.PPO removes TRPO’s trust-region constraint while retaining a computationally efficient clipped-surrogate formulation and strong sample complexity.
  • Critic-based and critic-free optimization: Critic-free methods require only sequence-level rewards, reducing computation and simplifying training relative to critic-based methods.In rule-based environments, clearly defined rewards can also avoid reward hacking caused by an ill-trained critic, making critic-free approaches more scalable.
  • Stability and scalable optimization: KL penalties help prevent destructive updates, maintain stability, and avoid entropy collapse, but their mechanisms for scalable RL remain under exploration.Periodic reference-policy resets can reduce excessive constraints from the KL term, while KL variants may substitute for PPO’s clipped policy loss.

3.3. Sampling Strategy

Sampling strategy is a first-class lever for RL fine-tuning of reasoning LLMs, balancing informative exploration, sampling efficiency, and structured reuse. Recent hyperparameter methods jointly tune exploration, staged length curricula, and sequence management for scalable GRPO/PPO-style training.

  • Dynamic Sampling: Dynamic sampling allocates prompts and compute using online signals such as success rate, advantage, uncertainty, or difficulty, emphasizing informative examples.Efficiency-oriented methods filter overly easy, difficult, saturated, or degenerate prompts, while exploration-oriented methods use entropy-guided rollouts and attention-based branching.
  • Structured Sampling: Structured sampling organizes reasoning traces as trees or shared prefixes, enabling node-level rewards, partial-computation reuse, and improved sample efficiency.MCTS-based methods provide fine-grained process signals, while segmented or shared-prefix schemes address their generation-efficiency limitations.
  • Hyperparameter Adjustment: Hyperparameter adjustment balances exploration and exploitation through temperature, entropy regularization, and PPO clipping, alongside efficient sequence-length management.The surveyed strategies target competing objectives of discovering effective reasoning paths, refining high-reward solutions, and controlling computational cost.
  • Hyperparameter Adjustment: Staged context lengthening starts RL with shorter windows and progressively increases them, while inference-time extrapolation can extend models trained on shorter sequences.The passage gives an example progression from 8k to 16k and 24k context lengths and identifies Yarn as an inference-time extrapolation technique.
  • Hyperparameter Adjustment: Effective adjustment jointly tunes exploration, staged length curricula, and sequence management, including overlength filtering, penalties, or inference-time extrapolation.These methods are presented as directly applicable to most GRPO/PPO-style RL pipelines for LLMs.

4. Foundational Problems

The section examines whether RL primarily sharpens latent capabilities or enables discovery, arguing that both can occur depending on exploration, exploitation, training duration, regularization, model priors, and reward design. It further identifies generalization, evaluation, data distribution, and hybrid training as foundational challenges.

  • Sharpening versus Discovery: RL initially produced “Aha” behaviors and reflection or verification skills, but early Pass@K evidence suggested it mainly narrows the search space rather than discovering novel solution trajectories.TinyZero reproduced similar phenomena with simplified recipes, while Logic-RL transferred rule-based reflection and verification to mathematical reasoning.
  • Sharpening versus Discovery: ProRL and related studies report that sufficiently prolonged, stabilized RL can extend the reasoning frontier, improving both Pass@1 and Pass@K through emergent compositions of existing capabilities.ProRL v2 adds engineering advances and demonstrates stronger results.
  • Sharpening versus Discovery: A unified view attributes sharpening to reverse-KL mode seeking and discovery to implicit reward learning plus sequential composition under sufficient training time and appropriate regularization.The proposed framing shifts the debate toward conditions determining which phenomenon dominates.
  • Generalization: RL generally preserves or improves broader reasoning and OOD performance, whereas math-focused SFT can cause representation drift, memorization, negative transfer, and catastrophic forgetting.Long-CoT SFT and rule-based RL with format and correctness rewards can expand reasoning depth and self-reflection.
  • Generalization: RL is not a panacea: its generalization depends strongly on initial data distributions and verification rewards, and it may fail under severe overfitting or abrupt distribution shifts.Unresolved issues include distinguishing genuine problem solving from memorization, preventing contamination, and establishing standardized reproducible OOD benchmarks.
  • Training Foundations: RL performance depends on powerful model priors, verifiable rewards, and model family choice, motivating hybrid or alternating RL-SFT paradigms and greater focus on environment and evaluation design.Base models can provide smoother RL improvement than heavily aligned instruct models, while one-shot RLVR can more than double MATH500 accuracy for Qwen2.5-Math-1.5B.

5. Training Resources

Training resources for RL with LLMs are shifting from scale-first static corpora toward high-quality, verifiable supervision and broader coverage of math, code, STEM, search, tool use, and agentic tasks. Because static datasets are increasingly insufficient for advanced generalizable reasoning, dynamic environments provide scalable data synthesis and step-level, multi-turn feedback, including code interaction and task-scaled reasoning generation.

  • Static RL Datasets: RL reasoning datasets are shifting from large raw corpora toward high-quality, verifiable supervision through distillation, filtering, and automated evaluation.This shift aims to improve sample effectiveness and process fidelity while supporting verifiable rewards.
  • Static RL Datasets: Dataset coverage spans mathematics, coding, STEM, search, tool use, and agentic tasks with traceable reasoning, planning, acting, and verification trajectories.These resources support policy pretraining, reward modeling, difficulty-aware sampling, and process-level evaluation.
  • Dynamic RL Environments: Static RL corpora are increasingly insufficient for advanced, generalizable reasoning, motivating synthesized data and interactive environments that jointly provide scalability and verifiability.Dynamic environments can automate limitless data synthesis or deliver step-level, multi-turn feedback on complete reasoning processes.
  • Dynamic RL Environments: Code-based environments require models to interact with compilable code during training, making scalable construction of software-engineering and tool-interaction tasks a significant research direction.ReCall constructs a Python-based tool interaction environment with advanced LLMs.
  • Dynamic RL Environments: Over 1000 general reasoning tasks across eight domains use difficulty-controllable generators and rule-based verifiers, while Synthetic-2 provides four million verified reasoning traces.The reported task-scaling result shows that increasing the number of training tasks significantly boosts reasoning performance and training efficiency.

6. Applications

RL applications span coding, autonomous agents, and multimodal reasoning, improving verifiable code generation, adaptive interaction, and data-efficient vision-language-action learning while leaving scalability and real-world deployment challenges.

  • Coding Tasks: RL advances code reasoning and generation across competitive programming and domain-specific tasks, supporting progress toward agentic, closed-loop coding.Research is organized around code generation, software engineering assistance, and agentic coding; scalability, cross-task generalization, and robust large-scale automation remain open challenges.
  • Agentic Behaviors: RL-trained agents extend language models toward autonomous research and GUI interaction through iterative tool use and fine-grained rule-based rewards.DeepResearch systems gather online information for real-world problems, while GUI agents optimize task-specific actions using rule-based rewards and limited human-curated data.
  • Multimodal Reasoning: RL strengthens multimodal models for limited-data settings, long-video reasoning, and numerically or attribute-sensitive cross-modal generation.Multimodal RL also supports interpretable reasoning, reduces hallucinations in long text-based chains of thought, and improves image captioning through language-model-derived rewards.
  • Vision-Language-Action Models: Minimal-supervision RL enables vision-language-action models to learn from environment interaction and simple rewards, achieving superior performance and novel behaviors.SimpleVLA-RL uses binary success/failure rewards with GRPO and, from one demonstration trajectory, surpasses state-of-the-art VLA models on LIBERO and RobotWin2.0.
  • Vision-Language-Action Models: VLA RL remains constrained by multi-round trajectories, continuous actions, reward design, simulation dependence, sample efficiency, reward sparsity, and sim-to-real transfer.Few studies deploy physical robots to collect real-world RL trajectories, although SimpleVLA-RL achieved real-world deployment through sim-to-real transfer.
  • Medical Applications: Rule-based rewards support multimodal medical reasoning, but medical generation lacks unique ground-truth answers, making large-scale RL on non-verifiable tasks an emerging direction.Medical applications include radiology reports, clinical dialogue, treatment planning, and diagnostic narratives, where preference optimization has also been applied.

7. Future Directions

The section presents a roadmap for RL research on LLMs, emphasizing continual adaptation, enhanced reasoning, new reasoning paradigms, scientific applications, and architecture–algorithm co-design. It identifies directions spanning continual, memory-based, model-based, efficient, latent-space, pre-training, diffusion-based, and discovery-oriented RL.

  • 7. Future Directions: Continual RL is highlighted as a direction for adapting LLMs to evolving data and tasks.The paper identifies continual RL as a promising area for the next wave of advances.
  • 7. Future Directions: Memory-based and model-based RL are proposed for enhancing LLM reasoning capabilities.These approaches are discussed as distinct future directions in the survey.
  • 7. Future Directions: Emerging approaches aim to teach LLMs efficient reasoning and latent-space reasoning.The section groups these approaches as complementary directions for advancing reasoning.
  • 7. Future Directions: Future work also includes RL during pre-training, RL for diffusion-based architectures, scientific discovery, and architecture–algorithms co-design.The co-design direction addresses the demands of ever-larger and high-efficiency intelligent models.

7.1. Continual RL for LLMs · 7.2. Memory-based RL for LLMs

The sections identify continual RL challenges in balancing stability, plasticity, and entangled knowledge-reasoning, while highlighting memory-based RL as an underexplored route for using experiential traces and improving cross-task generalization.

  • 7.1. Continual RL for LLMs: Mixing data from different tasks in unified RL training can outperform multi-stage RL on synthetic data, while increasing-difficulty curricula may be unnecessary.The cited synthetic-data studies report multi-stage RL performing worse than mixed-data training.
  • 7.1. Continual RL for LLMs: Continual RL for LLMs must balance stability and plasticity, with plasticity especially concerning because deep-learning techniques can impair continual-learning performance.The passage also distinguishes LLM continual learning through the entanglement of knowledge and reasoning.
  • 7.1. Continual RL for LLMs: Experience Replay, Policy Reuse, and Reward Shaping from traditional continual RL provide methodological foundations for frameworks tailored to LRMs.These approaches are presented as core insights for addressing LLM-specific continual-RL requirements.
  • 7.2. Memory-based RL for LLMs: Agentic RL has explored external long-term storage, insertion, internal memory processing, and working-memory control, but most mechanisms remain task-specific.The passage emphasizes limited generalization beyond the current task.
  • 7.2. Memory-based RL for LLMs: An experience-centric view aligns naturally with RL because agent-environment interactions generate experiential traces that can be effectively utilized.This frames interaction data as a central resource for memory-based RL.
  • 7.2. Memory-based RL for LLMs: Shared experience pools can retrieve relevant strategies from past histories and adapt other agents’ experiences to new tasks, but this direction remains underexplored.The passage identifies automatic learning of memory operation and management as a core challenge.

7.3. Model-based RL for LLMs

Model-based RL addresses the challenge of obtaining scalable, robust reward signals and meaningful state representations by using world models, including LLMs, to provide informative states for RL agents.

  • 7.3. Model-based RL for LLMs: World models have been investigated to supply informative states for RL agents, with LLMs more recently adopted as world models across various RL contexts.This line of work is especially relevant to RL with LLMs and language agents.

7.4. Teaching LRMs Efficient Reasoning · 7.5. Teaching LLMs Latent Space Reasoning

The sections examine adaptive compute allocation for efficient reasoning and the integration of reinforcement learning with latent-space reasoning. Both directions remain open challenges: principled resource rationality for LRMs and reliable supervision for continuous latent thought.

  • 7.4. Teaching LRMs Efficient Reasoning: Inference-time scaling can cause over-thinking on easy instances and under-thinking under aggressive truncation, motivating adaptive reasoning depth and halting.Over-thinking involves needlessly long reasoning chains, while under-thinking involves premature halting and brittle shortcuts.
  • 7.4. Teaching LRMs Efficient Reasoning: RL research explores hard-coded reasoning levels, adaptive length-based reward shaping, and length penalties in the loss function.These methods target compute allocation by adjusting reasoning length or explicitly penalizing excessive length.
  • 7.4. Teaching LRMs Efficient Reasoning: Developing a principled cost-performance trade-off remains open, making resource-rational reasoning a central unsolved RL problem.The goal is for models to reason longer only when the marginal utility justifies the additional computation.
  • 7.5. Teaching LLMs Latent Space Reasoning: Chain-of-thought prompting encourages step-by-step reasoning by eliciting intermediate steps, improving interpretability and accuracy.Recent work combines CoT and RL by sampling long-form thought before answering for model training.
  • 7.5. Teaching LLMs Latent Space Reasoning: Current CoT-and-RL implementations often use token-level sampling in a discrete scalar space, whereas latent-space reasoning offers smoother learning dynamics and stronger RL integration potential.The passage frames RL combined with latent-space reasoning as a route toward more powerful and adaptable reasoning models.
  • 7.5. Teaching LLMs Latent Space Reasoning: Evaluating continuous latent thought is harder than evaluating token-based thought, complicating accurate rewards and advantages.Providing reliable supervisory signals for latent reasoning remains an open challenge.

7.6. RL for LLMs Pre-training

Research is shifting reinforcement learning earlier than post-training, exploring RL during pre-training and even RL-from-scratch language-model training. These approaches reinterpret next-token learning or bootstrap token-level rewards while leveraging verification, referee scoring, or previously acquired computational knowledge.

  • 7.6. RL for LLMs Pre-training: RL research is moving beyond post-training by applying reinforcement learning during LLM pre-training.This extends the traditional paradigm of large-corpus next-token prediction, whose scaling has been central to foundation-model development.
  • 7.6. RL for LLMs Pre-training: Reinforcement Pre-Training reconceptualizes next-token prediction as a reinforcement-learning problem with verifia…
  • 7.6. RL for LLMs Pre-training: avataRL trains language models from random initialization purely with RL by bootstrapping token-level rewards and using iterative “referee” scoring.The initiative illustrates a concrete path toward RL-from-scratch training and aligns with reincarnated RL, which leverages a pre-trained critic’s computational knowledge.

7.7. RL for Diffusion-based LLMs

RL for diffusion large language models is an emerging research direction motivated by DLLMs’ decoding efficiency and self-correction potential. Its central challenges are estimating sampled-response log probabilities and designing rewards that guide denoising trajectories.

  • Motivation: Diffusion large language models offer superior decoding efficiency and greater potential for self-correction through multiple diffusion rounds compared with autoregressive models.Initial efforts have begun exploring reinforcement learning for DLLMs.
  • Challenges: Accurately and efficiently estimating sampled-response log probabilities is a central challenge because DLLMs model sample likelihoods differently from autoregressive models.Autoregressive models factorize joint probabilities through next-token prediction and the chain rule, whereas DLLMs approximate likelihood optimization differently.
  • Challenges: Multiple feasible decoding trajectories create an opportunity for RL to guide DLLMs toward optimal sampling traces.This introduces an additional research dimension beyond selecting final responses.
  • Reward Design: Intermediate denoising rewards can be designed through reward models, prefix-conditioned token-wise advantages, or edit-distance-based objectives for decoding efficiency.He et al. formulate denoising as a multi-step decision problem, Wang et al. propose diffusion-based value modeling for trajectory-level rewards, and Song et al. use edit-distance rewards.

7.8. RL for LLMs in Scientific Discovery

Reinforcement learning is improving LLM performance on reasoning-intensive scientific tasks, including biology and chemistry, but scalable result verification remains a central challenge. Existing lab-in-the-loop systems provide sparse, delayed, and costly feedback, motivating better reward design, oracle models, and RL environments.

  • Scientific reasoning: RL improves LLM performance on reasoning-heavy scientific tasks and can sometimes surpass specialized methods.The surveyed applications include biology and chemistry.
  • Verification: Scalable result verification is a core RL challenge in biology and chemistry because it conventionally depends on wet-lab experimentation.Several methods seek to replace or support this conventional verification process.
  • Training environments: Lab-in-the-loop systems such as Coscientist and Robin have succeeded, but their sparse, delayed, and costly feedback is impractical for directly training the underlying LLM.The broader challenge is constructing RL environments that enable rapid experimentation-feedback loops.
  • Future directions: Future work should improve reward formulation, oracle models, and RL environments that support rapid experimentation-feedback loops.These directions address the difficulty of obtaining practical verification signals for scalable training.

7.9. RL for Architecture-Algorithm Co-Design

This section proposes treating architecture as a first-class action space in RL, enabling LLMs to jointly optimize task performance, hardware-aware objectives, and input-dependent modular adaptation. It contrasts reinforced MoE with fixed-architecture neural architecture search and identifies robust architecture–algorithm co-design as an open challenge.

  • Motivation: Current RL pipelines optimize task accuracy on dense Transformer or MoE backbones while leaving architectural choices and hardware implications outside the learning loop.The section highlights hardware–architecture co-design as an emerging direction.
  • Architecture as Action Space: Architecture should become a first-class RL action space, allowing reinforced MoE to learn routing policies, expert activation, capacity allocation, and sparsity patterns.This reframes architecture decisions as part of the RL process rather than fixed design choices.
  • Hardware-Aware Adaptation: RL can optimize task reward alongside latency, memory traffic, energy consumption, and activation budgets while adapting topology to prompt difficulty and real-time compute constraints.The model is framed as reasoning across parameters and modules, not only tokens.
  • Contrast with NAS: Unlike classic NAS, which typically finds a fixed architecture for a task or dataset, reinforced MoE optimizes routing and modular adaptation per input during inference.This distinction suggests greater efficiency and flexibility from input-dependent adaptation.

8. Conclusion

The survey examines how reinforcement learning advances reasoning in large reasoning models, with particular emphasis on RLVR for transforming LLMs into LRMs. It organizes RLVR around reward design, policy optimization, and sampling strategies while surveying research directions in each area.

  • Conclusion: The survey focuses on reinforcement learning for reasoning in large reasoning models and its role in transforming LLMs into LRMs.The paper emphasizes recent advances in this area.
  • Conclusion: Unlike RLHF and DPO, which primarily target human alignment, the survey centers on RLVR for LLMs.RLVR provides direct outcome-level rewards for reasoning.
  • Conclusion: The survey organizes RLVR research around reward design, policy optimization, and sampling strategies, summarizing existing work across each direction.These are presented as the core components of RLVR.
Loading 2509.08827v3…