A Survey of Reinforcement Learning for Large Reasoning Models

Motivation

• RLVR, including GRPO, enables LLMs to perform long-form reasoning through verifiable rewards for mathematical correctness and coding tests, even from base models [Jaech et al., 2024, Guo et al., 2025a].OpenAI o1 scales with additional RL and inference-time thinking, while DeepSeek-R1 uses rule-based, compiler-based, or test-based rewards.Source: §1 · p. 4
• Reasoning-focused RL adds a scaling axis beyond pre-training by increasing test-time compute, using verifiers, and generating training data through self-play or self-improvement [OpenAI, 2025a,b].The survey frames this path as complementary to data and parameter scaling and relevant to continual progress toward ASI.Source: §1 · p. 4

Open challenges

• Scaling RL for LRMs remains constrained by computational resources, algorithm design, training data, and infrastructure, with its path to high-level intelligence unresolved.The survey therefore emphasizes strategies for making RL more scalable toward artificial superintelligence.Source: §1 · p. 4

Survey scope

• The survey establishes LRM definitions and development history, then examines reward design, policy optimization, and sampling strategies across RL-for-LRM research.These foundational components are introduced alongside the field’s progression since OpenAI o1.Source: §1 · pp. 4–5

Challenges and resources

• It analyzes controversial questions about RL’s role, RL versus SFT, model priors, training recipes, and reward definitions, alongside reusable but insufficiently standardized training resources.Resources include static corpora, dynamic environments, and infrastructure for both research and production.Source: §1 · p. 5

Applications and future directions

• Finally, the survey reviews RL applications in coding, agentic, multimodal, multi-agent, robotic, and medical tasks, and discusses future algorithms, mechanisms, features, and research avenues.These applications and directions define the review’s downstream and forward-looking coverage.Source: §1 · p. 5

2.1. Background

• RL frames language models as agents that generate actions sequentially to maximize cumulative reward, with prompts, tokens, states, transitions, and rewards mapped from the MDP formulation.Actions may be sequences, tokens, or segments; rewards may be assigned at sequence, token, or step level, and KL regularization stabilizes training against a reference policy.Source: §2.1 · pp. 5–6

2.2. Frontier Models

• RL-trained reasoning models have rapidly expanded from o1 and DeepSeek-R1 to open and proprietary language, agentic, and multimodal systems with advances in reasoning, coding, tool use, and long-context capabilities.The survey organizes these developments chronologically across LRMs, agentic LRMs, and multimodal LRMs, with representative models and training algorithms compiled in Figure 4 and Table 1.Source: §2.2 · p. 7, Figure 4 · p. 8, Table 1 · p. 8

2.3. Related Surveys

• Unlike surveys centered on general RL, reasoning, reward design, or alignment, this work systematically synthesizes RL across reward design, policy optimization, and sampling for scaling LRMs toward ASI.Its focus includes the full LLM training lifecycle and long-term interactions and evolution.Source: §2.3 · p. 10

• The foundational components are reward design (§ 3.1), policy optimization algorithms (§ 3.2), and sampling strategies (§ 3.3).Source: §3 · p. 11
• Figure 5 presents the taxonomy of these foundational components.Source: §3 · p. 11

3.1.1. Verifiable Rewards

• Rule-based verifiable rewards provide scalable, reliable signals for mathematics and coding through accuracy checks, unit tests, and format constraints [Guo et al., 2025a, Team, 2025c, Yu et al., 2025d].RLVR replaces learned reward models with programmatic verifiers, while accuracy and format rewards support reliable parsing and verification; objectively verifiable tasks remain easiest to optimize.Source: §3.1.1 · pp. 11, 13

3.1.2. Generative Rewards

• Generative reward models extend RL to subjective or difficult-to-parse tasks by producing structured critiques, rationales, preferences, and nuanced feedback [Mahan et al., 2024, Zhang et al., 2024a].Reasoning RMs, rubric-based evaluation, and co-evolving policy–reward systems increasingly support dynamic judgment, hybrid rewards, and process-level feedback for credit assignment.Source: §3.1.2 · pp. 13–15

3.1.3. Dense Rewards

• Dense rewards improve credit assignment, sample efficiency, and optimization stability by supervising tokens, steps, or turns rather than only final outcomes [Lightman et al., 2024, Uesato et al., 2022].Turn-level rewards bridge process- and outcome-based supervision in multi-turn agent RL, while model-based and sampling-based methods address fine-grained process estimation but face reward hacking and computational costs.Source: §3.1.3 · pp. 15–17

3.1.4. Unsupervised Rewards

• Unsupervised rewards eliminate human-annotation bottlenecks by deriving signals from model consistency, confidence, self-generated knowledge, heuristics, or external data [Zhang et al., 2025r].Model-specific methods are highly scalable but risk reward hacking and collapse, whereas model-agnostic methods depend on proxy quality; hybrid systems are proposed for scalable RL.Source: §3.1.4 · pp. 17–19

3.1.5. Rewards Shaping

• Reward shaping transforms sparse feedback into more stable, informative learning signals by combining verifiers with reward models and using group baselines or Pass@K-aligned objectives.These strategies aim to stabilize training, expand exploration, and align optimization with evaluation metrics at scale.Source: §3.1.5 · p. 19
• Reward shaping modifies or augments reward functions to encourage behaviors that advance desired solutions, complementing verifiable, generative, dense, and unsupervised rewards [Goyal et al., 2019, Gupta et al., 2022, Hu et al., 2020, Xie et al., 2023].The section distinguishes rule-based and structured-based reward shaping.Source: §3.1.5 · p. 19
• Rule-based shaping combines verifier, reward-model, outcome, format, or length rewards to rank correct responses, produce informative gradients, and support long chain-of-thought reasoning [Yang et al., 2024a, Guo et al., 2025a].Dynamic reward weighting using hypervolume-guided adaptation and gradient-based optimization improves multi-objective alignment, while multi-role training assigns distinct rewards to roles such as solver and critic [Lu et al., 2025f, Li et al., 2025k].Source: §3.1.5 · pp. 19–20
• Structure-based shaping uses group-level baselines and credit allocation to align optimization with set-level metrics such as Pass@K, stabilize training, and encourage broader exploration [Shao et al., 2024].Methods derive low-variance, unbiased gradients or efficient individual-sample approximations for set-level objectives, reducing premature convergence to suboptimal local solutions [Walder and Karkhanis, 2025, Chen et al., 2025z].Source: §3.1.5 · p. 20

3.2.1. Policy Gradient Objective

• Policy optimization for LLMs maximizes expected cumulative reward with first-order gradients, typically using PPO-style updates because vanilla policy gradients can be unstable.The section defines prompts, generated action sequences, rewards, value and quality functions, and advantage estimation as foundations for comparing algorithms.Source: §3.2.1 · pp. 20–21

3.2.2. Critic-based Algorithms

• Critic-based methods scale token-level supervision through learned value signals, but require a separate critic model that adds substantial computational overhead.GAE-based PPO and alternatives such as Value-Calibrated PPO, VAPO, VRPO, Implicit PRM, and PRIME address long-chain or noisy-reward challenges.Source: §3.2.2 · p. 23

3.2.3. Critic-Free Algorithms

• Critic-free methods use sequence-level rewards without a separate critic, reducing computation and avoiding critic-induced reward hacking in rule-based environments, which improves scalability.GRPO estimates a shared group-relative advantage for all response tokens, while related methods modify sampling, clipping, normalization, or importance weighting to improve stability.Source: §3.2.3 · pp. 23–24

3.2.4. Off-policy Optimization

• Off-policy optimization separates data-generating and target policies, improving sample efficiency through historical, asynchronous, or offline data while requiring correction for distributional shift.Practical forms include precision-mismatched training and inference, replay-buffer-based asynchronous learning, optimizer-level methods, offline learning, and SFT-RL hybrids.Source: §3.2.4 · pp. 25–27

3.2.5. Regularization Objectives

• Regularization stabilizes long-horizon policy optimization through KL and entropy controls, but the optimal form and strength remain unresolved.KL penalties may preserve knowledge and prevent drift, yet many reasoning-RL studies remove them to enable exploration, simplify implementation, reduce memory, and scale GRPO.Source: §3.2.5 · pp. 27–28
• Entropy regularization is contested: standard or targeted objectives may help, but can cause instability or collapse, especially with sparse rewards [Shrivastava et al., 2025, Wu et al., 2025e, An et al., 2025, Liao et al., 2025b].Entropy collapse without intervention can hinder policy exploration [Cheng et al., 2025a, Cui et al., 2025b, Yu et al., 2025d].Source: §3.2.5 · p. 28
• Entropy-control methods dynamically adjust entropy coefficients, update low-probability or high-entropy tokens, incorporate entropy into advantages, or constrain tokens with exceptionally high covariance [He et al., 2025d, Yu et al., 2025d, Wang et al., 2025n, Cheng et al., 2025a, Chen et al., 2025j, Cui et al., 2025b].Cui et al. [2025b] identify covariance between action probability and advantage as the entropy driver and propose Clip-Cov and KL-Cov.Source: §3.2.5 · p. 28
• Length-penalty objectives reduce long-CoT inference costs while retaining performance through user-specified limits, relative-length regularization, accuracy constraints, and difficulty-conditioned penalties [Agarwal et al., 2025a, He et al., 2025e, Aggarwal and Welleck, 2025, Yuan et al., 2025a, Luo et al., 2025a, Xiang et al., 2025, Liu et al., 2025b].Adaptive penalties are conditioned on problem difficulty to preserve model ability [Xiang et al., 2025, Liu et al., 2025b].Source: §3.2.5 · p. 29

3.3.1. Dynamic and Structured Sampling

• Dynamic sampling concentrates computation on informative prompts while balancing exploration against sampling efficiency through difficulty-, success-, uncertainty-, and advantage-aware selection.Efficiency-oriented methods filter saturated or unproductive prompts and prioritize under-mastered items, while exploration-oriented methods increase trajectory diversity, branch at informative reasoning steps, or guide hard-question rollouts.Source: §3.3.1 · pp. 29–30
• Structured sampling organizes reasoning traces as trees or shared segments, enabling node-level process rewards, prefix reuse, and improved sample efficiency under compute constraints.MCTS-based methods such as TreeRL [Hou et al., 2025] and ToTRL [Wu et al., 2025c] enrich search and reward granularity, while shared-prefix and segment-wise schemes reduce generation overhead and KV-cache burden.Source: §3.3.1 · pp. 30–31

3.3.2. Sampling Hyper-parameters

• Hyperparameter adjustment jointly manages exploration–exploitation dynamics and sequence length, since naive settings can cause inefficiency or unstable training such as entropy collapse.Key controls include temperature, entropy regularization, PPO clipping, context-length curricula, and overlength handling.Source: §3.3.2 · pp. 31–32
• Staged context lengthening typically starts RL with an 8k window before expanding to 16k, 24k, or 32k, encouraging concise reasoning while controlling computational cost.Inference-time length extrapolation offers an alternative, while overlong responses may be filtered at short budgets or penalized at larger budgets.Source: §3.3.2 · pp. 31–32
• Effective sampling hyperparameters combine exploration controls, staged length curricula, and sequence-management rules across most GRPO/PPO-style RL pipelines for LLMs.The surveyed strategies jointly tune temperature, entropy targets, clipping, context length, and overlength filters, penalties, or inference-time extrapolation.Source: §3.3.2 · p. 32

4. Foundational Problems

• The section frames RL for LRMs as a field whose unresolved foundations concern sharpening versus discovery, generalization versus memorization, model priors, training reliability, and reward granularity.These questions motivate further investigation into conditions, evaluation protocols, verifiable rewards, and scalable training design.Source: §4 · p. 32

4.1. RL’s Role: Sharpening or Discovery

• RL can narrow the search space toward high-performance regions: Limit-of-RLVR improves Pass@1 yet underperforms base models at large-k Pass@K, whereas prolonged, stabilized RL can extend the reasoning frontier.ProRL [Liu et al., 2025] improves both Pass@1 and Pass@K, and Yuan et al. [2025c] show that models can learn new skills through composition of existing capabilities.Source: §4.1 · p. 33

4.2. RL vs. SFT: Generalize or Memorize

• RL tends to achieve “true generalization” on verifiable tasks and under substantial distribution shifts, but initial data, verification-reward design, severe overfitting, and abrupt shifts limit it; hybrid paradigms combine both approaches [Chen et al., 2025c,h, Liu et al., 2025a, Lv et al., 2025, Wu et al., 2025i, Zhu et al., 2025e].Long-CoT SFT and rule-based RL with format/correctness rewards improve broader reasoning, while an SFT warmup before RL boosts cross-domain transfer. Modified SFT can help bridge remaining gaps in generalization.Source: §4.2 · pp. 34–35

4.3. Model Prior: Weak and Strong

• Model priors strongly shape RL: base models often provide smoother starting points, Qwen is more RL-responsive than Llama or OLMo, and strong distilled models benefit when curriculum and objectives are carefully designed.Mid-training with annealed learning rates and high-quality mathematics or code data can narrow weak-prior gaps, while strong models require multi-objective control of correctness, format, brevity, and instruction following.Source: §4.3 · pp. 36–37

4.4. Training Recipes: Tricks or Traps

• Training tricks can improve stability and efficiency, but evidence supports minimal, reproducible methods and unified evaluation over universal normalization claims.Discarding all incorrect samples yields the largest performance gains; GRESO speeds rollout 2.4× and overall training 2.0× with minimal loss in performance, while normalization evidence is contradictory.Source: §4.4 · pp. 37–38

4.5. Reward Type: Process or Outcome

• Outcome rewards are scalable for verifiable tasks, whereas process rewards provide denser and more reliable credit assignment but require costly annotations and may generalize poorly across domains.Outcome-only RL can encourage unfaithful reasoning and reward hacking; PRMs are more stable for mathematical reasoning, while combined process modeling or generative verifiers may balance scalability and interpretability.Source: §4.5 · pp. 38–39

5. Training Resources

• RL training resources determine scalability and stability across static corpora, dynamic environments, and specialized infrastructure.The section organizes resources into these three categories and relates them to large-scale RL training pipelines.Source: §5 · p. 39

5.1. Static Corpus

• Static RL datasets are shifting from raw scale toward distilled, filtered, annotated, and automatically evaluated data with verifiable process supervision.Coverage now spans mathematics, coding, STEM, search, tool use, and agentic tasks, supporting policy pretraining, reward modeling, and difficulty-aware sampling.Source: §5.1 · p. 39

5.2. Dynamic Environment

• Static corpora are increasingly insufficient for advanced generalizable reasoning, motivating dynamic environments that provide scalable synthesis and step-level, multi-turn feedback.These environments exclude evaluation-only benchmarks and serve as interaction objects for RL.Source: §5.2 · pp. 42–43
• Interactive environments span rule-based, code-based, game-based, model-based, and ensemble designs for controllable verification, tool interaction, continuous rewards, self-play, and task scaling.InternBootcamp supports over 1000 reasoning tasks across eight domains, while Synthetic-2 provides four million verified reasoning traces.Source: §5.2 · pp. 43–45

5.3. RL Infrastructure

• Open-source RL infrastructure combines distributed training frameworks, inference engines, flexible rollout-training pipelines, and specialized support for agentic, multimodal, and multi-agent RL.AReaL reports up to 2.77× training speedups at matched or better final accuracy and near-linear scaling to 512 GPUs.Source: §5.3 · pp. 45–47

6. Applications

• The application taxonomy spans coding, agentic behavior, multimodal reasoning, multi-agent systems, robotics, and medicine, illustrating broad potential alongside domain-specific requirements.Figure 6 presents the taxonomy and corresponding representative works.Source: §6 · p. 48, Figure 6 · p. 49

6.1. Coding Tasks

• RL advances coding from verifiable code generation toward software-engineering automation and agentic, closed-loop development, but scalability, cross-task generalization, and robust large-scale automation remain open challenges.RL improves repair effectiveness and generalization, while Afterburner raises Pass@1 from 47% to 62%. Repository-level methods integrate retrieval, reflection, testing, and continuous integration for cross-file maintenance.Source: §6.1 · pp. 48, 50, 51

6.2. Agentic Tasks

• RL enables autonomous agents for coding, tool-integrated reasoning, search, browsing, deep research, GUI control, AutoML, and other tasks, but high computation, long rollouts, and limited training data constrain scalability.Asynchronous rollouts and memory agents reduce latency and context-management costs; tool execution supplies verifiable feedback for iterative reasoning, while simulated environments reduce search-training expense.Source: §6.2 · pp. 51–52

6.3. Multimodal Tasks

• RL strengthens multimodal understanding and generation across images, videos, 3D spaces, and diffusion or autoregressive models, especially under limited data, long-video reasoning, and attribute-sensitive generation.Key unresolved issues include inconsistent reasoning and answers, collapse during long-chain exploration, sensitivity to training-data quality, and the need for unified RL frameworks spanning understanding and generation.Source: §6.3 · pp. 54–56
• Multimodal RL increasingly unifies understanding and generation through bidirectional or dual rewards, while multimodal chain-of-thought and chart-to-code reasoning offer promising domain-specific applications.The passage identifies unified post-training for multimodal understanding and generation as an urgent future direction.Source: §6.3 · p. 56

6.4. Multi-Agent Systems

• Multi-agent RL still faces difficult credit assignment, nonstationarity, communication, sample-efficiency, and scalability problems, motivating improved teamwork and interaction mechanisms for complex tasks.CTDE addresses training-time coordination with decentralized execution, but traditional methods remain difficult to apply as agent numbers and task complexity increase.Source: §6.4 · p. 57
• RL for LLM-based multi-agent systems combines language-based information sharing and credit assignment with actor-critic, group-relative, evolutionary, and reasoning-aware collaboration mechanisms.Examples include LLaMAC’s centralized natural-language critic, CTRL’s self-critique with RL, and MAPoRL, MAGRPO, ReMA, JoyAgents-R1, AlphaEvolve, and AutoAgents for multi-turn or specialized collaboration.Source: §6.4 · pp. 57–58

6.5. Robotics Tasks

• Robotics RL adapts LLM-style methods to VLAs to reduce reliance on costly trajectory data and improve generalization, but multi-round interaction, continuous actions, and scalable rewards remain challenges.Most current VLA RL research is simulation-based, with limited physical-robot trajectory collection and persistent issues involving sample efficiency, reward sparsity, Sim2Real transfer, resetting, and annotation.Source: §6.5 · pp. 58–60
• RL-trained VLAs achieve strong performance with minimal supervision, including SOTA results from a single demonstration and improved generalization to unseen environments, objects, and textures.SimpleVLA-RL uses binary success/failure rewards with GRPO, while related work compares GRPO and PPO and reports real-robot gains through Sim2Real transfer.Source: §6.5 · pp. 58–59

6.6. Medical Tasks

• Medical RL is mature for verifiable understanding tasks, where SFT followed by rule-based rewards and methods such as GRPO support correctness, formatting, reasoning, and generalization.Applications cover medical QA, structured prediction, clinical coding, visual grounding, medical imaging, information extraction, clinical reasoning, and code-based medical agents.Source: §6.6 · pp. 60–61
• Medical generation remains underexplored because reports, dialogue, treatment plans, and narratives lack unique ground-truth answers, requiring rubric-based rewards, preference optimization, curriculum methods, or offline RL.Existing systems target treatment planning, diagnostic dialogue, clinical decision support, radiology reports, and interactive clinical simulation, but scalable and stable reward design remains unresolved.Source: §6.6 · pp. 60–61

• Future research should address continual RL for evolving data and tasks, memory-based and model-based RL, and efficient and latent-space reasoning.These directions target adaptation and improved reasoning capabilities in LLMs.Source: §7 · p. 62
• Emerging frontiers include leveraging RL during pre-training, applying RL to diffusion-based architectures, and using RL to drive scientific discovery.Source: §7 · p. 62
• Architecture–algorithm co-design is identified as a direction for meeting the demands of increasingly large and high-efficiency intelligent models.The section aims to provide a roadmap and inspiration for future RL research on LLMs.Source: §7 · p. 62

7.1. Continual RL for LLMs

• Continual RL becomes necessary as LLMs must adapt to evolving data and tasks in dynamic environments, amid debate over mixed-data, multi-stage, and curriculum training [Guo et al., 2025a, Yang et al., 2025a, Chen et al., 2025d, Liu et al., 2025h, Xie et al., 2025c].Mixed-data training can outperform multi-stage RL, although multi-stage training may improve generalization to difficult or unseen problems.Source: §7.1 · p. 62
• LLM continual RL must balance stability and plasticity, while addressing the entanglement of knowledge and reasoning that prevents clean task and policy modularity [Pan et al., 2025a, Dohare et al., 2024].This differs from traditional RL settings with discretely defined tasks and modular policies [Chevalier-Boisvert et al., 2023, Towers et al., 2024, Todorov et al., 2012, Wołczyk et al., 2021].Source: §7.1 · p. 62
• Experience Replay, Policy Reuse, and Reward Shaping from traditional continual RL offer methodological foundations for specialized CRL techniques for LLMs and LRMs [Berseth et al., 2021, Li et al., 2021, Rolnick et al., 2019, Garcia and Thomas, 2019, Gaya et al., 2022, Jiang et al., 2021, Zheng et al., 2022].Such techniques are positioned as important for lifelong learning in dynamic environments.Source: §7.1 · p. 62

7.2. Memory-based RL for LLMs

• Memory mechanisms in agentic RL span external storage and insertion and internal memory processing and working-memory control, but most remain task-specific with limited cross-task generalization [Chhikara et al., 2025, Xu et al., 2025d, Zhong et al., 2024, Yu et al., 2025b, Zhou et al., 2025k].Silver and Sutton [2025] argue that future agents will acquire skills primarily through continual interaction and experience.Source: §7.2 · p. 62
• Memory should evolve from task-specific buffers into structured, reusable, transferable experience repositories that support adaptation and lifelong learning across diverse tasks [Silver and Sutton, 2025].This experience-centric view aligns with RL because agent-environment interactions produce rich experiential traces.Source: §7.2 · p. 62
• RL agents must learn to operate and manage shared experience pools, composing and generalizing experiential knowledge across tasks; despite recent retrieval and adaptation efforts, this direction remains underexplored [Tang et al., 2025].Solving this challenge would support an “experience era” in which collective interaction traces underpin broader agent intelligence.Source: §7.2 · p. 63

7.3. Model-based RL for LLMs

• World models can provide informative environmental states and reliable rewards for LLM-based RL agents, making model-based RL a promising scalable direction.LLMs have been adopted as world models, while generative world models, including video-pretrained variants, have shown practical effectiveness; seamless integration remains an open problem.Source: §7.3 · p. 63

7.4. Teaching LRMs Efficient Reasoning

• Inference-time scaling creates over-thinking on easy instances and under-thinking under aggressive truncation, motivating adaptive depth and halting policies for RL-based reasoning.Over-thinking produces needlessly long chains, whereas under-thinking causes premature halting and brittle shortcuts.Source: §7.4 · p. 63
• Existing efficiency approaches include prompted reasoning levels, adaptive length-based rewards, and loss-function length penalties.These methods aim to allocate reasoning compute according to task demands, but use different intervention points: prompting, rewards, or optimization loss.Source: §7.4 · p. 63
• A principled cost-performance trade-off remains unresolved, including when additional reasoning has sufficient marginal utility to justify its resource cost.The open problem is to make LRMs resource-rational rather than uniformly extending or truncating reasoning.Source: §7.4 · p. 63

7.5. Teaching LLMs Latent Space Reasoning

• CoT improves interpretability and accuracy by eliciting intermediate steps, while RL-based CoT methods sample long-form thought before answering.These approaches extend step-by-step prompting with reinforcement-learning-oriented training of reasoning trajectories.Source: §7.5 · p. 63
• Latent Space Reasoning (LSR) [Arriola et al., 2025, Geiping et al., 2025, Hao et al., 2024] performs reasoning in continuous latent space, potentially avoiding discrete token-level bottlenecks and enabling smoother RL learning.LSR is described as supporting more nuanced semantic reasoning and better integration with RL techniques.Source: §7.5 · pp. 63–64
• Combining RL with LSR faces an open supervision problem because continuous latent thought is harder to evaluate than token-based thought, complicating accurate rewards and advantages.The challenge is to provide reliable supervisory signals for latent reasoning trajectories.Source: §7.5 · p. 64

• Reinforcement Pre-Training [Dong et al., 2025c] reframes next-token prediction as RL with corpus-derived verifiable rewards and reports gains that increase with available compute.This positions RL as a potential scaling strategy for language-model pre-training.Source: §7.6 · p. 64
• avataRL [tokenbender, 2025] trains language models from random initialization using token-level rewards and iterative “referee” scoring, demonstrating RL-from-scratch pre-training.This approach is consistent with reincarnated RL [Agarwal et al., 2022], which leverages previously acquired computational knowledge through a pre-trained critic.Source: §7.6 · p. 64
• Scaling RL-based pre-training requires reducing verifier burden and reward-engineering costs while developing rewards that remain scalable and reliable.These concerns connect RL-style pre-training to unsupervised reward design discussed in § 3.1.4.Source: §7.6 · p. 64

7.7. RL for Diffusion-based LLMs

• DLLMs offer more efficient decoding and greater potential for self-correction than autoregressive models, while initial RL studies remain exploratory [Borso et al., 2025, Gong et al., 2025, Yang et al., 2025d].DLLMs support multiple rounds of diffusion, creating opportunities for RL-guided generation.Source: §7.7 · p. 64
• Efficient and accurate on-policy ELBO estimation remains unresolved because its timestep- and mask-dependent double expectation requires extensive sampling and otherwise causes high variance.A one-step estimator and sampling allocation strategy [Zhu et al., 2025b] have been proposed to mitigate variance, but the core problem persists.Source: §7.7 · p. 64
• Multiple feasible decoding trajectories create a need for RL methods that guide DLLMs toward optimal sampling traces.This expands RL beyond response-level optimization to trajectory selection during diffusion decoding.Source: §7.7 · p. 64
• Intermediate-step reward design is central: methods use rewards on denoising states, prefix-conditioned token-wise advantages, or edit distance to optimize trajectory quality and decoding efficiency.He et al. [2025c] model denoising as a multi-step decision process; Wang et al. [2025q] use a diffusion-based value model; Song et al. [2025c] use edit-distance rewards.Source: §7.7 · p. 65

• RL can improve LLMs on reasoning-heavy scientific tasks, in some cases surpassing specialized methods [Fallahpour et al., 2025, Fang et al., 2025c, Narayanan et al., 2025, Rizvi et al., 2025].Source: §7.8 · p. 65
• In biology and chemistry, Pro-1 uses a Rosetta energy function to reward protein-stability optimization, while rbio1 verifies gene-perturbation predictions with biological models and external knowledge.Source: §7.8 · p. 65
• Lab-in-the-loop verification offers sparse, delayed, and costly feedback, while in-silico simulations remain too limited, inaccurate, and insufficiently generalizable to replace realistic laboratory environments [Boiko et al., 2023, Ghareeb et al., 2025, Bunne et al., 2024, Noutahi et al., 2025, Ahlmann-Eltze et al., 2025, Kedzierska et al., 2023].Open challenges include reward formulation, oracle-model improvement, and constructing environments with rapid experimentation-feedback loops; domain-specific and generalist models may broaden LLMs toward open-ended scientific discovery [Fallahpour et al., 2025, Bigaud et al., 2025, Narayanan et al., 2025].Source: §7.8 · p. 65

RL for Architecture-Algorithm Co-Design

• Existing RL pipelines use dense Transformer [Vaswani et al., 2017] or MoE backbones [Dai et al., 2024, Jiang et al., 2024, Shazeer et al., 2017] and optimize rewards almost exclusively tied to task accuracy.This leaves architectural degrees of freedom and their hardware implications outside the learning loop, despite hardware–architecture co-design such as DeepSeek’s NSA [Yuan et al., 2025b] and Step-3 [Wang et al., 2025a].Source: §7.9 · p. 65
• Architecture-aware RL could jointly optimize task reward with latency, memory traffic, energy consumption, and activation budgets.Reinforced MoE may learn routing policies, expert activation, capacity allocation, and sparsity patterns during RL.Source: §7.9 · pp. 65–66
• Unlike classic RL-based NAS [Zoph and Le, 2016], reinforced MoE can adapt routing and modular structure per input during inference [Han et al., 2021].The model could reason across parameters and modules, adapting topology to prompt difficulty and real-time compute constraints.Source: §7.9 · p. 66
• Key challenges include robust multi-objective rewards, stable credit assignment under topology changes, and amortizing architecture-policy learning across prompts, tasks, and deployment scales.Reward design must avoid trivial solutions such as all-expert sparsity; solving these issues is presented as crucial for integrated architecture–algorithm co-optimization.Source: §7.9 · p. 66

8. Conclusion

• The survey focuses on RLVR for transforming LLMs into LRMs, emphasizing direct outcome-level rewards for enhancing reasoning rather than human-alignment methods such as RLHF or DPO.It covers reward design, policy optimization, sampling strategies, and debated issues in RL training.Source: §8 · p. 66

• Applications:

• It organizes RL resources across static datasets, dynamic environments, and RL infrastructure, and reviews downstream applications across multiple scenarios.The applications include coding, agentic, multimodal, multi-agent, robotics, and medical tasks.Source: §8 · p. 66, • Applications · p. 67

• Future Directions:

• The survey identifies promising future research directions aimed at achieving super-intelligence through RL-based LLMs.The listed directions include continual, memory, latent, pre-training, diffusion, and scientific approaches.Source: §8 · p. 66, • Future Directions · p. 68