PaperlayerView exact revision on arXiv

Source-linked AI summary

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Francisco Piedrahita-Velez, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Jun Wang, Shuicheng Yan, Philip Torr, Lei Bai

arXiv:2509.02547v5cs.AIcs.CL

TL;DR

Agentic RL addresses the limits of treating LLMs as static single-turn generators in realistic sequential, interactive settings. This survey formalizes Agentic RL with MDP/POMDP abstractions, organizes its capabilities and applications, and consolidates supporting resources while identifying challenges for scalable agents.

  • Problem

    Conventional LLM RL focuses on single-turn outputs, overlooking the broader sequential decision-making required in realistic interactive settings.

  • Method

    The survey formalizes Agentic RL through MDPs and POMDPs, presents capability- and application-centered taxonomies, and consolidates tasks, environments, frameworks, and benchmarks.

  • Results

    The survey maps Agentic RL across core capabilities and diverse task domains, distinguishing temporally extended, partially observable agent decision processes from conventional single-step LLM RL.

  • Takeaways & Limitations

    Future progress depends on trustworthy agents, scalable agentic training, and complex, adaptive environments.

  • Takeaways & Limitations

    Training slow-thinking agentic reasoning remains challenging because of stability, environment compatibility, latency, and overthinking concerns.

Abstract

from arXiv · show

The emergence of agentic reinforcement learning (Agentic RL) marks a paradigm shift from conventional reinforcement learning applied to large language models (LLM RL), reframing LLMs from passive sequence generators into autonomous, decision-making agents embedded in complex, dynamic worlds. This survey formalizes this conceptual shift by contrasting the degenerate single-step Markov Decision Processes (MDPs) of LLM-RL with the temporally extended, partially observable Markov decision processes (POMDPs) that define Agentic RL. Building on this foundation, we propose a comprehensive twofold taxonomy: one organized around core agentic capabilities, including planning, tool use, memory, reasoning, self-improvement, and perception, and the other around their applications across diverse task domains. Central to our thesis is that reinforcement learning serves as the critical mechanism for transforming these capabilities from static, heuristic modules into adaptive, robust agentic behavior. To support and accelerate future research, we consolidate the landscape of open-source environments, benchmarks, and frameworks into a practical compendium. By synthesizing over five hundred recent works, this survey charts the contours of this rapidly evolving field and highlights the opportunities and challenges that will shape the development of scalable, general-purpose AI agents.

1 Introduction

Agentic RL reframes LLMs as autonomous decision-makers operating over extended horizons in partially observable, dynamic environments. The survey unifies this paradigm through formal MDP/POMDP foundations, capability and domain taxonomies, and a compendium of supporting research resources.

  • Paradigm shift: Conventional LLM-RL primarily optimizes static conditional generators for single-turn alignment or benchmark performance, leaving sequential decision-making underrepresented.The survey contrasts this setting with interactive agent behavior.
  • Paradigm shift: Agentic RL treats LLMs as autonomous decision-makers that perceive, reason, plan, use tools, maintain memory, and adapt strategies over extended horizons.These behaviors occur in partially observable, dynamic environments rather than only in single-turn generation.
  • Research gap: Existing work spans RL-enhanced LLMs and LLM-based agents, but lacks a unified treatment of policy-optimized agents embedded in sequential decision processes.Prior studies often isolate capabilities, domains, or custom environments and use inconsistent terminology and evaluation protocols.
  • Formal foundations: The survey formalizes Agentic RL with MDPs and POMDPs to distinguish static sequence generation from interactive decision-making in dynamic environments.Its MDP tuple includes states, observations, actions, transitions, rewards, task horizon, and discount factor.
  • Survey scope: Its capability-centered taxonomy covers planning, tool use, memory, reasoning, reflection, interaction, perception, and related agentic modules.A second taxonomy organizes applications across diverse task domains.
  • Survey scope: The survey excludes RL for human-value alignment, non-LLM-based traditional RL, and RL aimed solely at improving static benchmark performance.These boundaries define the survey’s operational scope of Agentic RL.

2 Preliminary: From LLM RL to Agentic RL

This section distinguishes preference-based LLM reinforcement fine-tuning from Agentic RL, which models LLMs as policies acting over extended, partially observable environments. It formalizes the contrast through MDP/POMDP abstractions and describes Agentic RL’s joint language-and-action spaces and optimization challenges.

  • Paradigm shift: Preference-based reinforcement fine-tuning optimizes responses from fixed preference data, whereas Agentic RL targets task-specific behavior in dynamic environments.The transition from PBRFT to Agentic RL changes both the training setting and decision-making granularity.
  • Environment formulation: PBRFT is a degenerate single-step MDP with horizon T = 1, while Agentic RL is modeled as a multi-step POMDP with partial observations.Agentic RL accumulates signals from tools, users, and environment feedback over time.
  • Action space: Agentic RL combines free-form text actions with abstract environment-interactive actions in a unified action space.Aaction can invoke tools, modify environments, and recursively represent composite action sequences.
  • Agentic behavior: Unlike single-turn text alignment, Agentic RL supports multi-turn planning, adaptive tool invocation, stateful memory, and long-horizon credit assignment.These capabilities allow the LLM to operate as an autonomous decision-making agent.
  • RL algorithms: PPO relies on learned advantage estimates and typically a separate critic network, increasing training parameter count despite ongoing improvements to bias, variance, and update stability.REINFORCE likewise faces high gradient variance and sample inefficiency, while GRPO reduces computational overhead but can suffer from high-variance, inaccurate group advantages.

3 Agentic RL: The model capability perspective

The survey organizes Agentic RL around agent capabilities and examines how RL turns planning, tool use, memory, reflection, reasoning, and perception into adaptive components. It also identifies open challenges including structured-memory control, meta-reflection, slow-reasoning training, and long-horizon credit assignment.

  • Capability perspective: Agentic RL treats planning, tool use, memory, self-improvement, reasoning, and perception as capabilities or modules for autonomous agents.The survey follows a capability-centered perspective and examines these components in dedicated subsections.
  • Capability perspective: RL jointly optimizes agent components as interdependent policies, including trajectories for planning, retrieval dynamics for memory, invocation timing for tools, and self-supervision for reflection.This reframes components from static pipelines into policies shaped through reinforcement learning.
  • Planning: Planning research spans external RL guidance, which trains auxiliary value or heuristic functions, and internal RL, which directly refines the LLM policy through environmental interaction.A prospective synthesis aims to combine fast intuitive plan generation with slower deliberative search.
  • Tool use and reasoning: Tool-integrated reasoning supports precise tool-mediated operations alongside flexible verbal inference, while long-horizon extensions remain bottlenecked by temporal credit assignment.Current approaches often rely on sparse trajectory-level or outcome-based rewards, complicating multi-turn learning.
  • Memory: Structured-memory systems capture relational, temporal, or hierarchical dependencies, but RL has rarely controlled their construction, refinement, or evolution.Insertion, deletion, abstraction, and linkage updates remain governed largely by handcrafted rules or heuristics.
  • Self-improvement and reasoning: Open capability challenges include learning adaptive reflection strategies and reliably training slow reasoning while avoiding excessive latency or overthinking.Hybrid strategies such as adaptive test-time scaling seek to balance fast responses with extended deliberation.
  • Perception: RL can train active multimodal cognition by treating visual representations as manipulable and verifiable intermediate thoughts rather than relying only on text-output rewards.This supports multi-step seeing, manipulation, and reasoning over visual information.

4 Agentic RL: The Task Perspective

The survey examines Agentic RL across search, code generation, mathematical reasoning, GUI navigation, vision, embodied interaction, multi-agent systems, and other emerging domains. These applications illustrate how the paradigm extends from abstract concepts to real-world problem-solving.

  • Task domains: Agentic RL is surveyed in search and information retrieval, code generation and software engineering, and mathematical reasoning.These domains form the first group of application areas discussed.
  • Task domains: The task perspective also covers GUI navigation, vision understanding, and vision-language-model embodied interaction.These applications broaden Agentic RL beyond text-centric tasks.
  • Task domains: Beyond single-agent settings, the survey examines multi-agent systems and other emerging domains.The section concludes its application coverage with these broader settings.
  • Cross-domain perspective: Together, these application domains show Agentic RL transitioning from abstract paradigms into actionable, real-world problem-solving.The survey presents this transition as illustrated in Figure 6.

4.1 Search & Research Agent

Search and research agents are evolving from brittle retrieval pipelines into autonomous systems that conduct multi-step search, evidence synthesis, and report drafting. Agentic RL supports this shift by optimizing query generation, retrieval decisions, and long-horizon reasoning, while simulated search environments address instability and cost from live APIs.

  • Search & Research Agent: Deep research extends retrieval beyond finding documents to multi-step analysis, cross-source synthesis, and comprehensive report drafting.Early prompt-driven approaches relied on brittle query strategies and manual engineering.
  • Open-Source RL Methods: RL-based search methods optimize query generation and multi-step reasoning against live search results, with later systems rewarding retrieval quality between successive calls.These rewards help agents iteratively filter and structure noisy evidence during long-horizon reasoning.
  • Open-Source RL Methods: Open-source research agents increasingly combine supervised trajectories, RL fine-tuning, tool integration, and memory for autonomous web research.WebDancer, WebThinker, and related systems target benchmarks including GAIA and WebWalkerQA.
  • Search & Research Agent: Live-search training faces uncontrolled document quality, noisy information, instability, and high API costs that limit scalability.Controllable simulated search engines such as ZeroSearch are introduced to improve training efficiency, controllability, and stability.
  • Closed-Source RL Methods: Closed-source systems remain markedly stronger on difficult web research, with OpenAI Deep Research achieving 51.5% pass@1 on BrowseComp.The survey attributes the contrast to stronger foundation models and more high-quality data, alongside tool integration and memory.

4.2 Code Agent

Code agents provide a favorable setting for Agentic RL because compilation, tests, and runtime traces provide explicit feedback. The surveyed methods progress from final-outcome optimization toward process-level supervision, iterative refinement, executable world models, and long-horizon software engineering.

  • Code Agent: Code generation offers explicit execution semantics and readily available automated signals from compilation, unit tests, and runtime traces.These properties make software engineering an ideal testbed for LLM-based Agentic RL.
  • RL for Code Generation: Outcome-reward methods optimize final correctness using pass@k or unit-test success, while process-reward methods add compilation, execution, and intermediate error feedback.Process supervision is designed to reduce reward sparsity and improve credit assignment.
  • RL for Code Generation: +10.5% pass-rate improvements over the base model were reported for PRLCoder using line-level compiler and test feedback.Its reward model scores partial snippets and assigns positive or negative signals based on execution feedback.
  • RL for Iterative Code Refinement: Iterative refinement methods treat debugging as a trajectory, using failed tests, error messages, localized preferences, explanations, or critics to improve correction.RLEF reduces attempts relative to single-shot baselines, while LeDex reports consistent pass@1 gains over SFT-only coders.
  • RL for Automated Software Engineering: End-to-end software-engineering agents scale sparse validated rewards across realistic missions, benchmarks, repository histories, and multi-turn environments.Qwen3-Coder trains across 20,000 parallel environments and reports state-of-the-art performance on benchmarks such as SWE-Bench Verified.
  • Code World Models: Code World Models represent transition and reward dynamics as executable programs, enabling verifiable symbolic simulation for model-based planning.Repeated synthesis, refinement, and simulator querying can impose substantial inference costs, motivating quantization for deployment.

4.3 Mathematical Agent

Mathematical agents are studied through informal reasoning with programming-language tools and formal reasoning with mechanically verified proofs. RL methods use outcome, process, and hybrid rewards to develop tool-integrated reasoning, self-correction, exploration, and theorem-proving capabilities.

  • Mathematical Agent: Mathematical-agent research separates informal reasoning without formal verification from formal reasoning based on precisely specified languages and proof checking.The survey treats both paradigms as complementary approaches to mathematical-agent development.
  • Informal Mathematical Reasoning: Informal reasoning supports word problems, equation manipulation, and symbolic computation but lacks finite logical rules and precise formal semantics.Its flexibility captures high-level argument structure, while evaluation and formal-proof data remain challenging.
  • RL for Informal Mathematical Reasoning: Outcome-only RL with tool execution produces emergent adaptive tool use, self-reflection, self-correction from feedback, and adaptive computational reasoning.ARTIST and ToRL interleave code execution with natural-language reasoning while training from final correctness rewards.
  • RL for Informal Mathematical Reasoning: rStar2-Agent achieves average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25 after 510 RL steps.The 14B model uses Python execution, GRPO-RoC resampling on correct rollouts, and multi-stage training.
  • Formal Mathematical Reasoning: Formal theorem proving remains constrained by sparse high-quality data, sparse rewards, credit assignment, and tactic-space search explosion.Hybrid reward and expert-iteration methods address these constraints with denser signals, preference critics, and expanded theorem-proving data.
  • Formal Mathematical Reasoning: Formal reasoning produces mechanically checkable proof objects, with systems such as Lean providing deterministic pass/fail feedback for RL.Proof assistants include Lean, Isabelle, Coq, and HOL Light; error messages may provide auxiliary signals.
  • Formal Mathematical Reasoning: DeepSeek-Prover-v1.5 scales outcome-only RL in Lean using binary verifier feedback, improving proof success on miniF2F and ProofNet.Its RMaxTS variant adds intrinsic rewards for discovering novel tactic states.

4.4 GUI Agent

GUI agents have progressed from zero-shot screenshot-to-action systems and static supervised trajectories toward reinforcement learning that treats interaction as sequential decision-making. Static and interactive RL approaches use increasingly structured rewards and online rollouts to improve grounding, robustness, and long-horizon task performance.

  • Vanilla VLM-based GUI Agents: Early GUI agents mapped screenshots and prompts directly to single-step actions using pretrained vision-language models without task-specific fine-tuning.These systems demonstrated foundation-model generality but suffered from limited grounding accuracy and reliability on complex tasks.
  • Supervised Fine-Tuning with Static Trajectory Data: Supervised fine-tuning adapted vision-language models to offline screen-action trajectories, improving grounding and reasoning without online interaction.Representative systems used cross-entropy training on static pairs and introduced hierarchical or reflective reasoning.
  • RL in Static GUI Environments: Static reinforcement learning uses pre-collected deterministic traces with rule-based outcome evaluation to improve step-level GUI action prediction.GUI-R1 uses format and correctness rewards, while related methods add compact action interfaces, action-accuracy shaping, sub-goal supervision, recovery mechanisms, and grounding-aware training.
  • RL in Interactive GUI Environments: Interactive reinforcement learning optimizes GUI agents through online rollouts in dynamic environments with stochastic transitions and long-horizon dependencies.WebAgent-R1 applies end-to-end multi-turn reinforcement learning with asynchronous trajectory generation and group-wise advantages for diverse web tasks.

4.5 Vision Agents

Vision-agent research applies reinforcement learning across image, video, and 3D tasks, using task-specific rewards to improve reasoning, perception, structured prediction, and generation. The survey presents this coverage as selective rather than exhaustive because the literature spans many vision settings.

  • Scope: Reinforcement learning has been applied broadly to image, video, 3D perception, and 3D generation, but this survey does not provide an exhaustive vision-task overview.The authors direct readers to dedicated vision surveys for comprehensive coverage.
  • Image Tasks: Image-focused methods use RL to encourage long-form reasoning in LVLMs and support passive perception, tool use, visual imagination, and structured predictions.Structured outputs include bounding boxes, masks, and segmentation maps, with rewards based on measures such as IoU and confidence.
  • Image Tasks: Segmentation, detection, localization, and scene-graph methods combine GRPO with rewards including bounding-box IoU, L1 losses, point-wise distance, mAP, text matching, and IoU.These reward designs target structured visual outputs rather than text-only generation.
  • Video Tasks: Video-domain GRPO variants target temporal reasoning by emphasizing informative tokens and using soft multi-choice rewards to reduce optimization variance.TW-GRPO generates more focused reasoning chains through token weighting.
  • 3D Vision Tasks: 3D vision methods apply RL to spatial reasoning, understanding, and generation, including physics-aware constraints and rendered-image evaluations as rewards.MetaSpatial is described as the first RL-based framework for 3D spatial reasoning.

4.6 Embodied Agents

Embodied agents integrate perception, reasoning, and action in physical or simulated environments, with RL supporting navigation, manipulation, generalization, and long-horizon skill acquisition. The main practical constraint is scaling RL beyond simulation because real-world interaction is costly and the sim-to-real gap remains significant.

  • Embodied Agents: Embodied agents must integrate perception, reasoning, and action to pursue goals in complex physical or simulated environments.The category includes vision-language-action models and language-driven open-ended agents.
  • RL in VLA Navigation Agents: Navigation agents use RL to optimize future action sequences through directional, trajectory-level, or internal-deliberation rewards.VLN-R1 aligns predicted and ground-truth paths, OctoNav-R1 reinforces thinking before acting, and S2E combines video pretraining with RL for interactive and safe navigation.
  • RL in VLA Manipulation Agents: Manipulation agents apply RL to improve instruction following and trajectory prediction, especially for generalization across tasks and environments.The supplied passage describes VLM-based evaluators assigning feedback to manipulation trajectories.
  • RL in VLA Manipulation Agents: VIKI-R combines chain-of-thought fine-tuning with multi-level RL and a unified benchmark to support compositional coordination across diverse embodiments.The framework targets multi-agent embodied cooperation.
  • Challenges: Real-world embodied RL remains difficult because simulation leaves a significant sim-to-real gap and physical robot experiments are costly and complex.Many RL algorithms also require millions of interaction steps, creating substantial time and resource demands.
  • Case Study: Voyager: Voyager integrates planning, skill acquisition, and RL-based curriculum learning through iterative exploration, reusable skill extraction, and a growing skill library.A curriculum scheduler selects new tasks based on the agent’s interaction loop.

4.7 Multi-Agent Systems

LLM-based multi-agent systems use RL to optimize coordination at different levels, from external communication structures and selected agents to full joint training. These regimes trade flexibility and emergent collaboration against sample complexity, credit-assignment difficulty, and long-horizon reward sparsity.

  • Multi-Agent Systems: LLM-based multi-agent systems coordinate multiple autonomous agents through structured interaction, role specialization, task decomposition, reasoning, and memory management.The survey contrasts early static or hand-designed systems with debate-based collaborative refinement.
  • RL-Free Multi-Agent Evolution: RL-free evolution improves multi-agent coordination through symbolic learning, dynamic graph optimization, or workflow rewriting while keeping foundation-model parameters unoptimized.These methods change system structures rather than directly updating foundation-model parameters.
  • RL-Based Multi-Agent Evolution: External-structure RL keeps agent parameters frozen and optimizes communication topologies, routing policies, or workflow graphs, making credit assignment operate at the topology or message-routing level.Rewards are typically delayed and sparse, while agent-level gradients are unavailable.
  • Communication Protocols: Fixed communication protocols favor stability in low-data or specialized domains, whereas learnable protocols can discover emergent communication but require higher sample complexity and regularization.The comparison highlights a stability-flexibility trade-off in protocol design.
  • Partial Multi-Agent Optimization: Partial optimization updates only selected agents, such as leaders or coordinators, reducing sample complexity and avoiding instability from fully decoupled credit assignment.Role-conditioned DPO, local advantage estimation, and counterfactual baselines address semi-local credit assignment.
  • End-to-End Multi-Agent Reinforcement Learning: Full multi-agent RL jointly optimizes collaboration and reasoning, enabling emergent division of labor and communication conventions under shared or decentralized objectives.MAGRPO formalizes cooperation as a Dec-POMDP, while other methods use validation rewards, pseudo-labels, parameter sharing, hybrid rewards, or evolutionary search.
  • End-to-End Multi-Agent Reinforcement Learning: End-to-end multi-LLM training intensifies temporal and structural credit assignment because rewards can depend on long multi-turn interaction chains.Role-conditioned advantages, hierarchical controller-worker architectures, and self-play curricula are used to densify reward signals.

4.8 Other Tasks

The survey extends Agentic RL across specialized tasks, emphasizing adaptive reasoning, multi-turn interaction, structured rewards, and domain-specific training pipelines.

  • TextGame: ARIA, GiGPO, RAGEN, and SPA-RL address action-space sparsity, temporal credit assignment, multi-turn stability, and delayed rewards in TextGame.Their mechanisms include intention-driven reward aggregation, hierarchical grouping, trajectory filtering, gradient stabilization, and per-step reward decomposition.
  • Other Tasks: SkyRL-SQL enables agents to probe, refine, and verify SQL queries interactively using only 653 training examples.The resulting SkyRL-SQL-7B model surpasses GPT-4o and o4-mini on SQL generation benchmarks.
  • Time Series: Time-R1 and TimeMaster apply progressive curricula, dynamic rewards, SFT, and GRPO to support structured temporal reasoning over time-series inputs.
  • General QA: Agent and L-Zero develop general-purpose agent behavior through chain-of-action generation, supervised fine-tuning, reinforcement learning, and scalable worker pools.
  • Social: Sotopia-RL and AML address socially intelligent interaction using utterance-level multidimensional rewards and adaptive switching among reasoning modes.These methods target partial observability and multifaceted social objectives.

5.1 Environment Simulator

Agentic RL environments range from controlled web benchmarks to dynamic operating systems, executable coding tasks, scientific workflows, and broad multi-environment evaluations.

  • Environment Simulator: Agentic RL environments provide observations, actions, state transitions, and rewards for training agents on specific interactive tasks.
  • Web Environments: WebShop, WebArena, and VisualWebArena support reproducible web interaction involving product search, multi-tab workflows, user roles, and visual comprehension.Their domains include e-commerce, forums, collaborative development, and content management.
  • GUI Environments: AndroidWorld offers 116 tasks across 20 applications and millions of parameterized variations on a live Android emulator.OSWorld extends evaluation to real Ubuntu, Windows, and macOS workflows, including files, desktop applications, and cross-application tasks.
  • Code Environments: Coding environments combine interactive debugging and repository manipulation with curated benchmarks such as HumanEval’s 164 Python tasks and MBPP’s 974 tasks.
  • Programmatic World Models: Programmatic world-model benchmarks test whether agents can synthesize executable environment dynamics and use them for planning.CWMB evaluates transition fidelity and downstream planning utility.
  • Domain and General Benchmarks: Science, machine-learning, biomedical, cybersecurity, text-game, and broad agent benchmarks cover specialized workflows and diverse interactive capabilities.Examples include PaperBench, MLE-Bench’s 75 Kaggle competitions, SecRepoBench’s 27 repositories and 15 CWE categories, TextWorld, AgentGym, and AgentBench.

5.2 RL Framework

The survey organizes reinforcement-learning infrastructure into Agentic RL, RLHF/LLM fine-tuning, and general-purpose RL frameworks, covering scalable training and distributed execution.

  • RL Framework: The framework landscape is categorized into Agentic RL, RLHF and LLM fine-tuning, and general-purpose RL codebases.
  • Agentic RL Frameworks: Agentic RL frameworks support verifiable environments, long-horizon training, asynchronous distribution, language reasoning, and multi-agent optimization.Examples include Verifiers, SkyRL-v0, AREAL, and MARTI.
  • RLHF and LLM Fine-Tuning: RLHF and fine-tuning frameworks provide scalable alignment, baseline RLHF implementations, distributed training, and experiment-management support.Examples include OpenRLHF, TRL, trlX, and HybridFlow.
  • General-Purpose RL: General-purpose RL libraries supply unified APIs, modular distributed components, and support for on-policy, off-policy, online, offline, hierarchical, and multi-agent methods.RLlib, Acme, and Tianshou exemplify these capabilities.

6 Open Challenges and Future Directions

The survey identifies trustworthiness, scalable training, scalable environments, and the conditions for genuine capability gains as central frontiers for Agentic RL.

  • Trustworthiness: Trustworthiness is challenged by expanded attack surfaces, hallucination, sycophancy, and reward hacking across tools, memory, planning, and autonomous behavior.
  • Trustworthiness: Outcome-only rewards can amplify hallucination, while SFT combined with verifiable-reward RL can mitigate this degradation.
  • Trustworthiness: Sycophancy can arise when RLHF rewards agreeable responses, motivating sycophancy-aware reward models and online policy–reward co-optimization.Cooper uses rule-based verifiers and LLM-generated negatives to reduce exploitation of static reward-model loopholes.
  • Scaling up Agentic Training: Longer RL training improves tool-use frequency, reasoning depth, task accuracy, and can enable smaller models to rival larger ones on mathematics, code, and logic benchmarks.
  • Scaling up Agentic Training: Scaling model capacity increases potential but risks entropy collapse, reduced diversity, compute demands, and hyperparameter sensitivity.Hybrid SFT+RL and advantage-based methods are presented as responses to these trade-offs.
  • Scaling up Agentic Training: Cross-domain RL produces both synergies and interference, while gains depend on domain combinations and model initialization.
  • Scaling up Agentic Training: Efficiency research emphasizes difficulty calibration, diversity sampling, longer reasoning, simple strategy combinations, and low-cost RL-like gradient scaling.

7 Conclusion

The survey formalizes Agentic RL as a shift from single-step LLM reinforcement learning toward temporally extended, partially observable decision-making. It maps the field through capability- and application-centered taxonomies and consolidates resources for future research.

  • Agentic RL distinguishes temporally extended, partially observable decision processes from conventional single-step reinforcement learning for LLMs.
  • The survey organizes Agentic RL research around core capabilities such as planning, tool use, memory, reasoning, self-improvement, and perception.
  • A complementary taxonomy maps Agentic RL applications across diverse task domains.
  • The survey argues that reinforcement learning transforms agentic capabilities from static, heuristic modules into adaptive, robust behavior.
  • The authors consolidate open-source environments, benchmarks, and frameworks into a practical compendium for future research.
Loading 2509.02547v5…