PaperlayerView exact revision on arXiv

Source-linked AI summary

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Francisco Piedrahita-Velez, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Jun Wang, Shuicheng Yan, Philip Torr, Lei Bai

arXiv:2509.02547v5cs.AIcs.CL

TL;DR

Agentic RL addresses the limits of single-step LLM reinforcement learning for autonomous decision-making in complex, dynamic worlds. This survey formalizes the distinction with MDP/POMDP abstractions, organizes the field by capabilities and applications, and synthesizes its environments, benchmarks, and frameworks. It charts Agentic RL as a developing paradigm centered on temporally extended, partially observable agent behavior.

  • Problem

    Conventional LLM reinforcement learning treats models largely as static generators, leaving systematic comparison and cross-domain generalization for interactive agentic settings limited.

    Source: §1 · pp. 1, 3

  • Method

    The survey formalizes Agentic RL with MDP/POMDP abstractions and provides capability-centered and application-centered taxonomies alongside a compendium of tasks, environments, frameworks, and benchmarks.

    Source: §1 · p. 3, §2.1 · p. 5, §7 · p. 49

  • Results

    The survey maps Agentic RL as temporally extended, partially observable decision-making across capabilities including planning, tool use, memory, reasoning, self-improvement, and perception.

    Source: §1 · p. 3, §7 · p. 49

  • Takeaways & Limitations

    Agentic RL provides a framework for studying LLMs as autonomous decision-making agents situated in complex, dynamic worlds.

    Source: §7 · p. 49

    Agentic reasoning still faces training-stability challenges across diverse environments and can exhibit excessive latency or overthinking.

    Source: §3.5 · p. 17

Abstract

from arXiv · show

The emergence of agentic reinforcement learning (Agentic RL) marks a paradigm shift from conventional reinforcement learning applied to large language models (LLM RL), reframing LLMs from passive sequence generators into autonomous, decision-making agents embedded in complex, dynamic worlds. This survey formalizes this conceptual shift by contrasting the degenerate single-step Markov Decision Processes (MDPs) of LLM-RL with the temporally extended, partially observable Markov decision processes (POMDPs) that define Agentic RL. Building on this foundation, we propose a comprehensive twofold taxonomy: one organized around core agentic capabilities, including planning, tool use, memory, reasoning, self-improvement, and perception, and the other around their applications across diverse task domains. Central to our thesis is that reinforcement learning serves as the critical mechanism for transforming these capabilities from static, heuristic modules into adaptive, robust agentic behavior. To support and accelerate future research, we consolidate the landscape of open-source environments, benchmarks, and frameworks into a practical compendium. By synthesizing over five hundred recent works, this survey charts the contours of this rapidly evolving field and highlights the opportunities and challenges that will shape the development of scalable, general-purpose AI agents.

1 Introduction

Agentic RL reframes LLMs as learnable policies and autonomous decision-makers operating in long-horizon, partially observable, dynamic environments rather than static single-turn generators. This survey unifies the paradigm’s foundations, capabilities, applications, resources, scope, and open challenges.

Paradigm shift

  • Agentic RL treats LLMs as learnable policies in sequential decision-making loops, using RL to enable planning, reasoning, tool use, memory, self-reflection, and long-horizon interaction.The paradigm targets autonomous behavior in partially observable, dynamic environments rather than single-turn output alignment.Source: §1 · pp. 1–2

Related work

  • Prior work spans RL for improving LLM instruction-following, ethical alignment, and code generation, LLMs used in RL for planning and reward design, and autonomous LLM agents for reasoning, planning, acting, and tool use.Representative surveys cover agent methodologies, core capabilities, tool use, planning, self-evolution, collaboration, multimodality, memory, and perception.Source: §1 · p. 2

Research gap

  • The survey addresses a gap caused by fragmented studies of agent capabilities, domains, environments, terminology, and evaluation protocols that hinder systematic comparison and cross-domain generalization.It connects theoretical foundations with algorithmic approaches and practical systems through a unified treatment of Agentic RL.Source: §1 · pp. 2–3

Contributions

  • Its contributions formalize Agentic RL with MDP and POMDP abstractions and organize RL-optimizable agent capabilities around planning, tool use, memory, reasoning, reflection, and interaction.The survey also consolidates representative tasks, environments, frameworks, and benchmarks, then discusses open challenges and future directions.Source: §1 · p. 3

Scope and organization

  • The survey excludes RL for human value alignment, traditional non-LLM RL, and RL aimed solely at improving static benchmark performance.Its sections progress from formal foundations and capability taxonomies to applications, resources, open challenges, and conclusions, with the structure illustrated in Figure 1.Source: §1 · p. 3

2 Preliminary: From LLM RL to Agentic RL

This section formalizes the shift from preference-based reinforcement fine-tuning toward Agentic RL, distinguishing static, single-turn sequence optimization from adaptive decision-making in dynamic environments. It defines the resulting differences in state, actions, transitions, rewards, objectives, and canonical optimization algorithms.

2 Preliminary: From LLM RL to Agentic RL

  • Researchers are shifting from preference-based RFT on fixed preference datasets toward Agentic RL for specific tasks and dynamic environments.This follows advances in reasoning, self-evolving capabilities, and tool use in contemporary LLMs.Source: §2 · pp. 3–4

2.1 Markov Decision Processes

  • PBRFT is a degenerate MDP, whereas Agentic RL is a POMDP that optimizes sentence sequences under full observation versus semantic behaviors under partial observation.The formalization uses the tuple ⟨S, O, A, P, R, T, γ⟩ to characterize states, observations, actions, transitions, rewards, horizons, and discounting.Source: §2.1 · p. 5

2.2 Environment State

  • PBRFT terminates after one response with horizon T = 1, while Agentic RL operates across multiple time-steps and accumulates tool, user, and environmental feedback.Agentic RL therefore models temporally extended interaction with a changing world state and partial observations.Source: §2.2 · pp. 5–6

2.3 Action Space

  • Agentic RL jointly models natural-language outputs and structured actions that invoke tools or modify environments, including recursively composed action sequences.Atext communicates without directly changing external state, whereas Aaction acquires information or changes physical or simulated environments.Source: §2.3 · p. 6

2.4 Transition Dynamics

  • Unlike PBRFT’s deterministic one-shot mapping, Agentic RL sequentially combines communication, information acquisition, and environment manipulation under uncertain transitions.Text actions leave the environment unchanged, while structured actions query tools or directly alter it.Source: §2.4 · pp. 6–7

2.5 Reward Function

  • PBRFT uses a scalar final preference score without intermediate feedback, whereas Agentic RL supports task-completion and step-level rewards that may be dense, sparse, or learned.Examples include unit-test passes and symbolic-verifier success.Source: §2.5 · p. 7

2.6 Learning Objective

  • Agentic RL maximizes discounted rewards through exploration and long-term credit assignment, enabling multi-turn planning, adaptive tool use, stateful memory, and autonomous decision-making.PBRFT instead optimizes single-turn text quality without explicit planning, tool use, or environmental feedback.Source: §2.6 · p. 7

2.7 RL Algorithms

  • The section reviews REINFORCE, PPO, GRPO, and DPO as canonical algorithms, while noting variance, critic overhead, and static-data limitations across these methods.REINFORCE has high-variance, sample-inefficient estimates; PPO relies on a separate critic; DPO depends on static preference data; and GRPO’s group-relative estimates can be noisy and inaccurate.Source: §2.7 · pp. 7–9

3 Agentic RL: The model capability perspective

This section examines how Agentic RL jointly develops interdependent agent capabilities, including planning, tool use, memory, self-improvement, reasoning, and multimodal perception, beyond static pipelines. It surveys approaches for improving adaptive cognition and strategic coherence over long-horizon interactions through mechanisms such as verifiable and process-level rewards, grounding, tool use, generation, and segment-level optimization.

3 Agentic RL: The model capability perspective

  • Agentic RL treats planning, tool use, memory, reflection, and reasoning as interdependent policies jointly optimized through environmental feedback rather than static modules.The section provides a conceptual delineation of RL’s applications for agent capabilities rather than an exhaustive survey of related work.Source: §3 · pp. 9–10

3.1 Planning

  • Planning research divides RL into external guidance of structured search and internal optimization of the LLM’s planning policy, with future systems combining intuitive generation and deliberative search.External methods train value or heuristic functions, whereas internal methods refine planning through trial-and-error, trajectory preferences, skill libraries, or efficiency-aware rewards.Source: §3.1 · p. 11

3.2 Tool Using

  • RL evolves tool-enabled agents from imitative ReAct pipelines into outcome-optimized, multi-turn tool-integrated reasoning systems that adapt tool selection, timing, composition, and error recovery.Long-horizon TIR remains bottlenecked by temporal credit assignment because sparse trajectory-level rewards obscure which tool invocation caused success or failure.Source: §3.2 · pp. 12–13

3.3 Memory

  • Agentic RL transforms memory from passive storage into a controllable subsystem governing storage, retrieval, updating, deletion, and forgetting across explicit and latent token representations.Structured memories such as temporal, atomic, and hierarchical graphs offer richer relational organization, but their construction and evolution remain largely heuristic and underexplored with RL.Source: §3.3 · pp. 13–14

3.4 Self-Improvement

  • Self-improvement progresses from verbal self-correction to parameter-level internalization and autonomous iterative self-training driven by reflection, search, execution feedback, and shared experience.The prospective challenge is meta-evolution: learning a policy that selects and improves reflection strategies rather than relying on handcrafted, static reflective procedures.Source: §3.4 · pp. 15–16

3.5 Reasoning

  • Reasoning spans fast, efficient, heuristic inference and slower, structured deliberation, with reinforcement learning increasingly supporting longer, more robust reasoning trajectories despite greater computational cost.Fast reasoning is vulnerable to factual errors, biases, and shallow generalization, whereas slow reasoning improves accuracy and robustness in knowledge-intensive tasks through explicit traces, verification, and multi-step inference.Source: §3.5 · pp. 16–17
  • Agentic reasoning requires RL methods that balance fast responses with slow, reliable deliberation while avoiding excessive latency, overthinking, and training instability across environments.Hybrid strategies such as adaptive test-time scaling aim to select between rapid responses and extended deliberation according to task complexity.Source: §3.5 · p. 17

3.6 Perception

  • RL shifts multimodal agents from passive visual answering toward active cognition by making visual representations manipulable, verifiable intermediate thoughts in multi-step reasoning.This transfer depends on well-designed verifiable rewards, including vision-specific metrics, partial rewards, spatial consistency objectives, and curricula; R1-style methods show strong results even on smaller 3B-parameter models and can improve robustness and out-of-distribution generalization.Source: §3.6 · p. 18
  • Grounding-based active perception anchors reasoning steps to image regions, enabling repeated visual querying and more valid, verifiable multimodal reasoning.Methods interleave bounding boxes or point-and-copy retrieval with textual CoT, use IoU or reflection rewards, and can add zooming operations; DeepEyes shows that end-to-end RL can induce grounding behavior.Source: §3.6 · p. 19
  • Tool-driven and generation-driven perception expands the agent’s action space to include visual tools, code, image manipulation, sketches, and imagined rollouts for planning and reasoning.RL trains models to select or create tools, supports operations such as crop, erase, and paint with curiosity rewards, and jointly optimizes semantic and token-level visual reasoning stages.Source: §3.6 · p. 19

3.7 Others

  • For long-horizon interactions, Agentic RL addresses sparse delayed feedback through process-based intermediate supervision and multi-step preference optimization to improve temporal credit assignment and strategic coherence.External reward models or programmatic rules evaluate intermediate steps, while Segment-level DPO learns from conversational snippets or action sequences whose early decisions affect long-term success.Source: §3.7 · pp. 19–20

4 Agentic RL: The Task Perspective

Agentic RL is examined through diverse application domains that test and shape its evolving capabilities. The section spans individual and multi-agent settings, showing how Agentic RL moves from abstract paradigms toward actionable, real-world problem-solving.

Application domains

  • The survey covers search and information retrieval, code generation and software engineering, and mathematical reasoning as core Agentic RL application domains.Source: §4 · p. 20
  • It also examines GUI navigation, vision understanding, and VLM embodied interaction, each presenting distinct task settings for Agentic RL.Source: §4 · p. 20

Multi-agent and emerging domains

  • Beyond single-agent tasks, the section considers multi-agent systems and other emerging domains, illustrating Agentic RL’s transition into actionable, real-world problem-solving.Source: §4 · p. 20

4.1 Search & Research Agent

Search and research agents are evolving from retrieval systems into RL-optimized, autonomous systems that plan, search, reason, verify, synthesize, and draft across long-horizon tasks. Open-source methods address training and scalability challenges, while closed-source systems demonstrate stronger benchmark performance through integrated tools, memory, and control.

4.1 Search & Research Agent

  • RL transforms search agents from brittle query systems into end-to-end, multi-step processes that coordinate query generation, reasoning, evidence synthesis, and report drafting.Prompt-based approaches lack fine-tuning signals, limiting generalization and effectiveness in multi-turn search–reasoning–synthesis settings.Source: §4.1 · p. 20

4.1.1 Open Source RL Methods

  • Open-source methods optimize web search with outcome, retrieval-quality, intermediate-step, and tool-use rewards, enabling iterative refinement, long-horizon reasoning, multimodal search, and autonomous browsing.Examples include DeepRetrieval, Search-R1, DeepResearcher, AutoRefine, StepSearch, WebDancer, WebWatcher, and ASearcher, which supports 40+ tool calls.Source: §4.1.1 · pp. 20–21
  • Offline and simulated search methods mitigate uncontrolled document quality, training instability, API cost, and scalability limits while retaining transfer to online retrieval.ZeroSearch uses curriculum RL with a distilled pseudo-search engine, whereas SSRL trains through offline self-search and transfers to online inference.Source: §4.1.1 · pp. 21–22

4.1.2 Closed Source RL Methods

  • 51.5% pass@1 on BrowseComp is achieved by OpenAI Deep Research, while closed-source systems outperform most open-source agents on long-horizon planning, grounded tool use, and cross-source verification.Perplexity DeepResearch, Google Gemini DeepResearch, Kimi-Researcher, Grok AI DeepSearch, and Doubao with Deep Think combine RL-style fine-tuning with advanced tool integration and memory.Source: §4.1.2 · p. 22
  • Deep Research unifies planning, reasoning, web-tool decisions, persistent scratchpad memory, cross-source verification, and RL-shaped control over search depth, branching, and evidence integration.The agent decomposes requests into sub-goals, refines search trajectories, and trains search policies using research benchmarks such as BrowseComp.Source: §4.1.2 · pp. 22–23

4.2 Code Agent

Code agents are a well-suited domain for Agentic RL because execution provides explicit semantics and verifiable feedback, with research progressing from generation to refinement, software engineering, and executable world models. Across these stages, RL converts compilation, tests, traces, and task outcomes into signals for increasingly autonomous coding behavior.

4.2 Code Agent

  • Code generation offers explicit execution semantics and automated compilation, testing, and runtime signals, motivating RL methods that progress from single-round synthesis toward more complex software engineering.Early multi-agent systems coordinated roles through prompting, while CodeRL introduced execution-based reward modeling and actor–critic training.Source: §4.2 · p. 23, §4.2.1 · p. 23

4.2.1 RL for Code Generation

  • +10.5% pass-rate improvements over the base model show that line-level process supervision can guide code synthesis more effectively than outcome-only signals.Process-based methods use compilation, execution traces, intermediate errors, and partial-snippet rewards to reduce reward sparsity and improve credit assignment.Source: §4.2.1 · p. 23

4.2.2 RL for Iterative Code Refinement

  • Iterative refinement methods treat debugging as a multi-attempt trajectory, using execution feedback, learned verifiers, localized preferences, critics, and per-turn rewards to improve correction and recovery.RLEF optimizes final pass rates from real error messages, while ReVeal autonomously generates tests and learns from each refinement turn.Source: §4.2.2 · pp. 24–25

4.2.3 RL for Automated Software Engineering

  • End-to-end software-engineering agents scale RL to long-horizon missions using validated task completion, repository histories, execution feedback, and multi-turn interactions across diverse coding environments.Reported systems target SWE-bench-style bug fixing, Linux kernel tuning, retrieval-augmented completion, automated machine learning, and other repository- or environment-level tasks.Source: §4.2.3 · pp. 25–26

4.2.4 Emerging Paradigms

  • Code World Models represent transitions and rewards as executable programs, enabling verifiable symbolic simulation and model-based planning instead of opaque latent world models.Systems synthesize, repair, and query executable simulators, though repeated program generation and inference can require quantization for practical deployment.Source: §4.2.4 · p. 26

4.3 Mathematical Agent

Mathematical agents are evaluated through informal and formal reasoning paradigms, with reinforcement learning turning tool use, verification, exploration, and self-correction into adaptive mathematical behavior. Informal methods emphasize flexible natural-language reasoning, whereas formal methods generate mechanically checked proofs in interactive theorem provers.

4.3.1 RL for Informal Mathematical Reasoning

  • Informal mathematical reasoning uses flexible natural-language expression without explicit syntax or formal semantics, supporting word problems, equation manipulation, and symbolic computation with programming tools.General-purpose languages such as Python are treated as informal because they lack the rigor and semantics of proof-assistant languages.Source: §4.3.1 · p. 27
  • Outcome-reward RL for informal mathematics produces emergent adaptive tool use, self-correction, reflection, context-aware reasoning, and computational strategies, while verifier- and entropy-based rewards target trustworthiness and continual improvement.ARTIST, ToRL, ZeroTIR, TTRL, RENT, and Satori illustrate these outcomes; 1-shot RLVR also reports near-large-dataset performance and post-saturation generalization.Source: §4.3.1 · p. 27
  • 80.6% on AIME24 and 69.8% on AIME25: rStar2-Agent achieves these average pass@1 scores after 510 RL steps using Python execution, GRPO-RoC, and multi-stage training.The 14B-parameter model resamples correct rollouts amid tool noise and achieves state-of-the-art results.Source: §4.3.1 · p. 28
  • Informal methods remain limited for advanced theorem proving because they lack machine-verifiable evaluation and high-quality formal proof data.These methods excel at word problems and symbolic computation but struggle to extend effectively to automated theorem proving.Source: §4.3.1 · p. 28

4.3.2 RL for Formal Mathematical Reasoning

  • Formal mathematical reasoning models theorem proving as tactic-based MDP search, where Lean, Isabelle, Coq, and HOL Light deterministically provide binary verifier rewards for mechanically checkable proof objects.Some methods additionally use proof-assistant error messages as auxiliary training signals.Source: §4.3.2 · p. 28
  • Formal RL advances combine sparse verifier rewards with exploration, subgoal-level feedback, expert iteration, and conjecturer–prover curricula to improve proof search, credit assignment, adaptability, and generalization.DeepSeek-Prover, Seed-Prover, ProofNet++, Leanabell-Prover-v2, ExIt, Lean-STaR, and STP exemplify these strategies; Lean-STaR finds that interleaved thoughts increase exploration diversity, while STP reports large LeanWorkbook gains and competitive miniF2F and ProofNet results.Source: §4.3.2 · pp. 29–30

4.4 GUI Agent

GUI agents have evolved from zero-shot and supervised screenshot-to-action systems toward reinforcement-fine-tuned agents that learn sequential behavior in static and interactive environments. This progression improves grounding, robustness, and long-horizon execution while addressing the scarcity of human operation traces.

4.4 GUI Agent

  • GUI-agent training has shifted from single-step screenshot-to-action prediction toward reinforcement fine-tuning as sequential decision-making with sparse or shaped rewards for complex, long-horizon scenarios.Table 7 presents the majority of works discussed in this section.Source: §4.4 · p. 30

4.4.1 RL-free Methods

  • RL-free GUI agents progress from zero-shot VLM action prediction to SFT on static screen-action trajectories, but remain constrained by grounding reliability and scarce human operation traces.Zero-shot systems such as MM-Navigator (Yan et al., 2023), SeeAct (Zheng et al., 2024), and TRISHUL (Kunal Singh, 2025) rely on off-the-shelf VLMs, while SFT systems add grounding, hierarchical or reflective reasoning, continuous rewards, resampling, decomposed grounding, and large-scale tutorial data.Source: §4.4.1 · pp. 30–31

4.4.2 RL in Static GUI Environments

  • UI-Venus (Gu et al., 2025) achieves new state-of-the-art performance in UI grounding and navigation through screenshot-based reinforcement fine-tuning with custom rewards and self-evolving trajectories.Other static-environment methods use unified action schemas, group-relative optimization, spatial-reasoning distillation, sub-goal supervision, recovery mechanisms, or concise JSON actions to improve grounding, action accuracy, and long-horizon robustness.Source: §4.4.2 · pp. 31–32

4.4.3 RL in Interactive GUI Environments

  • Interactive GUI reinforcement learning uses online rollouts to handle stochastic transitions and long-horizon dependencies across web, desktop, and mobile environments.Representative systems add asynchronous generation, milestone tracking, reflection, offline-to-online learning, curricula, generated-task adaptation, efficiency rewards, parallel infrastructure, or alternating reinforcement learning and supervised fine-tuning to improve robustness, scalability, efficiency, and generalization.Source: §4.4.3 · p. 32

4.5 Vision Agents

This section surveys reinforcement learning for vision agents across image, video, and 3D tasks, emphasizing the shift toward interactive, visually grounded reasoning and structured, task-specific rewards. It is selective rather than exhaustive and points readers to dedicated vision surveys for broader coverage.

Scope

  • The section is selective rather than exhaustive because RL spans image, video, 3D perception, and generation, with dedicated surveys recommended for broader coverage.The cited surveys are Wu et al., 2025h and Zhou et al., 2025a.Source: §4.5 · p. 32

Image Tasks

  • RL for image agents is shifting from passive visual perception toward active, visually grounded reasoning through grounding, tool use, and visual imagination.Structured rewards support outputs such as boxes, masks, segmentations, and scene graphs, while RL also improves diffusion, flow, and autoregressive image generation.Source: §4.5 · pp. 32–33

Video Tasks

  • Video-agent RL extends GRPO and related methods to temporal reasoning, interpretable traces, data-efficient task adaptation, smaller models, long-video training, and embodied video reasoning.Video-generation methods additionally target temporal coherence, controllability, and semantic alignment, including DanceGRPO, GAPO, GRADEO, InfLVG, Phys-AR, VideoReward, TeViR, and InstructVideo.Source: §4.5 · pp. 33–34

3D Vision Tasks

  • RL advances 3D understanding and generation through physics-aware spatial rewards, supervision-light scene reasoning, shared 3D representations, preference alignment, and 2D reward signals.MetaSpatial uses physics-aware constraints and rendered-image evaluations; Scene-R1 avoids point-wise 3D supervision; SpatialReasoner unifies perception, computation, and reasoning stages.Source: §4.5 · p. 34

4.6 Embodied Agents

Embodied agents integrate perception, reasoning, and action to pursue instructed goals in physical or simulated environments, with RL refining VLA policies through interactive rewards. The section distinguishes navigation and manipulation applications, presents Voyager’s self-improving loop, and identifies sim-to-real scalability as a central challenge.

Overview

  • Embodied agents use RL to refine pretrained VLA policies through interaction, improving exploration, sample efficiency, and generalization across physical or simulated environments.These systems integrate perception, reasoning, and action, and divide broadly into navigation agents and manipulation agents.Source: §4.6 · p. 34

RL in VLA Navigation Agent

  • Navigation RL enhances future action-sequence planning through directional or trajectory-level rewards, with deliberation-focused methods promoting more anticipatory and robust behavior.VLN-R1 uses trajectory alignment and GRPO, OctoNav-R1 reinforces internal deliberation, and S2E combines video pretraining with RL for NavBench-GS generalization.Source: §4.6 · p. 34

RL in VLA Manipulation Agent

  • Manipulation RL improves instruction following, trajectory prediction, and cross-environment generalization by refining VLA policies with evaluator feedback or rule-based trajectory rewards.RLVLA and VLA-RL use pretrained VLM evaluators, while TGRPO applies GRPO to rule-defined rewards.Source: §4.6 · pp. 34–35
  • VIKI-R combines Chain-of-Thought fine-tuning with multi-level RL to support compositional coordination across diverse embodiments in multi-agent embodied cooperation.It also provides a unified benchmark and a two-stage framework.Source: §4.6 · p. 35

Challenges

  • Scaling embodied RL remains difficult because simulation does not eliminate the sim-to-real gap, while real-world training is costly, complex, and interaction-intensive.Millions of interaction steps make direct physical-robot training impractical, leaving scalable simulation-to-deployment pipelines as an open problem.Source: §4.6 · p. 35

Case Study: Voyager

  • Voyager couples planning, environmental interaction, reusable skill memory, and RL-driven curriculum learning into a self-improving Minecraft agent.It selects tasks from the current skill set and refines, commits, or discards behaviors based on RL objectives.Source: §4.6 · p. 35

4.7 Multi-Agent Systems

Multi-agent systems evolve from static role-specialized or debate-based designs toward reinforcement-learning and evolutionary methods that optimize coordination, selected policies, or entire agent collectives. End-to-end training enables emergent collaboration but faces severe credit-assignment and scalability challenges in long multi-agent interactions.

4.7.1 RL-Driven Optimization of Non-Parametric Coordination Modules

  • Non-parametric coordination methods optimize communication topologies, routing, or workflows while keeping foundation-model parameters frozen, whereas RL-free evolution uses symbolic learning, graph optimization, or workflow rewriting.Graph-level policy gradients face delayed sparse rewards and topology-level credit assignment; fixed protocols favor low-data stability, while learnable protocols require higher sample complexity and regularization.Source: §4.7.1 · p. 35, §4.7 · p. 35

4.7.2 RL-Driven Optimization of Selected Agent Policies

  • Selected-agent optimization trains leaders, coordinators, or specialists while freezing others, improving sample efficiency and stability while enabling specialized roles through semi-local credit assignment.MALT (Motwani et al., 2025) uses heterogeneous search trees and SFT/DPO on successful and failed reasoning paths; role-conditioned DPO, local advantages, and counterfactual baselines limit reward hijacking.Source: §4.7.2 · p. 36

4.7.3 End-to-End Multi-Agent Reinforcement Learning

  • End-to-end MARL jointly trains agents under Dec-POMDP objectives, enabling emergent role division and communication through methods including MAGRPO, MAPoRL, ReMA, CURE, and SPIRAL.Role-conditioned advantages, hierarchical architectures, self-play curricula, centralized critics, and shared baselines address sparse rewards and credit assignment, but sample complexity grows roughly linearly with agent count and quadratically with interaction depth.Source: §4.7.3 · pp. 36–37

4.8 Other Tasks

This section surveys Agentic RL across text games, Text-to-SQL, chart-to-code, time-series reasoning, general question answering, and social intelligence. Across these tasks, methods address sparse or delayed rewards, temporal credit assignment, structured reasoning, scalability, and partial observability.

TextGame

  • Text-game methods improve learning by aggregating intention-driven rewards, hierarchically assigning temporal credit, filtering trajectories, stabilizing gradients, and decomposing delayed rewards into per-step signals.Trinity-RFT additionally provides a modular reinforcement-fine-tuning framework supporting diverse RL modes and data pipelines across tasks.Source: §4.8 · p. 37

Text-to-SQL and Chart-to-Code

  • SkyRL-SQL-7B surpasses GPT-4o and o4-mini on SQL generation benchmarks using just 653 training examples, while MSRL achieves state-of-the-art chart understanding performance.SkyRL-SQL enables interactive database probing, refinement, and verification; MSRL uses multimodal structured rewards at multiple granularities to overcome the SFT plateau.Source: §4.8 · p. 37

Cross-Task Overview

  • Table 9 maps qualitative dependencies between agentic capabilities and application domains, linking the capability taxonomy with the surveyed tasks.Source: Table 9 · p. 38

Time Series

  • Time-R1 uses progressive reinforcement learning and dynamic rule-based rewards, while TimeMaster combines SFT with GRPO for structured, interpretable reasoning over visualized time series.Source: §4.8 · p. 38

General QA

  • General-agent methods combine supervised fine-tuning with reinforcement learning for chain-of-action generation or scalable end-to-end training through low-cost, sandboxed concurrent worker pools.Source: §4.8 · p. 38

Social

  • Sotopia-RL refines episode rewards into utterance-level multidimensional signals under partial observability, while AML dynamically switches reasoning granularity and outperforms fixed-depth GRPO in accuracy and reasoning length.Source: §4.8 · p. 38

5 Enviroment and Frameworks · 5.1 Environment Simulator

The survey catalogs environments and benchmarks that operationalize Agentic RL across web, GUI, coding, domain-specific, simulated, game, and general-purpose settings. These environments vary in modality, dynamism, task scope, and evaluation design, from reproducible websites and emulators to executable world models and multi-agent games.

5.1 Environment Simulator

  • Environments define agentic interaction by converting observations and actions into state transitions and rewards, with Table 10 organizing them by capability, domain, and modality.The survey frames environments as worlds that respond to agent actions while providing sensory input and reward signals.Source: §5.1 · p. 38, Table 10 · p. 39

5.1.1 Web Environments

  • Web environments range from deterministic e-commerce and crowdsourced tasks to self-hostable multimodal websites and multi-application ecosystems with 457 APIs and about 100 virtual characters.WebShop, Mind2Web, WebArena, VisualWebArena, and AppWorld support navigation, visual interaction, cross-website generalization, and code-mediated API use.Source: §5.1.1 · pp. 38–39

5.1.2 GUI Environments

  • GUI environments provide dynamic interaction through live emulators or real computers, including AndroidWorld’s 116 tasks across 20 applications and OSWorld’s cross-platform workflows.AndroidWorld generates millions of task variations, while OSWorld covers Ubuntu, Windows, and macOS with application, file, and multi-application tasks.Source: §5.1.2 · pp. 39–40

5.1.3 Coding & Software Engineering Environments

  • Coding environments support interactive software engineering, curated programming evaluation, and long-horizon workplace tasks, with changes occurring through agent actions and execution-based verification.Debug-Gym, R2E-Gym, and TheAgentCompany provide debugging, over 8K generated tasks, and simulated professional workflows; benchmarks span synthesis, repair, development, and continual GitHub mining.Source: §5.1.3 · p. 40
  • Programmatic world-model benchmarks evaluate whether agents can synthesize executable dynamics and predict simulations, shifting assessment beyond functional correctness toward transition fidelity and planning utility.CWMB tests generated Environment step functions, while Code Simulation evaluates line-by-line execution prediction and algorithmic generalization.Source: §5.1.3 · p. 40

5.1.4 Domain-specific Environments

  • Domain-specific environments target science, machine-learning engineering, biomedical coding, data science, and cybersecurity through simulations, Kaggle workflows, executable analysis, and vulnerability repair.Examples include PaperBench’s reproduction of 20 ICML 2024 papers, MLE-Bench’s 75 Kaggle competitions, and SecRepoBench’s 27 repositories across 15 CWE categories.Source: §5.1.4 · p. 41

5.1.5 Simulated & Game Environments

  • Text, game, and multi-agent environments test language interaction, exploration, long-horizon reasoning, decentralized cooperation, and adaptation to worlds that can change without agent action.LMRL-Gym and TextWorld support language-based interaction; Crafter, Craftax, SMAC variants, and Factorio extend evaluation to survival, coordination, and tick-based industrial dynamics.Source: §5.1.5 · p. 41

5.1.6 General-Purpose Environments

  • General-purpose frameworks broaden evaluation and training through diverse interactive environments, standardized interfaces, self-correction, and more than 1000 verifiable reasoning tasks.AgentGym targets generalization, AgentBench spans SQL, game, and web scenarios, and InternBootcamp supports automated task generation across programming, logic, and games.Source: §5.1.6 · p. 41

5.2 RL Framework

This section organizes relevant codebases into Agentic RL, RLHF and LLM fine-tuning, and general-purpose RL frameworks. Table 11 summarizes these frameworks by type and key features.

Framework taxonomy

  • The framework landscape spans three categories: Agentic RL, RLHF and LLM fine-tuning, and general-purpose RL systems.Table 11 provides an overview of prevailing Agentic RL and LLM RL frameworks.Source: §5.2 · p. 41, Table 11 · p. 42

Agentic RL frameworks

  • Agentic RL frameworks support verifiable, long-horizon, distributed, multi-agent, multimodal, tool-using, and asynchronous agent training.Examples include Verifiers, SkyRL-v0, AREAL (Fu et al., 2025), MARTI (Zhang et al., 2025n), EasyR1, VerlTool (Jiang et al., 2025a), and AgentRL (Zhang et al., 2025a).Source: §5.2 · p. 42
  • 14.6× speedup over single-node execution enables AWorld to scale experience generation through massively parallel distributed rollouts.AgentFly (Wang et al., 2025j), ROLL (Wang et al., 2025o), and Agent Lightning further provide high-throughput execution, resource management, or decoupled training.Source: §5.2 · p. 42

RLHF and LLM fine-tuning frameworks

  • RLHF and LLM fine-tuning frameworks provide scalable alignment, distributed training, experiment management, and asynchronous or disaggregated post-training workflows.The category includes OpenRLHF (Hu et al., 2025b), TRL, trlX, HybridFlow (Sheng et al., 2025), and SLiMe.Source: §5.2 · p. 43

General-purpose RL frameworks

  • General-purpose RL frameworks supply reusable algorithms and distributed execution engines for agentic LLM systems, spanning production, research, online, offline, hierarchical, and model-free RL.Representative systems are RLlib (Liang et al., 2018), Acme (Hoffman et al., 2020), Tianshou (Weng et al., 2022), Stable Baselines3, and PFRL.Source: §5.2 · p. 43

6 Open Challenges and Future Directions

The field’s research frontier centers on trustworthiness, scalable agentic training, and adaptive environments, while also confronting unresolved RL mechanisms, deployment architecture, and societal impacts. Together, these challenges define the conditions for reliable, efficient, general-purpose agentic systems.

6 Open Challenges and Future Directions

  • Agentic RL faces three central challenges: trustworthy behavior, scalable training under computational and data constraints, and complex adaptive environments that expand agents’ capabilities.These challenges concern reliability, safety, alignment, efficient scaling, and the creation of effective training grounds.Source: §6 · p. 43

6.1 Trustworthiness

  • RL can amplify security, hallucination, and sycophancy risks, motivating sandboxing, process-based and sycophancy-aware rewards, adversarial training, verification, abstention, and continuously adaptive reward models.Agents’ expanded attack surfaces include tools, memory, planning, poisoned environments, and inter-agent communication; outcome-only rewards can reinforce unsafe shortcuts, ungrounded reasoning, and agreement with incorrect user beliefs.Source: §6.1 · pp. 43–44

6.2 Scaling up Agentic Training

  • Scaling agentic training requires balancing extended RL computation, model capacity, cross-domain data, and efficiency, because longer training can expand reasoning while data interference, entropy collapse, hyperparameter sensitivity, and resource costs remain limiting factors.Promising directions include hybrid SFT+RL recipes, calibrated difficulty and diversity sampling, dynamic fine-tuning, and extracting generalization from limited data.Source: §6.2 · pp. 45–46

6.3 Scaling up Agentic Environments

  • Agentic environments should become dynamic, optimizable systems that automate reward design and generate curricula targeting agent weaknesses, creating a scalable training flywheel for self-improving agents.Explorer agents can produce trajectories for reward modeling, while environment generators such as EnvGen adapt tasks within the agent’s zone of proximal development.Source: §6.3 · p. 46

6.4 The Mechanistic Debate on RL in LLMs

  • The mechanistic debate remains open: RL may mainly amplify pre-existing reasoning by reshaping sampling distributions, yet under verifiable, compositional rewards and intermediate-capability models it can also reorganize strategies and expand pass@k frontiers.In mathematical reasoning, approximately 2/3 of surveyed studies emphasize pass@1 improvements, whereas about 1/3 report expanded pass@k frontiers.Source: §6.4 · pp. 46–47

6.5 Architectural Patterns for Real-World Agent Deployment

  • Reliable deployment of RL-optimized agents requires independent guardrails, human oversight, hierarchical orchestration, and communication protocols that support safety, corrective feedback, scalable workflows, and interoperable coordination.These patterns include sandboxing and validation, synchronous or asynchronous HITL review, supervisor–worker decomposition, and fixed or learnable communication channels.Source: §6.5 · p. 47

6.6 Broader Social Impact

  • Broader deployment raises dual-use, environmental, labor, bias, and evaluation-contamination risks, requiring stronger preparedness, sustainable optimization, fairness and diversity objectives, and dynamic contamination-resistant benchmarks.These concerns include deceptive alignment and evolving capabilities, higher RL carbon costs, displacement of routine cognitive work, bias amplification, and benchmark overfitting that inflates apparent robustness.Source: §6.6 · p. 48

7 Conclusion

The survey frames Agentic RL as a shift from single-step LLM reinforcement learning to temporally extended, partially observable decision-making, and organizes the field through capabilities and applications.

Conceptual shift

  • Agentic RL elevates LLMs from passive text generators to autonomous, decision-making agents in complex, dynamic worlds.The survey distinguishes its temporally extended, partially observable MDPs (POMDPs) from conventional single-step LLM-RL decision processes.Source: §7 · p. 49

Taxonomy

  • The survey proposes a twofold taxonomy covering core capabilities—including planning, tool use, memory, reasoning, self-improvement, and perception—and diverse application domains.This taxonomy systematically maps both what agents can do and where Agentic RL is applied.Source: §7 · p. 49

Central thesis

  • Reinforcement learning transforms agentic capabilities from static, heuristic modules into adaptive, robust behavior.This mechanism is the survey’s central thesis about RL’s role in Agentic RL.Source: §7 · p. 49

Research resources

  • The survey consolidates open-source environments, benchmarks, and frameworks into a practical compendium to ground and accelerate future research.The compendium supports continued work in the emerging Agentic RL field.Source: §7 · p. 49

Related papers

Loading 2509.02547v5…