PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework

Hanchen Zhang, Xiao Liu, Bowen Lv, Xueqiao Sun, Bohao Jing, Iat Long Iong, Zhenyu Hou, Zehan Qi, Hanyu Lai, Yifan Xu, Rui Lu, Hongning Wang, Jie Tang, Yuxiao Dong

arXiv:2510.04206v1cs.AI

TL;DR

Multi-turn, multi-task RL for LLM agents faces challenges in scalable infrastructure and stable training, while prior work has largely focused on single-turn or single-task settings. AGENTRL combines asynchronous training, heterogeneous-environment deployment, and algorithmic changes. It reports outperforming several leading agents and matching task-specific models with one jointly trained model.

  • Problem

    LLM reinforcement learning has largely focused on single-turn, single-task settings, while scalable infrastructure and stable algorithms for multi-turn, multi-task agentic RL remain challenging.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    AGENTRL combines asynchronous rollout-training, scalable environment deployment, cross-policy sampling, and task advantage normalization for multi-turn, multi-task agentic RL.

    Source: §1 · p. 2, §3.1 · pp. 4–5

  • Results

    AGENTRL significantly outperforms GPT-5, Claude-Sonnet-4, and DeepSeek-R1; one model trained on five tasks matches the best performance of five task-specific models and generalizes to unseen tasks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The reported results support the potential of training a generalist LLM agent across diverse agentic tasks.

    Source: §1 · p. 3

    Cross-policy sampling can introduce minor distribution shifts that manifest as mild, transient training instabilities; the evaluation focuses on controlled environments.

    Source: §G.1 · p. 24

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions. However, applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms. In this work, we present the AgentRL framework for scalable multi-turn, multi-task agentic RL training. On the infrastructure side, AgentRL features a fully-asynchronous generation-training pipeline for efficient multi-turn RL. To support heterogeneous environment development in multi-task RL, we design a unified function-call based API interface, containerized environment development, and a centralized controller. On the algorithm side, we propose cross-policy sampling to encourage model exploration in multi-turn settings and task advantage normalization to stabilize multi-task training. Experiments show that AgentRL, trained on open LLMs across five agentic tasks, significantly outperforms GPT-5, Clause-Sonnet-4, DeepSeek-R1, and other open-source LLM agents. Multi-task training with AgentRL matches the best results among all task-specific models. AgentRL is open-sourced at https://github.com/THUDM/AgentRL. The algorithm and framework are adopted in building \textsc{\href{https://autoglm.zhipuai.cn}{AutoGLM}}.

1 Introduction

RL for LLMs has largely focused on single-turn, single-task settings, while scalable multi-turn, multi-task agentic RL requires infrastructure and algorithmic advances. AGENTRL combines these designs and reports strong results across five agentic tasks, including multi-task performance matching task-specific models.

Motivation

  • Prior RL-for-LLM work has largely remained in single-turn, single-task settings, while generalist agents for dynamic multi-turn interactions require advances in infrastructure and algorithms.The introduction frames agentic RL as autonomous multi-turn reasoning, interaction with tools or environments, and adaptation over extended trajectories.Source: §1 · p. 2

Contributions

  • AGENTRL combines asynchronous training, scalable heterogeneous-environment deployment, cross-policy sampling, and task advantage normalization for multi-turn, multi-task agentic RL.The infrastructure targets efficiency and environment scaling; the algorithms target exploration and training stability.Source: §1 · pp. 2–3

Results

  • The reported results significantly outperform GPT-5, Claude-Sonnet-4, and DeepSeek-R1 across five agentic tasks.The experiments use open LLMs across ALFWorld, DB, KG, OS, and Webshop.Source: §1 · pp. 2–3
  • One model trained jointly on five tasks matches the best performance of five separately trained task-specific models and generalizes to unseen tasks such as BFCL-v3.The paper also reports consistent benefits from its algorithmic design choices in extensive ablations.Source: §1 · p. 3

2 The Agentic RL Problem and its Challenges

Agentic RL extends LLM interaction from a single response to multi-turn reasoning and adaptation, while generalist training across tasks remains underexplored. The section identifies infrastructure and algorithmic challenges in scaling this setting.

The Agentic RL Problem

  • Agentic RL involves multi-turn LLM reasoning, interaction with tools or environments, and adaptation over extended trajectories.The paper frames this as a multi-step Markov decision process rather than a single-step bandit.Source: §2 · p. 3

Multi-Task Challenges

  • Most LLM agents train a separate policy for each task, leaving generalist agents that handle diverse tasks largely unexplored.Multi-task training must jointly optimize tasks while maintaining stability.Source: §2 · pp. 3–4

Multi-Turn Infrastructure Challenges

  • Variable, long multi-turn trajectories leave GPUs waiting in synchronous training, reducing efficiency and impeding RL scaling.The section motivates an asynchronous training framework to address this imbalance.Source: §2 · p. 3

Environment Infrastructure Challenges

  • Interactive multi-turn training requires concurrent deployment and management of many environments.Across tasks, differing interfaces, state-action representations, and computational demands complicate scalable environment integration.Source: §2 · p. 3

Algorithm Challenges

  • Existing sampling strategies largely target single-turn settings, making exploration and sampling efficiency important multi-turn algorithm challenges.Joint optimization across agentic tasks also remains difficult when training stability must be maintained.Source: §2 · pp. 3–4

3 The AGENTRL Framework

AGENTRL combines asynchronous rollout and training, scalable environment deployment, and algorithmic designs for exploration and stable multi-task optimization. Its framework addresses both multi-turn and heterogeneous multi-task training.

3.1 Multi-Turn Agentic RL

  • Asynchronous rollout and training run concurrently, allowing training to consume available data without waiting for a full rollout batch and improving throughput over synchronous training.The scheduler fills idle GPU slots with available coroutines; the throughput comparison is for 14B Qwen2.5 models on Webshop.Source: §3.1 · pp. 4–5, Figure 3 · p. 4, Figure 4 · p. 5
  • AGENTRL deploys scalable task environments through a unified function-call API, isolated containers, and a centralized controller managing thousands of parallel episodes.Workers handle task lifecycles, while the controller orchestrates the training engine.Source: §3.1 · p. 5, Figure 5 · p. 5
  • Cross-policy sampling randomly draws actions at each step from multiple models, broadening candidate trajectories while keeping language states valid.The strategy targets goal-relevant states that might not appear under any single model.Source: §3.1 · pp. 5–6

3.2 Multi-Task Agentic RL

  • Consistent worker and controller interfaces let heterogeneous tasks use shared lifecycle operations and a single gateway to the RL engine.These interfaces abstract task heterogeneity and support scaling the task set in size and diversity.Source: §3.2 · p. 6
  • Task advantage normalization addresses differences in task difficulty, sequence length, and sampling efficiency that can produce uneven learning and instability.Advantages are normalized within each task batch so token-level advantages have zero mean and unit variance, reducing inter-task variance.Source: §3.2 · p. 6

4 Experiments

Across five agentic tasks, AGENTRL achieves a 70.4% average success rate and its multi-task model approaches specialist performance while generalizing to BFCL-v3. Ablations and WebShop experiments examine how cross-policy sampling and task advantage normalization affect exploration and training stability.

4 Experiments

  • Experiments use Qwen2.5-Instruct and GLM-4-9B-0414 across five tasks, replicate smaller datasets for approximately uniform sampling, and compare against closed APIs, open models, and trained agents.The five tasks are ALFWorld, DB, KG, OS, and WebShop.Source: §4 · pp. 6–7, §4.1 · p. 7

4.1 Main Results

  • AGENTRL reaches a 70.4% average success rate across five AGENTBENCH-FC tasks, and its 3B–32B models outperform GPT-5, Claude-Sonnet-4 Thinking, and DeepSeek-R1.Table 3 reports task success rates as averages and standard deviations over four repeats.Source: §4.1 · pp. 7–8, Table 3 · p. 7
  • One multi-task AGENTRL model achieves nearly the best-of-five specialist performance while maintaining strong results across tasks, unlike single-task agents that transfer poorly.The authors describe this as acquiring generalizable skills without sacrificing peak performance.Source: Table 4 · p. 7, §4.1 · p. 8
  • On BFCL-v3, AGENTRL shows clear improvements on multi-turn tasks and modest gains on single-turn tasks after training on five other agentic tasks.The authors present this as evidence of improved function-calling generalization.Source: §4.1 · p. 8, Table 5 · p. 8

4.2 Ablation Study

  • Removing cross-policy sampling worsens performance, particularly in some environments; on KG, capability reaches its peak earlier during training.The authors attribute the gains to exploring more possible states, especially in open-ended environments.Source: §4.2 · p. 8
  • Removing task advantage normalization reduces performance and training efficacy, produces fluctuations, and lets tasks learn at different rates rather than jointly.The authors say per-task advantage normalization stabilizes multi-task training and reduces negative interference.Source: §4.2 · p. 8, Table 6 · p. 8

4.3 Verifying the Effect of the Cross-Policy Sampling Strategy

  • In WebShop inference, cross-policy sampling trails the best single model at low k but surpasses individual models and mixed trajectories in pass@k as k grows.The comparison evaluates Qwen and Llama models.Source: §4.3 · p. 9, Figure 8 · p. 9
  • In WebShop RL training, both models improve pass@1 over the untrained base, while cross-policy sampling maintains an advantage as k increases.The authors interpret this as preserving model diversity while improving overall ability.Source: §4.3 · p. 9, Figure 8 · p. 9

5 Related Work

Prior work applies RL to LLM agents, GUI tasks, and long-horizon research, while infrastructure and environment platforms support training or evaluation. The cited literature leaves exploration and multi-task training under-studied, and existing RL frameworks lack built-in support for multi-turn agent interactions.

Reinforcement Learning AI Agents

  • PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024) are widely adopted in LLM agent training, alongside work on GUI agents and long-horizon research.The passage cites DeepSeek-R1 (DeepSeek-AI et al., 2025) as an example of reward-driven fine-tuning for reasoning.Source: §5 · p. 9
  • The related work identifies exploration and multi-task training as under-studied in current approaches, motivating cross-policy sampling and task advantage normalization.The passage describes this as a gap in existing methods.Source: §5 · p. 9

Reinforcement Learning Infrastructure

  • Existing RL frameworks use modern training and rollout engines, while platforms such as E2B and OpenHands support evaluation environments; these frameworks lack built-in multi-turn interaction support and agent-specific training optimizations.The passage also notes standardized benchmarks addressing reproducibility gaps in multi-turn interaction evaluation.Source: §5 · pp. 9–10

6 Conclusion

AGENTRL combines asynchronous rollout–training, scalable environment deployment, and algorithmic methods for training LLM agents across tasks; experiments show competitive benchmark results and encouraging generalization to unseen tasks.

6 Conclusion

  • AGENTRL combines asynchronous rollout–training, scalable environment deployment, cross-policy sampling, and task advantage normalization for more efficient and stable multi-task agent training.Experiments report competitive results across agentic benchmarks and encouraging generalization to unseen tasks.Source: §6 · p. 10

A Background of Reinforcement Learning in Large Language Models

  • PPO clips policy ratios, GRPO uses group-relative advantages, and DAPO adds decoupled clipping and dynamic sampling to address long-CoT entropy collapse and training instability.DAPO also normalizes loss at token level to weight responses of varying lengths.Source: A Background of Reinforcement Learning… · p. 14, §B.2 · p. 16

B.1 Problem Definition

  • Agentic tasks are MDPs with composite environment-and-context states; LLM actions are token sequences with factorized probabilities, enabling token-level policy gradients and advantage estimates.A trajectory records successive states, actions, and rewards.Source: §B.1 · p. 15
  • Across multiple tasks, samples produce groups of trajectories that GRPO uses to compute group-based advantage estimates.Each task has its own samples, and executing a sample yields a group of trajectories.Source: §B.1 · p. 15

B.2 Reinforcement Learning with Verifiable Rewards (RLVR)

  • RLVR computes trajectory rewards deterministically and objectively from observed interaction data, and is commonly optimized with PPO or GRPO.GRPO compares trajectories within groups to produce group-relative advantage estimates.Source: §B.2 · pp. 15–16

C Formal Description and Intuition of the Cross-Policy Sampling Strategy

  • Cross-policy sampling broadens coverage of valid language states that can reach success, increasing success-state probability by exploring more paths without leaving valid language.The formulation treats a language state as stochastically mapping to environment states through a grounding function.Source: C Formal Description and Intuition of… · p. 16

D Dataset Details

The dataset work refines AGENTBENCH, assembles training data suited to different environments, and adapts its environments and interfaces for function-call interactions.

D.1 Extending AGENTBENCH

  • AGENTBENCH-FC is the refined AGENTBENCH version used for training, while the overall framework remains benchmark-decoupled.Source: §D.1 · p. 16

D.2 Synthesizing Training Set

  • Training data combines official datasets for AlfWorld and WebShop, Self-Instruct generation for OS, KnowledgeGraph, and DB, and BIRD samples for DB.The original AGENTBENCH training data was considered scarce.Source: §D.2 · p. 16

D.3 Modifications to AGENTBENCH Environment

  • The five environments were converted to a function-call framework with environment-specific tools and revised Controller and Worker interfaces for task initiation, interaction, and monitoring.The KG environment, for example, received seven tools, including get_relations, get_neighbors, and count.Source: §D.3 · p. 17

E.1 Environments and Tasks

The evaluation tasks cover five varied multi-turn environments, with rewards standardized across tasks and a distinct no-one-shot condition for trained KG models.

E.1 Environments and Tasks

  • The five AgentBench tasks span OS, DB, KG, text adventure (ALFWorld), and web shopping (WebShop), covering varied reasoning, decision-making, and environmental interaction demands.The selected tasks have standardized evaluation protocols and open-source code environments.Source: §E.1 · pp. 17–18
  • All task rewards are normalized to [0, 1], with binary correctness rewards when intrinsic signals are absent and a −0.2 penalty for abnormal termination.Source: §E.1 · p. 17
  • KG API results use one-shot testing, but trained models are trained and evaluated without one-shot assistance.Source: §E.1 · p. 17

E.2 Training and Evaluation Settings

Training uses asynchronous multi-task RL across five interaction tasks, with model-specific initialization and standardized evaluation; the deployment framework isolates and manages environment workers.

E.2 Training and Evaluation Settings

  • AGENTRL builds on Verl and trains Qwen2.5 and GLM4-9B models for over 1,000 steps in a mixed five-task setting.The models include Qwen2.5-3B, 7B, 14B, and 32B Instruct variants and GLM4-9B.Source: §E.2 · p. 18
  • Qwen training starts from base models, whereas GLM4-9B first receives limited SFT data to adapt to function-call format; training uses H800 GPUs and scales with additional GPU resources.Source: §E.2 · p. 18
  • GRPO training uses temperature 0.8, eight samples per rollout, binary trajectory correctness rewards, and a −0.2 penalty for exceeding interaction or response limits.SGLang inference and FSDP support training efficiency.Source: §E.2 · p. 18

E.3 Deployment Framework Details

  • Containerized workers manage concurrent isolated task lifecycles, while the controller uses non-blocking dispatch, timeout-driven fault recovery, and periodic lifecycle cleanup.Worker instrumentation supports session- and task-level telemetry; controller routines can de-register and reintegrate unstable nodes.Source: §E.3 · p. 18

E.4 Results Analysis

AgentRL-trained models show scaling, strong frontier-model performance, and gains on an unseen benchmark. The analysis also reports benefits from its algorithmic choices and illustrates cross-policy sampling on a knowledge-graph task.

E.4 Results Analysis

  • Performance improves consistently as AgentRL-trained models increase in size, and the framework also applies to a different model architecture.The reported scaling trend runs from the smallest to the largest model variants.Source: §E.4 · p. 19
  • AgentRL’s largest model surpasses leading API-based models on certain multi-turn and overall metrics.The comparison describes a substantial improvement over the model’s pre-RL base version.Source: §E.4 · p. 19
  • On unseen BFCL-v3 tasks, the RL-trained model improves overall performance over its base model, especially on multi-turn tasks.The passage presents BFCL-v3 as an out-of-distribution evaluation of generalization beyond training tasks.Source: §E.4 · p. 19
  • Ablations report that cross-policy sampling raises average performance, while task advantage normalization mitigates negative interference and rate disparities across tasks.The paper associates the latter technique with greater training stability and consistency.Source: §E.4 · p. 19

E.5.1 Case Study on the Efficacy of Cross-Sampling

  • GLM-4 fails by looping on a premature conclusion, whereas Llama repeatedly makes tool calls with incorrect logic and parameters.The cross-policy case study switches to Llama’s policy to prompt tool interaction while retaining guidance from GLM-4’s logic.Source: §E.5.1 · pp. 19–20
  • In a knowledge-graph question-answering case, cross-policy sampling combines GLM-4’s logic with Llama’s tool interaction to reach the correct answer.The figure caption says GLM-4 enters a conclusion loop and Llama falters with tool comprehension when acting individually.Source: Figure 9 · p. 20, §E.5.1 · p. 20

E.5.2 Error Analysis

AgentRL improves completion and reduces task-limit terminations across five environments. In an ALFWorld example, the trained model succeeds where the base model fails, and the appendices specify how agents interact with each task environment.

E.5.2 Error Analysis

  • In AlfWorld, Completed rates rise from 0.070 to 0.926 and Task Limit Reached rates fall from 0.68 to 0.074 with AGENTRL.The analysis says completion improves and task-limit termination decreases across all five environments.Source: §E.5.2 · p. 20

E.5.3 What Reinforcement Learning Teaches Models in ALFWorld

  • On the saltshaker task, the base model fails in all four runs, while the RL-trained model succeeds in all four.The trained model uses the environment tool correctly, searches countertops, then navigates to a drawer and places the saltshaker.Source: §E.5.3 · p. 21
  • The base model repeats invalid actions and fixates on cabinets, while the trained model shows tool proficiency, strategic exploration, and effective action sequencing.The reported contrast includes prioritizing likely locations and avoiding procedural errors.Source: §E.5.3 · p. 21

F.1 AlfWrold Task

  • The AlfWorld prompt requires agents to submit each chosen household action through the provided tool and use environment feedback to plan subsequent turns.Source: §F.1 · p. 21

F.2 Knowledge Graph (KG) Task

  • The Knowledge Graph prompt directs agents to query a knowledge base with tools, track generated variables, and return the final answer as a variable ID.It sets a maximum of 15 actions and instructs agents not to call tools after identifying the answer.Source: §F.2 · p. 22

F.3 DB Task

  • The DataBase prompt has agents explain their solution, issue one SQL statement per turn through execute_sql, and submit the final answer with commit_final_answer.Source: §F.3 · p. 23

F.4 OS Task

  • The Operating System prompt requires one provided tool call per turn and directs agents to use finish_action or answer_action when appropriate.It also warns that truncated output may require handling the truncation.Source: §F.4 · p. 23

F.5 Webshop Task

  • The Web Shopping prompt asks agents to act from each round’s observation and available actions, using search or valid clicks through tools.Source: §F.5 · p. 23

G.1 Limitations

The authors identify mild, transient training instability from cross-policy sampling and limit validation to controlled environments. They propose refinement of the sampling strategy and evaluation in more complex, dynamic real-world settings as future work.

G.1 Limitations

  • Cross-policy sampling can introduce minor distributional shifts that manifest as mild, transient instability during training.The authors describe this as a manageable trade-off for broader state-space coverage and suggest adaptive policy weighting as a possible refinement.Source: §G.1 · p. 24
  • The framework is validated across controlled environments; applying it to more complex, dynamic real-world scenarios remains future work.The authors identify broader real-world application as a next step after their controlled-environment validation.Source: §G.1 · p. 24, §G.2 · p. 24

Related papers

Loading 2510.04206v1…