AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning

Zhiheng Xi; Jixuan Huang; Chenyang Liao; Baodai Huang; Honglin Guo; Jiaqi Liu; Rui Zheng; Junjie Ye; Jiazheng Zhang; Wenxiang Chen; Wei He; Yiwen Ding; Guanyu Li; Zehui Chen; Zhengyin Du; Xuesong Yao; Yufei Xu; Jiecao Chen; Tao Gui; Zuxuan Wu; Qi Zhang; Xuanjing Huang; Yu-Gang Jiang

Source-linked AI summary

AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning

Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang

arXiv:2509.08755v1Sep 10, 2025cs.LGcs.AIcs.CL

TL;DR

LLM-agent RL research lacks a unified framework for training multi-turn decision-making across diverse, realistic environments. AgentGym-RL provides a modular framework and ScalingInter-RL progressively increases interaction horizons; experiments report broad gains while identifying procedural and interaction-efficiency limitations.

Problem
Existing methods lack a unified interactive RL framework that trains LLM agents from scratch across diverse, realistic environments.
Method
AgentGym-RL uses a modular, decoupled framework with diverse environments and mainstream RL algorithms, while ScalingInter-RL progressively increases interaction horizons to balance exploitation and exploration.
Results
33.65 points average improvement was achieved by open-source models across five tasks spanning five scenarios, matching—or outperforming—larger commercial models.
Takeaways & Limitations
The open-sourced framework and method provide a practical toolkit and empirical resources for developing and studying long-horizon, multi-turn LLM agents.
Takeaways & Limitations
Scientific-task agents still exhibit procedural-execution failures and insufficient systematic exploration, while web-navigation agents show excessive, inefficient interaction sequences.

Abstract

from arXiv · show

Developing autonomous LLM agents capable of making a series of intelligent decisions to solve complex, real-world tasks is a fast-evolving frontier. Like human cognitive development, agents are expected to acquire knowledge and skills through exploration and interaction with the environment. Despite advances, the community still lacks a unified, interactive reinforcement learning (RL) framework that can effectively train such agents from scratch -- without relying on supervised fine-tuning (SFT) -- across diverse and realistic environments. To bridge this gap, we introduce AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL. The framework features a modular and decoupled architecture, ensuring high flexibility and extensibility. It encompasses a wide variety of real-world scenarios, and supports mainstream RL algorithms. Furthermore, we propose ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization. In early stages, it emphasizes exploitation by restricting the number of interactions, and gradually shifts towards exploration with larger horizons to encourage diverse problem-solving strategies. In this way, the agent develops more diverse behaviors and is less prone to collapse under long horizons. We perform extensive experiments to validate the stability and effectiveness of both the AgentGym-RL framework and the ScalingInter-RL approach. Our agents match or surpass commercial models on 27 tasks across diverse environments. We offer key insights and will open-source the complete AgentGym-RL framework -- including code and datasets -- to empower the research community in developing the next generation of intelligent agents.

1 Introduction

AgentGym-RL addresses the lack of a unified framework for training LLM agents through multi-turn RL across complex, diverse environments. It combines a modular framework with ScalingInter-RL and reports broad performance gains.

Existing RL studies largely target single-turn tasks, while multi-turn approaches remain limited in task complexity, environment diversity, optimization stability, and efficiency.
AgentGym-RL separates agents, environments, and learning algorithms within a flexible framework supporting mainstream RL algorithms and diverse real-world scenarios.Supported scenarios include web navigation, deep search, digital games, embodied tasks, and scientific tasks.
ScalingInter-RL progressively extends the agent–environment interaction horizon, beginning with exploitation before increasing exploration to improve stability and develop richer behaviors.The strategy is intended to encourage planning, reflection, and broader skill acquisition while reducing collapse under long horizons.
33.65 points average improvement was achieved by open-source models trained with AgentGym-RL and ScalingInter-RL, matching—or outperforming—larger commercial models across five tasks and five scenarios.
The project open-sources AgentGym-RL as a unified, modular, flexible end-to-end framework for multi-turn interactive decision-making research.

2 Preliminaries

The paper models multi-turn agentic tasks as POMDPs in which an LLM policy generates actions, receives observations, and obtains a final outcome reward. Policy-gradient RL then updates the policy toward higher expected cumulative reward.

2.1 Formulation: Agentic tasks are modeled as POMDPs with instruction, state, action, observation, transition, and reward components.
2.1 Formulation: Given an instruction, the LLM policy generates sequential actions, receives environment observations, and transitions through states until a final outcome reward describes task completion.
2.2 Policy Gradient: Policy-gradient methods directly search policy parameters rather than estimating a value function to derive the policy.
2.2 Policy Gradient: J(θ) is the expected cumulative reward over trajectories generated by policy πθ while interacting with the environment.
2.2 Policy Gradient: Mainstream algorithms integrated into the framework include PPO, GRPO, REINFORCE++, and RLOO.

3 The AgentGym-RL Framework

AgentGym-RL is a modular framework for training LLM agents through online RL across diverse, realistic multi-turn environments. It combines extensible architecture and algorithm support with ScalingInter-RL, which progressively increases interaction horizons to balance efficient exploitation, exploration, and stable optimization.

Architecture: The framework separates agents, environments, and learning algorithms through standardized interfaces, supporting flexible multi-turn training across heterogeneous environments.Parallel clients independently collect agent–environment trajectories before sending batches to the training module for policy updates.
Framework extensions: AgentGym-RL extends AgentGym with realistic environments, diverse online RL algorithms, and engineering optimizations for scalable agent training.Its extensions include Deep Search tasks, mainstream and state-of-the-art online RL methods, rollout parallelization, and memory-leak mitigation.
Environment coverage: AgentGym-RL covers web navigation, deep search, digital games, embodied tasks, and scientific tasks that require perception, planning, reasoning, reflection, and correction.These scenarios target sequential decision-making in dynamic, knowledge-intensive, and physically grounded settings.
Algorithm support: Online RL is central to AgentGym-RL, which implements mainstream algorithms including PPO, GRPO, and RLOO while also supporting SFT, DPO, and rejection sampling.The framework therefore supports both interactive policy optimization and complementary training paradigms based on demonstrations, preferences, or filtered trajectories.
Open-source and engineering support: The framework emphasizes extensibility, scalability, reliability, reproducibility, and transparency through engineering design, documentation, standardized APIs, and open-source availability.These design choices are presented as removing engineering bottlenecks and making large-scale RL experiments feasible across heterogeneous environments.
ScalingInter-RL: ScalingInter-RL progressively increases the interaction horizon, starting with constrained exploitation and later encouraging exploration, planning, reflection, strategic backtracking, and long-horizon generalization.The approach uses a monotonic horizon schedule and adaptive updates to align interaction depth with evolving policy capabilities.

4 Experiments

Experiments across five scenarios show that RL agents can learn from environment feedback without prior supervised fine-tuning, while AgentGym-RL and ScalingInter-RL produce strong cross-environment performance. Results indicate benefits from progressive interaction scaling, post-training compute, and environments with clear feedback.

RL agents learned from environment feedback without prior supervised fine-tuning, achieving performance comparable to or exceeding commercial closed-source models.
ScalingInter-RL consistently outperformed baseline training, improving WebArena by more than 10% and exceeding the TextCraft base model by 30 points.
58.6% average success rate for the 7B ScalingInter-RL model exceeded Llama3.1-70B at 47% and Qwen2.5-72B at 43%.
RL gains were strongest in structured environments: SciWorld improved from 1.50 to 50.50, while WebArena and Deep Search showed more moderate positive gains.
ScalingInter-7B achieved 26.00% on WebArena, 38.25 on Deep Search, 91.00 on the reported benchmark, 96.67% on BabyAI, and 57.00 on SciWorld.

5 Discussion

Discussion experiments examine how interaction turns, sampling, RL algorithms, and environmental structure affect agent learning. ScalingInter-RL improves stability and efficiency, but case studies reveal remaining procedural and action-selection failures.

Scaling interaction and sampling: Increasing interaction turns improves inference-time performance, supporting the value of deeper environmental exploration.
Scaling interaction and sampling: 64 sampling attempts produced a 5.5% improvement in Deep Search and a 7.05% improvement in another reported environment.
Algorithm comparison: GRPO consistently outperformed REINFORCE++ on TextCraft, BabyAI, and Deep Search, with GRPO-3B exceeding REINFORCE++-7B.
Case studies: RL case studies show improved backtracking, task interpretation, error recovery, and purposeful web navigation compared with base agents.
Case-study limitations: SciWorld failures persist when debugging procedures, conducting systematic exploration, and executing rigorous comparative analyses.
Case-study limitations: Web navigation still includes redundant clicking, hovering, and scrolling after agents reach correct target websites.

6 Related Work

Prior work uses prompting, reflection, planning, and self-correction to build multi-turn agents, while RL and inference-time compute provide additional routes for improving language-model capabilities. Existing approaches remain constrained by cost, scalability, and limited multi-turn environmental coverage.

Prompt-based agent workflows commonly combine tool use with self-reflection, long-horizon planning, and self-correction.
Self-reflection approaches are expensive, difficult to scale, and do not reliably support self-improvement through environmental interaction.
Most RL studies for language models remain confined to single-turn, static tasks despite algorithms including PPO, GRPO, REINFORCE++, and RLOO.
Inference-time scaling methods include long-chain-of-thought reasoning, majority voting, best-of-N sampling, beam search, and Monte Carlo tree search.

7 Conclusion and Future Work

AgentGym-RL and ScalingInter-RL provide a unified framework and staged interaction strategy for long-horizon, multi-turn agent training. The authors identify generalization, physical-world scaling, and multi-agent training as important future directions.

AgentGym-RL is a unified, extensible framework for training LLM agents on long-horizon, multi-turn decision-making tasks.
Future directions: Current agents perform well in in-domain settings but must adapt to novel environments and unfamiliar tools while maintaining performance.
Future directions: Most existing studies, including this work, focus on relatively simple digital tasks rather than longer-horizon, physically grounded tasks.
Future directions: Physically grounded tasks require richer sensory inputs and reasoning over larger action spaces, challenging RL training and test-time interaction.
Future directions: The current framework primarily targets single-agent training, while multi-agent architectures introduce additional uncertainty and demands on infrastructure and algorithm design.

A Details of the AgentGym-RL Architecture

AgentGym-RL uses modular Environment, Agent, and Training components to separate interaction, reasoning, and optimization. The architecture supports scalable, reproducible, and flexible RL experimentation.

The architecture comprises three core modules: Environment, Agent, and Training.
Environment module: Each environment runs as an independent service with optional replicas, exposing APIs for observations, available actions, stepping, and resetting.
Agent module: The Agent module implements the multi-turn reasoning–action loop and supports varied prompting strategies, sampling configurations, and reward functions.
Training module: The Training module provides online and offline RL pipelines covering trajectory collection, advantage estimation, policy optimization, reward shaping, curriculum learning, and staged interaction scaling.
Distributed execution, efficient batching, asynchronous logging, and diagnostics support throughput scaling and reproducible analysis.

B Implementation Details and Settings of Each Environment

The experiments use NVIDIA A100 GPUs and Ascend 910B NPUs, with the remaining settings described for each environment.

Experiments were conducted on NVIDIA A100 GPUs and Ascend 910B NPUs.
The section provides detailed settings for the different environments.

B.1 Web Navigation Scenario

The web navigation scenario evaluates agents that use browser actions and tool APIs to complete tasks across several online platforms. Its settings include five subtasks and exclude Content & Config operations for efficient parallel rollout.

WebArena supports web-based tasks across online shopping, Reddit discussions, GitLab collaboration, and CMS store management through tool APIs.
A web-navigation case supplies an objective, accessibility tree, current URL, open tabs, and previous action to the agent.
Action categories: Agents can perform page operations such as clicking, typing, hovering, pressing keys, and scrolling.
Action categories: Agents can open, focus, and close tabs, navigate URLs, move backward or forward, and submit a final answer.
Tools and APIs: The scenario provides a homepage utility for visiting other websites.
Example task: The displayed example asks the agent to identify book names from qualifying posts among the top ten posts in a books forum.
Settings: The scenario includes five subtasks—E-commerce, Reddit, GitLab, OpenStreetMap, and CMS—with 372 training queries and 50 testing queries.

B.2 Deep Search Scenario

The paper presents several interactive environments and specifies their APIs, task settings, and interaction limits for training and evaluation.

Deep Search Scenario: Deep Search uses search-engine APIs for iterative querying, retrieval, and incorporation of external information into subsequent reasoning.The setting samples 400 development examples from seven datasets and limits agents to four interaction turns.
Scenario Design: Across these scenarios, the benchmark descriptions define concrete action interfaces and experimental settings for multi-turn agent-environment interaction.Examples include sequential search, crafting, navigation, and scientific manipulation tasks.
Digital Games Scenario: TextCraft mirrors Minecraft with APIs for crafting, inventory management, navigation, quest generation, and recursive task decomposition.Tasks are grouped by crafting-tree depths from 1 through 4, with a maximum of 20 interactions.
Embodied Scenario: BabyAI provides a controllable grid world where agents navigate, move objects, unlock doors, and interact using natural-language instructions.The benchmark divides tasks into six subsets by final goal and allows up to 20 interactions.
Scientific Scenario: SciWorld supports scientific exploration through text-driven cycles involving apparatus interaction, measurement, circuit connection, and chemical mixing.The evaluation selects eight task subsets and allows up to 20 agent-environment interactions.

C Trajectory examples and visualizations of our RL agent

The trajectory examples compare baseline and RL-trained agents across embodied, scientific, and web-navigation tasks. They highlight stronger exploration and task execution alongside persistent logical and interaction failures.

Overview: The appendix reports superior RL-agent performance in exploration, task execution, and interaction patterns across multiple environments.It also documents common failure modes that remain after training.
BabyAI: In BabyAI, the examples include a direct RL-agent comparison and a trajectory visualization for an embodied navigation task.The supplied example asks the agent to reach a blue box through a room with closed doors.
SciWorld: In SciWorld, the examples include successful RL-agent behavior, logical errors, and a task involving moving a living thing to a target box.The successful case concerns identifying and manipulating objects in a multi-room scientific environment.
WebArena: WebArena examples specifically document over-interaction failures in web-navigation behavior.The figure set treats repetitive or excessive interaction as a remaining failure mode rather than only showcasing successes.