PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang

arXiv:2406.04151v1cs.AIcs.CL

TL;DR

LLM agents face a tension between supervision-heavy imitation and self-improvement confined to isolated tasks. AGENTGYM pairs diverse environments and trajectory resources with AGENTEVOL, and the paper reports that evolved agents surpass strong baselines on many tasks.

  • Problem

    Imitation-based LLM agents require costly skilled annotation and provide limited exploration, while self-improvement in isolated environments yields specialists with limited generalization.

    Source: §1 · p. 2

  • Method

    AGENTGYM combines diverse environments and trajectory sets with AGENTEVOL, which alternates exploration using environmental feedback and learning from sampled experience.

    Source: §1 · pp. 2–3, §4.2 · p. 6

  • Results

    AGENTEVOL surpasses BClarge and SOTA models on many tasks, including WebShop, ALFWorld, and BabyAI.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    AGENTGYM is presented as a framework to help the community develop generally-capable agents and explore self-evolution.

    Source: §1 · p. 2

    For computational efficiency, the study does not perform multiple samplings per iteration, although more sampling improves results by a nonsignificant amount in its analysis.

    Source: A Limitations · p. 16

Abstract

from arXiv · show

Building generalist agents that can handle diverse tasks and evolve themselves across different environments is a long-term goal in the AI community. Large language models (LLMs) are considered a promising foundation to build such agents due to their generalized capabilities. Current approaches either have LLM-based agents imitate expert-provided trajectories step-by-step, requiring human supervision, which is hard to scale and limits environmental exploration; or they let agents explore and learn in isolated environments, resulting in specialist agents with limited generalization. In this paper, we take the first step towards building generally-capable LLM-based agents with self-evolution ability. We identify a trinity of ingredients: 1) diverse environments for agent exploration and learning, 2) a trajectory set to equip agents with basic capabilities and prior knowledge, and 3) an effective and scalable evolution method. We propose AgentGym, a new framework featuring a variety of environments and tasks for broad, real-time, uni-format, and concurrent agent exploration. AgentGym also includes a database with expanded instructions, a benchmark suite, and high-quality trajectories across environments. Next, we propose a novel method, AgentEvol, to investigate the potential of agent self-evolution beyond previously seen data across tasks and environments. Experimental results show that the evolved agents can achieve results comparable to SOTA models. We release the AgentGym suite, including the platform, dataset, benchmark, checkpoints, and algorithm implementations. The AgentGym suite is available on https://github.com/WooooDyy/AgentGym.

1 Introduction

The introduction frames generalist agents as needing to move from imitation toward learning across environments, while existing LLM-agent approaches face supervision and generalization limits. It identifies diverse environments, a trajectory set, and an evolution method as ingredients for investigating self-evolution.

1 Introduction

  • Imitation from expert trajectories requires skilled annotators and substantial resources, while isolated-environment self-improvement produces specialists with limited generalization.The paper also notes possible bottlenecks in performance and adaptability from insufficient exploration.Source: §1 · p. 2
  • The proposed research goal is to investigate self-evolution from imitation to interactive learning across tasks and environments, using diverse environments, a trajectory set, and an evolution method.The trajectory set equips a base agent with preliminary instruction-following abilities and knowledge; the evolution method is intended to adapt across varying task difficulties.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper presents AGENTGYM as a framework for developing generally capable LLM-based agents and exploring self-evolution.The paper describes this as an initial step toward agents that acquire skills through imitation and then adapt through interaction.Source: §1 · pp. 1–2

1. An interactive platform that includes diverse environments, tasks, and goals for LLM-based

AGENTGYM combines a diverse interactive platform with expanded instructions, benchmark and trajectory resources, and AGENTEVOL for studying agent evolution across environments. The framework uses AGENTTRAJ to train a base agent before further exploration and learning.

1. An interactive platform that includes diverse environments, tasks, and goals for LLM-based

  • AGENTGYM provides 14 types of agent environments and 89 task types, with unified APIs for multi-round interaction, real-time feedback, online evaluation, trajectory sampling, and interactive training.The platform spans web, embodied, and other tasks and is designed to support expansion.Source: §1 · p. 2, Figure 2 · p. 3
  • The framework expands instructions, selects a diverse and challenging evaluation subset for AGENTEVAL, and collects uniform-format expert-annotated trajectories as AGENTTRAJ.AGENTTRAJ trains the base generally-capable agent that bootstraps further exploration and evolution.Source: §1 · pp. 2–3
  • AGENTEVOL starts from the base agent and explores evolution across multiple environments using feedback on previously unseen tasks and instructions.The paper reports that evolved agents achieve comparable or better performance than SOTA models, with gains on many tasks.Source: §1 · p. 3
  • The suite brings together the platform, AGENTEVAL, AGENTTRAJ, AGENTTRAJ-L, and AGENTEVOL to investigate self-evolution in generally-capable agents.The paper states that it will release the suite, algorithm implementations, and agent checkpoints.Source: §1 · p. 3

2 Preliminaries

The preliminaries model each environment’s task as a POMDP and describe an agent that alternates reasoning thoughts and actions in response to interaction feedback. A trajectory records that interaction, after which a final reward is computed.

2 Preliminaries

  • Each environment’s task is formalized as a POMDP with instruction, state, action, observation, deterministic transition, and reward components.The formalization uses the tuple (U, S, A, O, T, r)e.Source: §2 · p. 3
  • At each interaction step, the agent generates a reasoning thought before its next action, conditioned on the instruction and the history of actions and feedback.The paper adopts ReAct to model agent outputs.Source: §2 · pp. 3–4
  • An interaction trajectory comprises the agent’s thoughts, actions, and observations, and the final reward is computed from the environment, instruction, and trajectory.The interaction continues until task completion or the maximum number of steps.Source: §2 · p. 4

3 AGENTGYM: Platform, Benchmark Suite and Trajectory Set

AGENTGYM standardizes interaction across diverse environments and tasks, and combines that platform with an evaluation benchmark and filtered trajectory sets. The section describes the platform architecture, instruction and benchmark construction, and trajectory collection.

3 AGENTGYM: Platform, Benchmark Suite and Trajectory Set

  • AGENTGYM covers 14 environments and 89 tasks across web navigation, text games, household, digital game, embodied, tool-using, and programming tasks, using a unified ReAct format with real-time feedback and concurrency.Table 1 compares frameworks by environment count, interactive platform and usage, trajectory sets, and support and mode of evolution.Source: §3 · p. 4, Table 1 · p. 4
  • Separate environment services communicate over HTTP, while a controller provides agents an encapsulated, unified interface to environment functions and operations.The platform also includes evaluator, trainer, and data-collection components.Source: §3 · p. 4
  • The authors collected 20,509 instructions and queries across environments and tasks, then extracted a diverse, challenging subset of 1,160 evaluation instructions from each environment to construct AGENTEVAL.For tasks with fewer instructions, such as tool-using tasks, they expanded them using self-instruct and instruction evolution, prompting GPT-4 to generate new instructions [33].Source: §3 · p. 4
  • Filtering trajectories collected with SOTA models and crowdsourced annotations yields 6,130 AGENTTRAJ examples for base-agent training; AGENTTRAJ-L applies the same pipeline to all instructions as a BC performance upper bound.Table 2 reports task-type counts, instruction and evaluation set sizes, trajectory-set sizes, and average rounds by environment.Source: Table 2 · p. 4, §3 · p. 5

4 AGENTEVOL for Evolution of Generally-capable LLM-based Agents

AGENTEVOL starts from a behaviorally cloned base agent and alternates exploration with learning to investigate self-evolution across tasks. Empirically, the method facilitates evolution across seen and unseen tasks and instructions.

4.1 Behavioral Cloning with Collected Trajectories

  • Behavioral cloning trains the base agent to mimic expert trajectories step-by-step, including inner thoughts and actions; general-domain data helps maintain language ability.The resulting base agent starts later evolution across tasks and environments.Source: §4.1 · p. 5

4.2 Evolution through Exploration and Learning

  • AGENTEVOL estimates a reward-weighted trajectory distribution from the current policy, then updates the agent toward that distribution.This inference-inspired approach addresses standard RL's high computational complexity and training instability in long-horizon agent tasks.Source: §4.2 · pp. 5–6
  • Each iteration samples trajectories from a larger instruction set, scores them, retains the original trajectories, and trains on the resulting data with general-domain data.Source: §4.2 · p. 7
  • Alternating exploration and learning facilitates agent evolution across both seen and unseen tasks and instructions.Source: §4.2 · p. 7

5 Experiments and Discussion

Across diverse tasks, AGENTEVOL outperforms the larger behavioral-cloning baseline on most tasks and environments, and on several tasks surpasses SOTA models. Ablations show how trajectory merging, iteration count, sampling, exploration scope, backbone choice, and failed trajectories relate to evolution performance.

5.1 Experimental Setup

  • Experiments cover 11 environments, with fewer instructions used in behavioral cloning than during evolution to study generalization to exploration.Baselines include closed- and open-source models, AgentLM, and BClarge, trained on the larger AGENTTRAJ-L set.Source: §5.1 · p. 7

5.2 Main Results

  • AGENTEVOL outperforms BClarge on most tasks and environments, and surpasses it and SOTA models on WebShop, ALFWorld, and BabyAI.BClarge is the paper’s larger behavioral-cloning baseline and represents its stated performance upper limit for BC.Source: Table 3 · p. 8, §5.2 · p. 8

5.3 Discussion & Analysis

  • Merging each iteration’s trajectories with the initial set yields more stable improvements than merging with the previous iteration’s trajectories, which causes fluctuations.Performance tends to improve as M increases but gradually converges; the authors choose M = 4 to balance performance and efficiency.Source: §5.3 · p. 8, Figure 3 · p. 8
  • Increasing the number of samples K improves performance, but only modestly, so the authors select K = 1 for computational efficiency.The ablation covers four tasks.Source: §5.3 · p. 8, Table 4 · p. 9
  • With limited exploration instructions, the base agent improves, possibly from more diverse sampled trajectories, but the improvement is not significant.The authors conclude that effective evolution needs a more extensive environment.Source: §5.3 · p. 9
  • Across Llama-2-13B and DeepSeek-Coder-1.3B backbones, AGENTEVOL achieves performance comparable to or surpassing BClarge.These experiments also use AGENTGYM.Source: §5.3 · p. 9, Table 5 · p. 9
  • Using successful and failed trajectories with DPO still produces evolutionary effects, but performs worse than the authors’ method.The authors describe preference optimization in multi-task settings as more challenging than in single-task settings.Source: §5.3 · p. 9, Table 6 · p. 9

6 Related Work

Prior work trains LLM-based agents through expert imitation or environmental interaction, while existing self-evolution methods explore within a single environment or task. AgentGym applies agent evolution across multiple environments.

Benchmarking

  • AGENTEVAL offers more diverse environments and tasks than prior agent benchmarks, aiming for more comprehensive evaluation.Prior benchmarks include varied tasks, but the passage does not specify their comparative scope beyond this characterization.Source: §6 · p. 9

Behavioral Cloning

  • Behavioral cloning trains agents on expert trajectories, but collecting those trajectories is costly and they provide insufficient environmental exploration.Closed-source LLM agents can perform well on agent tasks, while open-source approaches have performed poorly.Source: §6 · p. 9

Interactive Learning and Self-Evolution

  • Multi-environment reinforcement learning can face reward-consistency and training-stability problems, while prior self-evolution methods explore only one environment or task.AgentGym uses AGENTEVOL to explore evolution across multiple environments.Source: §6 · pp. 9–10

7 Conclusion

AGENTGYM combines diverse interactive environments, a benchmark, and trajectory sets, while AGENTEVOL investigates self-evolution in generally capable LLM agents.

7 Conclusion

  • AGENTGYM includes an interactive platform with diverse environments and tasks, the AGENTEVAL benchmark, and the AGENTTRAJ and AGENTTRAJ-L trajectory sets.The paper also proposes AGENTEVOL to explore self-evolution across multiple environments.Source: §7 · p. 10
  • AGENTEVOL is proposed to explore self-evolution in generally capable LLM-based agents across multiple environments.Source: §7 · p. 10
  • The authors report empirical effectiveness, ablations, and analyses of how their method works.Source: §7 · p. 10

A Limitations

The authors identify computational efficiency and evaluation on larger models as limitations, and flag safety and ethical concerns as important considerations for agent self-evolution.

A Limitations

  • The study omits multiple samplings per iteration for computational efficiency, although its analysis finds more sampling improves results only modestly.The authors hope to increase K when sufficient computational resources are available.Source: A Limitations · p. 16
  • Validation covers Llama2-Chat-7B, Llama-2-Chat-13B, and DeepSeek-Coder-1.3B, with stronger and larger base models left for future study.Source: A Limitations · p. 16

B Broader Impacts

  • The paper focuses on capability self-evolution and says usage requires attention to safety and ethics, including stronger supervision and regulation to uphold human values.Source: B Broader Impacts · p. 16

C Details of Environments in AGENTGYM

AGENTGYM spans web, household, science, grid-world, word, crafting, academic, and tool-based tasks, using task-specific interactions and rewards.

C Details of Environments in AGENTGYM

  • WebShop asks agents to buy products matching instructions from over one million real products, while WebArena tests realistic web tasks across three evaluated sub-tasks.WebShop uses a 10-round limit; WebArena evaluates information-seeking, site navigation, and content and configuration operation with a 25-round limit.Source: C Details of Environments in AGENTGYM · p. 16
  • MAZE and Wordle use step-penalty rewards for reaching a goal or guessing a word, while TextCraft tests crafting recipes composed of one to four steps.Their maximum rounds are 15, 8, and 20, respectively.Source: C Details of Environments in AGENTGYM · pp. 16–17
  • ALFWorld tests household task execution, ScienceWorld elementary-level scientific reasoning, and BabyAI instruction-following in a textual grid world.Their maximum rounds are 30, 30, and 20, respectively.Source: C Details of Environments in AGENTGYM · p. 17
  • Movie, Academia, TODOList, and Weather are tool environments that reward matching reference answers or operations, with task sets expanded for Movie, TODOList, and Weather.Sheet instead scores similarity between the manipulated and reference tables on a 0–1 scale.Source: C Details of Environments in AGENTGYM · p. 18
  • BIRD wraps a text-to-SQL dataset for database querying, with 3,200 selected instructions and success rate as its evaluation metric.AGENTTRAJ adds thoughts to 2,000 training problems and AGENTTRAJ-L to 3,000; the task has a one-round limit.Source: C Details of Environments in AGENTGYM · pp. 18–19

D Platform Architecture of AGENTGYM

AGENTGYM separates environments behind HTTP services and uses EnvClients and an AgentController to connect agent interaction with platform operations.

D Platform Architecture of AGENTGYM

E More Implementation Details

The implementation uses distinct training settings for behavioral cloning, evolution, and evaluation, with additional settings for combined successful-and-failed trajectory training.

E More Implementation Details

  • The BC and evolution learning rates are both 1×10−5; BC runs for 3 epochs and evolution for 1 epoch per iteration.Evaluation uses few-shot prompting for models without expert-trajectory fine-tuning and zero-shot prompting for trained models; evaluation temperature is 0, while AGENTEvol exploration temperature is 0.7.Source: E More Implementation Details · p. 20
  • Four evolution iterations on Llama-2-Chat-7B take approximately twenty hours on eight A100-80GB GPUs, and testing takes about one hour.Source: E More Implementation Details · p. 20
  • For training with successful and failed trajectories, equal-weight DPO and BC objectives are used, with a 1 × 10−6 learning rate for greater stability.Source: E More Implementation Details · p. 20

F.1 Interactive Rounds in Main Experiments

Agents trained with AGENTTRAJ-L and AGENTEVOL complete tasks in relatively few interaction rounds; AGENTEVOL has the best performance and fewest rounds in ALFWorld and BabyAI. Case studies illustrate post-evolution behavior in WebShop and BabyAI, while the prompt appendix describes a shared template across tasks.

F.1 Interactive Rounds in Main Experiments

  • In ALFWorld and BabyAI, AGENTEVOL achieves the best performance and fewest interactive rounds; across tasks, fewer rounds generally accompany better performance.The authors suggest underperforming agents may struggle to find an optimal path or exceed the interaction limit.Source: §F.1 · p. 20, Table 7 · p. 21

F.2 Case Study

  • In WebShop, evolution changes a failed above-budget selection into multiple interactions that identify a product matching the requested color, size, and price.Before evolution, the agent fails to use environmental feedback effectively; afterward it parses returned item details.Source: §F.2 · p. 20, Figure 5 · p. 22
  • In BabyAI, evolution lets the agent locate and pick up the green box directly, whereas the earlier agent misreads spatial relations and reaches the interaction limit.The post-evolution action is “pickup green box 1.”Source: §F.2 · p. 20, Figure 6 · p. 23

G Prompt Details

  • The prompt appendix says task prompts share a template across human annotation, AI-based trajectory annotation, and evaluation, with environment-specific details listed in tables.The listed tables cover WebShop, ALFWorld, TextCraft, SciWorld, BabyAI, MAZE, Wordle, BIRD, WebArena, and other tasks.Source: G Prompt Details · pp. 20–21

Related papers

Loading 2406.04151v1…