PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories

Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, Sujian Li

arXiv:2410.07706v1cs.CLcs.AI

TL;DR

Open-source LLMs lag proprietary models on agent tasks, and prior trajectory datasets do not adequately test generalized agent capability. The paper introduces AgentBank, a large multi-task trajectory dataset and annotation pipeline, then fine-tunes Llama-2 into Samoyed. Samoyed outperforms GPT-3.5-Turbo on average on held-in tasks and performs better on held-out tasks, while the study finds that large-scale trajectory tuning supports generalized agent capabilities.

  • Problem

    Open-source LLMs lag proprietary models on agent capabilities, while prior trajectory-tuning datasets provide limited evidence about generalized capability across tasks.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    The paper constructs AgentBank with over 50,000 trajectories across 16 tasks and five skill dimensions, then fine-tunes Llama-2 to produce Samoyed.

    Source: §1 · p. 1, §8 · p. 9

  • Results

    SAMOYED outperforms GPT-3.5-Turbo on average on held-in tasks and also demonstrates superior performance on held-out tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Large-scale multi-task trajectory tuning can acquire generalized agent capabilities in open-source LLMs across held-in and unseen tasks.

    Source: §1 · p. 2, §8 · p. 9

    Experiments cover only 7B and 13B models, so the benefits of large-scale trajectory tuning for larger models remain unknown.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Fine-tuning on agent-environment interaction trajectory data holds significant promise for surfacing generalized agent capabilities in open-source large language models (LLMs). In this work, we introduce AgentBank, by far the largest trajectory tuning data collection featuring more than 50k diverse high-quality interaction trajectories which comprises 16 tasks covering five distinct agent skill dimensions. Leveraging a novel annotation pipeline, we are able to scale the annotated trajectories and generate a trajectory dataset with minimized difficulty bias. Furthermore, we fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. Our comparative experiments demonstrate the effectiveness of scaling the interaction trajectory data to acquire generalized agent capabilities. Additional studies also reveal some key observations regarding trajectory tuning and agent skill generalization.

1 Introduction

Open-source LLMs lag proprietary models on agent capabilities, while prior trajectory-tuning studies use limited, task-specific data. AgentBank scales diverse trajectories and yields Samoyed models that generalize across held-in and unseen tasks.

  • Open-source LLMs fall short of GPT-3.5 and GPT-4 on agent capabilities despite strong proprietary-model instruction following, reasoning, and planning.Source: §1 · p. 1
  • Prior trajectory-tuning work improved specialized abilities, but AgentTuning used only six tasks and 1.8k trajectories, limiting generalized capability gains in 7B and 13B models.Source: §1 · p. 1
  • AgentBank contains over 50,000 annotated trajectories spanning 16 tasks and five agent skill dimensions, using a scalable pipeline designed to reduce difficulty bias.Source: §1 · pp. 1–2
  • Samoyed models are produced by trajectory-tuning Llama-2 with AgentBank and evaluated on both held-in and unseen held-out tasks.Source: §1 · p. 1
  • SAMOYED outperforms GPT-3.5-Turbo on average on held-in tasks and also performs better on held-out tasks, supporting generalized agent capability acquisition.Source: §1 · p. 2
  • Analyses examine trajectory scaling, hybrid instruction-and-code training, catastrophic forgetting, and the role of CoT rationale in generalized capability acquisition.Source: §1 · p. 2

2 Related Work

Instruction tuning aligns LLMs and has been applied to specialized abilities, while LLM-based agent work adapts proprietary-model trajectories to open-source models. Limited tasks and expert trajectories leave generalized agent ability insufficiently explored.

2.1 Instruction Tuning

  • Instruction tuning aligns LLMs with human preferences and has been scaled to support generalized instruction following and specialized math, reasoning, and agent abilities.Source: §2.1 · p. 2

2.2 LLM-based Agent

  • ReAct combines chain-of-thought reasoning with actions, while Auto-GPT uses LLMs as controllers for complex agent frameworks.Source: §2.2 · p. 2
  • Open-source models lag GPT-3.5 and GPT-4 on agent tasks.Source: §2.2 · p. 2
  • FireAct, AgentTuning, and AgentOhana collect teacher-agent trajectories to fine-tune open-source LLMs, but limited tasks and expert trajectories leave generalized abilities insufficiently explored.Source: §2.2 · p. 3

3 Preliminary

An agent repeatedly maps instructions and environment observations to actions until task completion or a step limit, receiving a final reward. Existing GPT-exploration pipelines are difficult to scale and introduce difficulty bias.

3.1 Agent Task Formulation

  • An interaction trajectory records an instruction, sequential agent actions, environment observations, and a final reward based on task completion.Source: §3.1 · p. 3
  • The ReAct framework outputs chain-of-thought rationale before each agent action, using step-by-step reasoning to support inferential capability.Source: §3.1 · p. 3

3.2 Challenges in Trajectory Collection

  • GPT-based trajectory collection filters out failed interactions to preserve training quality, but this automation has substantial costs.Source: §3.2 · p. 3
  • AgentInstruct discards more than 90% of generated trajectories because of GPT failures, making large-scale collection difficult.Source: §3.2 · p. 3
  • Filtering failed trajectories skews training data toward easier instances than the test set, violating the i.i.d. assumption and potentially harming generalization.Source: §3.2 · p. 3

4 AGENTBANK

AGENTBANK addresses trajectory-collection scalability and difficulty bias with a broad multi-skill dataset and task-specific annotation pipeline. It combines curated tasks, specialized action construction, and LLM-generated rationales.

4 AGENTBANK

  • AGENTBANK is constructed as a new trajectory annotation pipeline intended to address prior collection challenges.Source: §4 · p. 3
  • AGENTBANK compiles 16 tasks across five skill dimensions, including reasoning, math, programming, web navigation, and embodied tasks.Source: Table 2 · p. 4, §4.1 · p. 4

4.1 Task and Instruction Collection

  • The dataset includes tool-use tasks, continuous and discrete action spaces, interaction lengths from 1 to 30 turns, and reformulated single-turn benchmarks for multi-turn tool environments.Source: §4.1 · p. 4

4.2 Action Annotation

  • The pipeline separates gold-action annotation from rationale annotation and uses task-specific techniques to obtain high-quality action sequences.Source: §4.2 · p. 4
  • Answer forcing reannotates failed continuous-action trajectories using gold final answers and validates the resulting actions in real environments.Source: §4.2 · p. 4
  • For discrete actions, heuristic depth-first search obtains optimal sequences; existing solving trajectories can also be reformatted into agent interaction actions.Source: §4.2 · pp. 4–5

4.3 Rationale Annotation

  • GPT generates chain-of-thought rationales for gold actions, while high-success tasks can use the GPT-exploration pipeline directly and dataset quality is assessed through human evaluation.Source: §4.3 · p. 5

5 Train SAMOYED with AGENTBANK

SAMOYED is trained by formatting AGENTBANK trajectories as chatbot-style interactions and applying loss only to ground-truth agent responses. The training mixture also includes general instruction and code data.

  • AgentBank trajectories are formatted as chatbot-style sequences containing a task instruction, environment observations, and agent actions with rationales.Source: §5 · p. 5
  • Training feeds complete trajectories to a decoder-only LLM while masking instructions and observations from the loss computation.Only autoregressive loss on ground-truth responses is counted.Source: §5 · p. 5
  • The trajectory-tuning loss is defined over input tokens but counts only tokens belonging to the ground-truth response set Y.The indicator function selects which tokens contribute to the loss.Source: §5 · p. 5
  • Fine-tuning mixes AGENTBANK agent data with general-domain instruction data and code data to investigate hybrid training effects.The authors examine the effectiveness of the generalist and code components through ablation experiments.Source: §5 · p. 5

6 Experiments

Experiments compare base and tuned LLMs on held-in and unseen held-out tasks, using grouped skill-dimension averages and overall task scores. Large-scale trajectory tuning improves all evaluated base models, with especially strong gains for SAMOYED on unseen tasks.

6.1 Experimental Setup

  • Experiments primarily use 7B and 13B models, with larger-scale comparisons left for future work.Llama-2-Chat is the main base model for fair comparison because many baselines are tuned from it.Source: §6.1 · p. 5
  • The evaluation uses nine held-in tasks and five unseen held-out tasks spanning the five skill dimensions, with average scores measuring overall capability.An additional AgentBench evaluation is reported in Appendix G.Source: §6.1 · p. 6

6.2 Main Results

  • SAMOYED shows remarkable improvement on held-out unseen tasks and outperforms AgentLM and Agent-FLAN, which use fewer trajectories.SAMOYED-7B exhibits a greater enhancement than SAMOYED-13B on these tasks.Source: §6.2 · p. 6
  • CodeLlama excels in web browsing, while Vicuna performs strongly on held-in and held-out tasks after generalist instruction fine-tuning.Source: §6.2 · p. 6
  • AgentLM-13B is surpassed by Vicuna-13B, while AgentLM’s 80% generalist-instruction training may contribute to its held-out performance through instruction following.Source: §6.2 · p. 7
  • After large-scale trajectory tuning, all evaluated base LLMs show significant performance improvements on held-in and held-out tasks.Figure 2 compares untrained base models with models trained on AGENTBANK.Source: §6.2 · pp. 6–7, Figure 2 · p. 7
  • Trajectory tuning benefits weaker LLMs more strongly: Llama-2 shows substantial gains, whereas Mistral and Llama-3 improve more modestly.The comparison also identifies code training as a factor associated with CodeLlama’s stronger agent performance.Source: §6.2 · p. 7

7 Further Analysis

Scaling tasks and trajectories improves held-out generalization, while data mixtures, CoT rationales, and cross-skill training reveal important conditions on agent capability transfer.

7.1 Scaling Trends of Generalization

  • Increasing the number of training tasks improves performance on held-out tasks, indicating stronger generalized agent capabilities.Source: §7.1 · p. 7
  • 50k+ trajectories substantially outperform 1k trajectories for generalized ability, although gains gradually plateau for the 13B model.Source: §7.1 · p. 7

7.2 The Effect of Data Mixture

  • A relatively low proportion of generalist or code data improves unseen-task generalization, whereas increasing generalist data or code-data ratios can reduce it.Source: §7.2 · pp. 7–8
  • CodeLlama performs strongly on programming and web-navigation tasks, while interaction trajectories further improve its coding proficiency.Source: §7.2 · p. 8
  • Mixture training mitigates catastrophic forgetting on general benchmarks by reducing overfitting to the specific ReAct style of agent trajectories.Source: §7.2 · p. 8

7.3 The Effect of CoT Rationale

  • Removing CoT rationales has little effect on held-in tasks but significantly decreases held-out performance, consistent with rationales supporting learned planning processes.Source: §7.3 · p. 8

7.4 Skill-Level Transfer

8 Conclusion

The paper introduces AGENTBANK and SAMOYED to study generalized agent capabilities from large-scale interaction trajectories. Across held-in and held-out tasks, SAMOYED outperforms strong baselines, while analysis highlights data-mixture effects and trajectory scaling.

  • AGENTBANK contains over 50k trajectories spanning 16 tasks and five agent skill dimensions, enabling large-scale trajectory tuning.Source: §8 · p. 9
  • SAMOYED models fine-tuned from Llama-2 significantly outperform strong baselines on held-in and held-out tasks.Source: §8 · p. 9
  • The analysis identifies data mixture and trajectory scaling as important factors in generalized agent capability acquisition.Source: §8 · p. 9

Limitations

The study is limited to 7B and 13B models, leaves advanced agent mechanisms and exploration-based optimization largely unexplored, and focuses on single-agent ReAct systems.

  • Experiments and analysis cover only 7B and 13B models, leaving larger models' benefits from trajectory tuning unknown.Source: Limitations · p. 9
  • The potential of sophisticated mechanisms such as Reflexion and ReWOO remains insufficiently explored.Source: Limitations · p. 9
  • The work relies on supervised fine-tuning over expert trajectories, leaving exploration-based optimization for future investigation.Source: Limitations · p. 9
  • The study focuses on strong ReAct-style single-agent models, while generalized multi-agent systems remain under-explored.Source: Limitations · p. 9

A Details of Tasks in AGENTBANK

AGENTBANK combines diverse task environments, alternative trajectory-construction methods, and rationale annotation to scale agent training while limiting difficulty bias. Its evaluations indicate that the resulting data retain comparable quality and support generalization beyond held-in tasks.

A Details of Tasks in AGENTBANK

  • AGENTBANK spans reasoning, math, programming, web, and embodied tasks, using tools and 16 datasets to represent varied agent capabilities.The collection includes held-in and held-out environments, with MiniWoB++, ScienceWorld, Bamboogle, and TheoremQA among the held-out tasks.Source: A Details of Tasks in AGENTBANK · p. 13
  • Its annotation pipeline combines exploration, reformatting, answer forcing, and heuristic search to construct or enrich trajectories across task types.GPT-based rationale annotation is applied to reformatted, human, exploratory, and algorithmically generated trajectories, while failed programming cases are re-annotated.Source: A Details of Tasks in AGENTBANK · p. 13

B Difficulty Bias in Trajectory Collection

  • AGENTBANK reduces difficulty bias: its pseudo-test and original-test performance are close, while AgentInstruct performs worse on the original test set than on its pseudo-test set.The comparison uses training, pseudo-test, and original-test WebShop splits to assess whether annotation data match the intended evaluation distribution.Source: B Difficulty Bias in Trajectory… · p. 14, Table 8 · p. 14

C CoT Rationales Generated by Different LLMs

E Data Contamination

The paper examines contamination in SAMOYED’s mixed training data, reports complete held-in and AgentBench evaluations, and documents the prompts used to construct trajectory annotations. It finds overlap mainly in held-in data-construction patterns, while held-out evaluations are described as uncontaminated.

E Data Contamination

  • Contamination analysis matches 9-grams and 13-grams between evaluation instances and AGENTBANK, ShareGPT, and Evol-CodeAlpaca.The analysis is designed to detect overlap that could overestimate performance in the mixed-data training setup.Source: E Data Contamination · p. 15, Table 11 · p. 15
  • Held-in contamination is high in some tasks, largely reflecting yes/no formats, templated instructions, and common Python imports or code snippets.Manual inspection attributes overlap in StrategyQA, WebShop, ALFWorld, and MBPP to these construction patterns.Source: E Data Contamination · p. 15
  • The authors conclude contamination minimally affects results because original train-test splits are preserved and held-out tasks do not show contamination.The held-out tasks are used to assess generalized capabilities, and the paper presents their evaluations as trustworthy and robust.Source: E Data Contamination · pp. 15–16

H Prompts for Trajectory Annotation

  • The appendix supplies prompts for trajectory reformatting, rationale generation, answer forcing, and step-by-step interaction with interleaved Thought, Action, and Observation fields.The prompts also instruct answer forcing to avoid repeating failures and to generate reasoning for every action.Source: H Prompts for Trajectory Annotation · pp. 17–18

Related papers

Loading 2410.07706v1…