PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models

Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, Feng Zhao

arXiv:2403.12881v1cs.CL

TL;DR

Agent-task approaches using closed-source LLMs face steep financial costs and safety issues that impede deployment. Agent-FLAN redesigns agent tuning with aligned formats, capability-based data decomposition, and negative samples; it outperforms prior works by 3.5% across a spectrum of agent evaluation benchmarks.

  • Problem

    Agent-task approaches using closed-source LLMs face steep financial costs and safety issues that impede deployment.

    Source: §1 · p. 1

  • Method

    Agent-FLAN aligns agent tuning to chat format, decomposes data by capability, and constructs negative samples to mitigate hallucinations.

    Source: §1 · p. 2

  • Results

    3.5%: Agent-FLAN surpasses prior works across a spectrum of agent evaluation benchmarks on the Llama2-series.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Negative samples are reported to mitigate format and action hallucinations in agent tasks.

    Source: §1 · p. 2, §4.3 · p. 6

    The training and validation datasets cover only part of agent tasks, leaving other interactive scenarios outside their scope.

    Source: §7 · p. 8

Abstract

from arXiv · show

Open-sourced Large Language Models (LLMs) have achieved great success in various NLP tasks, however, they are still far inferior to API-based models when acting as agents. How to integrate agent ability into general LLMs becomes a crucial and urgent problem. This paper first delivers three key observations: (1) the current agent training corpus is entangled with both formats following and agent reasoning, which significantly shifts from the distribution of its pre-training data; (2) LLMs exhibit different learning speeds on the capabilities required by agent tasks; and (3) current approaches have side-effects when improving agent abilities by introducing hallucinations. Based on the above findings, we propose Agent-FLAN to effectively Fine-tune LANguage models for Agents. Through careful decomposition and redesign of the training corpus, Agent-FLAN enables Llama2-7B to outperform prior best works by 3.5\% across various agent evaluation datasets. With comprehensively constructed negative samples, Agent-FLAN greatly alleviates the hallucination issues based on our established evaluation benchmark. Besides, it consistently improves the agent capability of LLMs when scaling model sizes while slightly enhancing the general capability of LLMs. The code will be available at https://github.com/InternLM/Agent-FLAN.

1. Introduction

Agent-FLAN redesigns agent tuning for general LLMs around corpus alignment, capability decomposition, and negative samples. On Llama2-series models, it outperforms prior works across agent benchmarks.

1. Introduction

  • Open-sourced LLMs lag API-based models on agent tasks, while specialized fine-tuning risks compromising general capabilities.Closed-sourced models also bring steep financial costs and safety issues that impede deployment.Source: §1 · p. 1
  • The authors identify entangled format-following and reasoning data, uneven learning speeds across capabilities, and overlooked hallucinations as agent-tuning challenges.These issues motivate decomposing and redesigning agent-training data.Source: §1 · p. 1
  • Agent-FLAN aligns tuning to chat format, decomposes agent tasks by capability, and uses negative samples to mitigate hallucinations.The approach also establishes Agent-H to assess hallucination issues from various aspects.Source: §1 · p. 2
  • 3.5%: Agent-FLAN surpasses prior works across a spectrum of agent evaluation benchmarks on the Llama2-series.The benchmarks include general agent tasks and tool utilization.Source: §1 · p. 2

2. Related Work

Related work covers language agents and language-model fine-tuning, while positioning Agent-FLAN as an approach for integrating agent capabilities into general LLMs.

2.1. LLM as Agent

  • Prior language-agent studies span task-specific applications, action-focused methods such as ReAct, social properties, and multi-agent collaboration.Agent-FLAN instead targets integration of agent capabilities into general LLMs for complex real-world problems.Source: §2.1 · pp. 2–3

2.2. Language Model Fine-Tuning

  • Fine-tuning research addresses model capabilities, methods, data selection, and datasets across diverse task areas.Examples include reasoning, tool proficiency, planning, and retrieval-augmented fine-tuning.Source: §2.2 · p. 3

3. Pilot Observations

The pilot observations identify format entanglement, uneven learning speeds across capabilities, and hallucinations as issues in agent tuning.

Observation 1

  • Formatted agent data shifts tuning away from natural conversation; its format loss falls faster while content loss remains high (0.54 vs 0.04).The authors attribute this pattern probably to overfitting to fixed ReAct or JSON structures.Source: §3 · p. 3

Observation 2

  • Instruction following is the simplest capability to learn, retrieval and understanding are more manageable than reasoning, and their loss curves differ.The authors use this variation in learning speeds to motivate capability-based data decomposition and balancing.Source: §3 · p. 3, Figure 3 · p. 3

Observation 3

  • Agent hallucinations include rigidly following a training format despite the user query and invoking nonexistent functions when prompted.The authors argue that existing approaches have limited impact on these issues and that appropriate benchmarks are needed to assess and mitigate them.Source: §3 · pp. 3–4, Figure 4 · p. 4

4. Agent-FLAN

Agent-FLAN redesigns agent tuning through chat-domain alignment, capability-wise data balancing, and negative-sample learning. Its experiments evaluate Llama2-series models on held-in and held-out agent tasks.

4. Agent-FLAN

  • Experiments tune Llama2-series models, use Llama2-7B for ablations, and cover seven held-in sources and four held-out interactive tasks.Source: §4 · p. 4
  • Agent-FLAN significantly outperforms previous agent-tuning approaches on both held-in and held-out tasks.Source: Table 1 · p. 5

4.1. Aligning Agent Tuning to Pretrain Domain

  • 3.1% improvement on T-Eval and 2.5% on HotpotQA followed alignment of agent training data to the chat domain.The method converts formatted agent data into multi-turn dialogue and decomposes JSON arguments with eliciting statements, while adding instruction-following pairs for requested ReAct and JSON outputs.Source: §4.1 · pp. 4–5

4.2. Capabilities Decomposition and Data Balancing

  • Reducing reasoning data by half lowered performance 1.1 points, and reducing understanding data by half lowered it 0.3 points; retrieval and instruction-following reductions had little influence or improved results.The capability mixtures are reasoning, retrieval, understanding, and instruction following; the ablation removes half of a capability subset from an equally weighted mixture.Source: §4.2 · pp. 5–6, Table 2 · p. 5

4.3. Negative Sample Learning for Hallucination Elimination

  • Agent-H measures format-level instruction adherence and action-level behavior across four circumstances, corresponding to the paper’s two agent-hallucination categories.Source: §4.3 · p. 6
  • Negative samples cover tool-request/no-tool and normal-response/tool-provided mismatches, mitigating hallucinations while maintaining high T-Eval performance.Source: §4.3 · p. 6

5. Analysis

The analysis examines how agent-tuning performance changes with data and model scale, and how agent training relates to general capabilities. It reports gains from Agent-FLAN over AgentTuning in selected comparisons.

5.1.1. DATA SCALING LAW FOR AGENT TUNING

  • With only 25% of Agent-FLAN training samples, agent ability gains the most; improvements continue more slowly as data increases to 50% and 75%.The authors suggest that enriching data diversity or quality may be more useful than simply enlarging the corpus.Source: §5.1.1 · p. 7, Figure 7 · p. 7

5.1.2. MODEL SCALING LAW FOR AGENT TUNING

  • As Llama2 model size increases from 7B to 70B, performance continuously improves without saturation, while specific agent tuning steadily improves over vanilla ReAct tuning.The authors infer that larger models already possess basic reasoning and retrieval abilities, making appropriate agent tuning more important for eliciting agent capability.Source: §5.1.2 · p. 7, Figure 8 · p. 7

5.2. General Ability v.s Agent Ability

  • Agent training corpus improves general capabilities as well as agent-task ability, according to evaluations on MMLU, GSM8K, and HumanEval.The authors speculate that shared reasoning and instruction-following skills may account for the general-capability gains.Source: §5.2 · p. 7
  • Figure 9 contrasts Agent-FLAN with AgentTuning: on ToolBench, Agent-FLAN handles specific API information in long tool descriptions, while on Agent-H it gives the preferred response instead of meaningless tool use.The figure attributes the ToolBench contrast to capability decomposition and greater focus on “understand.”Source: Figure 9 · p. 8

6. Conclusion

The conclusion presents Agent-FLAN as a corpus-decomposition and redesign approach for agent tuning. It reports that Llama2-7B outperforms previous works across a wide spectrum of agent tasks.

6. Conclusion

  • Agent-FLAN’s decomposition and redesign of the training corpus enables Llama2-7B to outperform previous works across a wide spectrum of agent tasks.Source: §6 · p. 8

7. Limitations

The paper identifies limits in task coverage and in the portion of ToolBench used for training.

7. Limitations

  • Training and validation cover only part of agent tasks, leaving other interactive scenarios for future evaluation.Source: §7 · p. 8
  • The authors select roughly 20,000 valid ToolBench samples, 10% of the dataset, and leave fuller use for future work.Source: §7 · p. 8

8. Ethical Considerations

The paper reports using public reference materials and human review of generated data, with privacy safeguards.

8. Ethical Considerations

  • Experiments use publicly available reference documents and APIs to circumvent possible harm toward individuals or groups.Source: §8 · p. 8
  • Humans carefully select and process LLM-generated data to secure privacy and confidentiality.Source: §8 · p. 8
  • The study reports no personal identification information and says data were anonymized before analysis.Source: §8 · p. 8

A. Experimental Details

Agent-FLAN experiments mix ShareGPT and agent data, with a small ReAct-format portion, and fine-tune for one epoch. The setup also specifies task-balancing weights and additional instruction-following data.

A. Experimental Details

  • A 1:1 mix of ShareGPT and agent corpus is used, with 10% ReAct-format data and 90% Agent-FLAN-format data.The authors report that a small amount of ReAct-format data empirically improves performance.Source: A. Experimental Details · p. 12
  • Models are fine-tuned for 1 epoch, with reasoning, retrieval, and understanding weighted 1:0.25:0.75.Hyper-parameters for different model scales are listed in Table 5.Source: A. Experimental Details · p. 12, Table 5 · p. 12
  • The training setup additionally includes 2000 curated instruction-following samples.Source: A. Experimental Details · p. 12

B. Dataset Details

Agent-FLAN builds its dataset from AgentInstruct and ToolBench, filtering samples according to format and content. Its default mix uses 10% ReAct-format data and 90% conversation-format data.

B. Dataset Details

  • The dataset uses AgentInstruct, which comprises six agent tasks, and ToolBench as its basic datasets.AgentInstruct few-shot examples are removed, and data not following Thought-Action-ActionInput format is filtered out.Source: B. Dataset Details · p. 12
  • ToolBench samples are kept only if they contain FINALANSWER, while samples beginning with the specified DFSDT phrase are removed.Source: B. Dataset Details · p. 12
  • The default dataset mix is 10% ReAct-format and 90% conversation-format data.Table 6 presents the dataset statistics used in Agent-FLAN.Source: B. Dataset Details · p. 12, Table 6 · p. 12

C. Negative Samples Construction Details

The negative-sample construction starts with ToolBench queries answered without tool information, then creates tool-informed conversations by appending irrelevant tools to half of the queries.

C. Negative Samples Construction Details

Loading 2403.12881v1…