Source-linked AI summary
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, Le Sun
TL;DR
Compact language models’ ability to generalize tool use remained uncertain because existing methods either relied on very large models or covered limited tool scopes. ToolAlpaca automatically generates diverse tool-use data with multi-agent simulation and fine-tunes compact models, which perform comparably to GPT-3.5 in generalized tool-use scenarios.
Problem
It remained uncertain whether compact language models could learn generalized tool-use abilities without tool-specific training, despite large models demonstrating such abilities.
Method
ToolAlpaca automatically constructs a diverse toolset, simulates multi-turn interactions among user, assistant, and tool-executor agents, and fine-tunes compact language models on the resulting corpus.
Results
ToolAlpaca performs comparably to GPT-3.5 in generalized tool-use scenarios when evaluated on previously unseen tools.
Takeaways & Limitations
The results support the feasibility of equipping compact language models with generalized tool-use capabilities using automatically generated training data.
Takeaways & Limitations
Preliminary experiments found that ChatGPT occasionally failed to follow prescribed output formats during corpus construction.
Abstract
from arXiv · showhide
Enabling large language models to utilize real-world tools effectively is crucial for achieving embodied intelligence. Existing approaches to tool learning have either primarily relied on extremely large language models, such as GPT-4, to attain generalized tool-use abilities in a zero-shot manner, or utilized supervised learning to train limited scopes of tools on compact models. However, it remains uncertain whether smaller language models can achieve generalized tool-use abilities without tool-specific training. To address this question, this paper introduces ToolAlpaca, a novel framework designed to automatically generate a diverse tool-use corpus and learn generalized tool-use abilities on compact language models with minimal human intervention. Specifically, ToolAlpaca first automatically creates a highly diversified tool-use corpus by building a multi-agent simulation environment. The corpus contains 3938 tool-use instances from more than 400 real-world tool APIs spanning 50 distinct categories. Subsequently, the constructed corpus is employed to fine-tune compact language models, resulting in two models, namely ToolAlpaca-7B and ToolAlpaca-13B, respectively. Finally, we evaluate the ability of these models to utilize previously unseen tools without specific training. Experimental results demonstrate that ToolAlpaca achieves effective generalized tool-use capabilities comparable to those of extremely large language models like GPT-3.5, demonstrating that learning generalized tool-use ability is feasible for compact language models.
1 Introduction
ToolAlpaca addresses whether compact language models can acquire generalized tool-use abilities by automatically generating diverse simulated tool-use data and fine-tuning on it. The resulting models perform competitively with GPT-3.5 on unseen tools.
- Motivation: Compact language models lacked generalized tool-use abilities, while larger models such as GPT-4 had demonstrated effective tool use through integrated plugins.This gap motivated testing whether compact models could learn generalized tool use without tool-specific training.
- Motivation: Constructing a diversified tool-use corpus efficiently is difficult because manual annotation must cover many tools and multi-round real-world interactions.The paper identifies both tool diversity and iterative trial-and-error usage as barriers.
- Approach: ToolAlpaca automatically builds a structured toolset and generates multi-turn tool-use instances through multi-agent simulation with minimal manual intervention.The framework uses LLMs to construct tool documentation and simulate interactions among users, assistants, and tool executors.
- Corpus: 3938 tool-use instances cover more than 400 distinct tools, forming the corpus used to train compact models.The contribution summary characterizes the corpus as spanning 50 distinct categories.
- Results: ToolAlpaca effectively equips models to use numerous unseen tools and achieves competitive performance with GPT-3.5.The evaluation includes real-world APIs and multi-modal tools, with performance assessed through machine evaluation by GPT-4.
2 Related Work
Prior tool-learning research either prompts large language models to interact with tools or trains models for narrower tool scopes. ToolAlpaca targets the less explored problem of generalized tool use in compact models through automatically generated data.
- Tool Use: Existing approaches commonly use large language models to interact with specialized tools such as code interpreters, search engines, retrieval models, and AI models.This line of work relies primarily on the capabilities of large models rather than compact-model training.
- Tool Use: ToolLLM supports mastering massive APIs, but its data collection requires accumulating, manually obtaining, and verifying authentic APIs.The cited limitation concerns the labor required before training data can be assembled.
- LLMs for Data Generation: LLM-synthesized data has improved specific tool-use capabilities, whereas generalized tool-use data generation remains more challenging.The paper positions its work at the intersection of model-generated data and generalized tool learning.
Multi-agent Simulation
ToolAlpaca constructs a standardized toolset and uses three LLM-based agents to simulate multi-turn tool interactions. These simulations produce structured instances containing instructions, actions, and final responses.
- Toolset Construction: Toolset construction represents each tool as {name, introduction, description, function documentation, OpenAPI specification}.LLMs enrich brief internet-sourced introductions with structured documentation.
- Multi-agent Simulation: The simulation environment uses three virtual LLM agents: a user, an assistant, and a tool executor.Their interaction is designed to reflect real-world tool utilization scenarios.
- Toolset Construction: The tool collection begins with public-apis, which contains over 1400 APIs spanning more than 50 categories.The repository supplies the initial pool from which the framework constructs its toolset.
- Toolset Construction: LLMs generate tool descriptions, function documentation, and formal OpenAPI specifications to make heterogeneous APIs more uniform and readable.OpenAPI specifies endpoints, expected input/output types, and possible error responses.
- Multi-agent Simulation: Each generated instance contains an instruction, ordered actions with thoughts, function names, inputs, and tool responses, followed by a conclusive response.The assistant repeatedly selects actions and collects tool responses until it has sufficient information.
- Multi-agent Simulation: The user agent drafts diverse instructions and supplies missing information during interaction, while the assistant agent interprets requests and selects actions.Multiple prompts are used to vary instruction formats, and generation proceeds through multiple interaction rounds.
4 ToolAlpaca Corpus
ToolAlpaca constructs a diverse corpus through automated multi-agent simulation and filtering, producing 3,938 instances across 426 tools and 50 categories. Human review indicates that the generated data has decent quality.
- Construction Details: ChatGPT generates documentation and tool-use cases from randomly selected APIs, while filtering removes nontextual tools, overly long interactions, missing calls, and parsing errors.The filtering process excludes tools with nontextual inputs or outputs and instances exceeding five interaction steps.
- Construction Details: 3938 instances cover 426 distinctive tools from 50 categories in the final ToolAlpaca corpus.The corpus is automatically constructed after filtering the generated cases.
- Construction Details: Preliminary experiments found that ChatGPT occasionally failed to follow prescribed output formats.This motivated quality-control considerations during corpus construction.
- Corpus Diversity: The toolset spans common and specialized categories, averages five functions per tool, and includes function inputs ranging from simple values to arrays and objects.These properties provide diversity in categories, capabilities, and input complexity.
- Quality: The three simulated-agent quality metrics all exceed 80% in a human review of 100 sampled instances.The review assesses instruction solvability, tool-executor output precision, and assistant action and response accuracy.
5 Experiment
Experiments evaluate compact models on unseen simulated, real-world, and multimodal tools, while testing how toolset diversity affects generalization. ToolAlpaca improves performance over Vicuna and approaches GPT-3.5 or GPT4Tools in reported evaluations.
- Evaluation Setup: Zero-shot evaluation covers unseen simulated tools, real-world APIs, and out-of-dataset multimodal tools, using GPT-4 and additional manual assessment for simulated tools.The evaluation dataset contains 10 simulated tools and 11 real-world APIs; GPT-4 scores procedure, response, and overall performance.
- Simulated Tools: 73 (+57) and 75 (+50) human accept rates are achieved by ToolAlpaca-7B and ToolAlpaca-13B, versus 16 and 25 for the corresponding Vicuna models.ToolAlpaca-13B achieves performance comparable to GPT-3.5 on the simulated set.
- Real-World Tools: 55.3 and 61.4 overall accuracy are achieved on real-world APIs by ToolAlpaca-7B and ToolAlpaca-13B, respectively, surpassing Vicuna models.The result suggests that simulated-data training adapts to real-world tool-use scenarios.
- Out-of-Dataset Tools: 83.7 success rate is achieved on the GPT4Tools out-of-dataset evaluation after training on 3.9k cases, close to GPT4Tools trained on 71k cases.The evaluation uses eight multimodal tools.
- Impact of Diversity: 70 overall accuracy with 400 tools exceeds the 51 achieved with 10 tools when the number of training instances is held constant.Validation performance gradually improves as toolset diversity increases.
6 Conclusion
ToolAlpaca is an automated framework that builds a broad tool-use corpus and uses it to fine-tune compact language models. Its results indicate performance comparable to GPT-3.5 in generalized tool-use scenarios, supporting the feasibility of generalized tool use in compact models.
- ToolAlpaca automatically creates a comprehensive corpus spanning diverse tools and usage instances.The corpus is used as the basis for fine-tuning compact language models.
- The corpus serves as training data for fine-tuning compact language models into ToolAlpaca models.
- ToolAlpaca performs comparably to GPT-3.5 in generalized tool-use scenarios.The finding highlights the feasibility of mastering generalized tool use in compact-size models.
A Implementation Details
The implementation details section documents the prompt templates used throughout ToolAlpaca’s toolset construction and multi-agent simulation process. It identifies prompts for user, assistant, and tool executor agents.
- Prompt templates: Figures 6–8 present the prompt templates used for toolset construction.
- Prompt templates: Figures 9–10 show prompts for the user agent’s responsibilities: generating instructions and providing missing information.
- Prompt templates: Figures 11–12 detail prompts for the assistant agent and tool executor agent, with ${...} marking placeholders.
B Experiment Details
The experiment details record the fine-tuning configuration for ToolAlpaca in Table 5. The supplied table caption identifies the table’s purpose but does not provide its parameter values.
- The fine-tuning configuration for ToolAlpaca is recorded in Table 5.
- The supplied experiment details identify the configuration table but do not state individual configuration settings.
- Table 5 is specifically designated as the fine-tuning configuration table for ToolAlpaca.
B.2 Evaluation Dataset Details
The evaluation dataset contains simulated and real-world tool subsets, and GPT-4 is used as the evaluator following Vicuna’s evaluation method. The evaluation toolset is documented separately in Table 6.
- Evaluation subsets: The simulated evaluation subset contains 10 simulated tools and 100 instances.
- Evaluation subsets: The real-world evaluation subset contains 11 real-world APIs and 114 instances.
- Evaluation subsets: Table 6 details the toolset used in the evaluation dataset.
- Evaluation procedure: GPT-4 evaluates generalized tool-use ability following Vicuna’s evaluation method.The evaluation prompt is shown in Figure 13.
B.4 Case Study
The case study illustrates how ToolAlpaca’s simulated tool-use training supports interactions with real-world APIs. It presents the prompts, evaluation setup, and example cases used to demonstrate this capability.
- Real-world API cases: ToolAlpaca equips models to use real-world APIs, with examples involving Nager.Date, Currency Beacon, and weatherstack.The examples cover public holidays, currency exchange rates, and weather conditions for a road trip.
- Prompt pipeline: The case-study pipeline generates API descriptions, function documentation, OpenAPI specifications, user instructions, and multi-agent responses.Separate prompts support description generation, function documentation, user-agent instruction generation, missing-information requests, assistant actions, API execution, and GPT-4 evaluation.
- Evaluation cases: The evaluation examples include tool requests that supply parameters such as country codes, years, authentication information, and access keys.One example requests Australian public holidays for 2023, while other cases request currency rates and weather information using authentication details.
- Figures: Figure 14, Figure 15, and Figure 16 present selected cases for the Nager.Date, Currency Beacon, and apilayer weatherstack APIs.These figures correspond to the real-world API examples discussed in the case study.