Source-linked AI summary
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, Maosong Sun
TL;DR
Open-source LLMs have limited API tool-use ability because instruction tuning largely neglects this domain. ToolLLM constructs ToolBench, adds DFSDT, ToolEval, and API retrieval, and fine-tunes ToolLLaMA; ToolEval agrees with human pass-rate judgments at 87.1% and win-rate judgments at 80.3%, while ToolLLaMA performs nearly on par with ChatGPT.
Problem
Open-source LLMs remain limited in using APIs for complex instructions, while instruction tuning largely focuses on basic language tasks.
Method
ToolLLM combines ChatGPT-based construction of ToolBench, DFSDT solution-path search, ToolEval, LLaMA fine-tuning, and neural API retrieval.
Results
ToolEval agrees with human annotators at 87.1% for pass rate and 80.3% for win rate, while ToolLLaMA performs almost on par with ChatGPT.
Takeaways & Limitations
ToolLLaMA handles single-tool and complex multi-tool instructions and generalizes to unseen APIs using API documentation.
Takeaways & Limitations
Main experiments simulate users specifying a preferred API set by providing each model with N APIs.
Abstract
from arXiv · showhide
Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.
1 INTRODUCTION
ToolLLM addresses the limited tool-use ability of open-source LLMs with a framework covering dataset construction, model training, retrieval, reasoning, and evaluation. ToolBench supports diverse single- and multi-tool tasks, while ToolLLaMA combines fine-tuning with API retrieval and achieves performance nearly comparable to ChatGPT.
- Open-source LLMs remain limited in using APIs for complex instructions because instruction tuning largely emphasizes basic language tasks.
- ToolLLM introduces ToolBench, a ChatGPT-constructed instruction-tuning dataset, alongside model training and automatic evaluation components.The framework includes data construction, training, and evaluation.
- ToolBench collects 16,464 REST APIs spanning 49 categories and generates instructions covering both single-tool and multi-tool scenarios.The collected APIs include documentation such as functionality descriptions, parameters, and API-call code snippets.
- DFSDT uses a depth-first search-based decision tree to evaluate multiple reasoning traces and expand the search space for solution-path annotation.The approach improves annotation efficiency and completes complex instructions that ReACT cannot fulfill.
- ToolEval automatically evaluates tool-use reasoning with pass rate and win rate, providing a scalable assessment backed by ChatGPT.Pass rate measures successful execution within limited budgets, while win rate compares the quality and usefulness of two solution paths.
- ToolLLaMA is fine-tuned on ToolBench and paired with a neural API retriever that recommends relevant APIs for multi-round tool use.Experiments report performance surpassing Text-Davinci-003 and Claude-2 while approaching ChatGPT.
2 DATASET CONSTRUCTION
ToolBench is constructed through API collection, instruction generation, and solution-path annotation, using RapidAPI metadata and ChatGPT-based procedures. The dataset emphasizes diverse single-tool and multi-tool instructions, while DFSDT broadens solution-path search beyond one reasoning direction.
- Dataset construction: ToolBench uses a three-stage construction process: API collection, instruction generation, and solution-path annotation.All procedures are based on ChatGPT and require minimal human supervision.
- API collection: 16,464 REST APIs spanning 49 categories are retained after filtering 10,853 tools and 53,190 initially collected APIs.The final set contains 3,451 high-quality tools.
- Instruction generation: Instructions target both single-tool and multi-tool scenarios to cover individual API use and combinations for complex tasks.The process samples APIs and prompts ChatGPT to generate instructions and relevant API pairs.
- Instruction generation: For multi-tool instructions, sampling 2-5 tools from the same category or collection addresses sparse interconnections among unrelated tools.At most three APIs are sampled from each selected tool.
- Instruction generation: Nearly 200k qualified instruction–relevant API pairs are collected, including 87,413 single-tool, 84,815 intra-category, and 25,251 intra-collection instances.These pairs are used to train the API retriever.
- Solution path annotation: DFSDT constructs a decision tree that evaluates multiple reasoning paths, continues promising nodes, abandons others, and expands distinct alternatives.It is designed to address error propagation and limited exploration in CoT or ReACT.
3 EXPERIMENTS
The experiments evaluate ToolEval, API retrieval, DFSDT, ToolLLaMA, and generalization across instruction and API settings. Results show strong retrieval, reasoning, tool-use, and out-of-distribution performance.
- Evaluation: 87.1% pass-rate agreement and 80.3% win-rate agreement show that ToolEval closely reflects human evaluation.ToolEval uses pass rate and win rate to assess execution success and solution quality.
- API Retrieval: The API retriever consistently outperforms BM25 and text-embedding-ada-002 across single-tool and multi-tool settings.The evaluated settings include single-tool, intra-category multi-tool, and intra-collection multi-tool instructions.
- Reasoning Search: DFSDT significantly outperforms ReACT and ReACT@N across all instruction scenarios, with larger gains on harder multi-tool settings.Using DFSDT enables more successful solution-path annotations under the same budgets and expands the search space for difficult instructions.
- Main Results: ToolLLaMA with DFSDT performs much better than Text-Davinci-003 and Claude-2 and achieves performance almost on par with ChatGPT.Across the reported scenarios, its pass rate is second to GPT4+DFSDT while generalizing to unseen instructions and tools.
- Main Results: Using retrieved APIs improves ToolLLaMA's pass rate and win rate over the ground-truth API set.The retriever can identify similar APIs with better functionality and select among more than 16,000 APIs.
- OOD Generalization: ToolLLaMA achieves remarkable OOD generalization on all three APIBench datasets despite different training and instruction domains.With its API retriever, ToolLLaMA outperforms Gorilla+BM25 on AST accuracy for HuggingFace and TorchHub.
4 RELATED WORK
Related work covers tool learning, instruction tuning, and prompting methods for decision making. These lines of research motivate ToolLLM's focus on tool interaction, data construction, and reasoning over actions.
- Tool Learning: Tool-learning studies examine how LLMs use tools and make decisions in complex environments.External tools can provide real-time factual knowledge and multimodal functionality.
- Instruction Tuning: Instruction tuning improves instruction understanding and response generation, while self-instruct reduces the burden of manual data annotation.Recent work extends this data-curation trend to multi-turn dialogue.
- Decision Making: Prompting methods decompose tasks and ground plans, but the cited decision-making studies lack decision retraction mechanisms.Without retraction, an initial error can lead to downstream problems in the reasoning process.
5 CONCLUSION
The paper presents ToolLLM as a framework for eliciting tool use in LLMs through data, reasoning, evaluation, and API retrieval. ToolLLaMA matches ChatGPT and generalizes to unseen and out-of-distribution APIs.
- Conclusion: ToolBench covers 16k+ real-world APIs and practical single-tool and multi-tool scenarios with minimal human supervision.Its construction uses ChatGPT for instruction tuning data generation and solution-path annotation.
- Conclusion: DFSDT reinforces planning and reasoning by enabling strategic navigation through multiple reasoning paths.ToolEval provides automatic evaluation, while a neural API retriever recommends relevant APIs.
- Conclusion: ToolLLaMA matches ChatGPT and shows strong generalization to unseen APIs and out-of-distribution domains.The conclusion frames the work as a foundation for research at the intersection of instruction tuning and tool use.
A.1 DETAILS FOR FILTERING RAPIDAPI
ToolBench filters RapidAPI tools through operational testing and example-response evaluation to retain reliable, functional APIs.
- API Filtering: Initial testing checks whether each API is operational and discards APIs that fail this basic criterion.This step removes nonfunctional APIs before later evaluation.
- API Filtering: Example-response evaluation calls each API and assesses the resulting response for response time and quality.The process uses actual API responses to evaluate effectiveness.
- API Filtering: The filtering process is designed to ensure that the retained ToolBench API set is reliable and functional.The passage describes operational testing and response evaluation as components of this filtering process.
A.2 API RESPONSE COMPRESSION
API responses can exceed LLM context limits because they contain redundant information, so ToolLLM compresses them while preserving critical content.
- Redundant and overly long API responses can exceed LLM context limits.
- Response compression removes unimportant response keys while retaining critical information.ChatGPT analyzes an example response for each API to identify removable keys.
A.3 DETAILS FOR TRAINING TOOLLLAMA
ToolLLaMA is trained in a multi-round conversation format that mirrors ChatGPT inputs and outputs, with function-call information concatenated into the prompt.
- ToolLLaMA uses multi-round conversation training with ChatGPT-compatible inputs and outputs.
- Function-call information is concatenated into the input because ChatGPT’s function-call field organization is unclear.
- Training uses a 5 × 10^-5 learning rate, 4 × 10^-2 warmup ratio, batch size 64, and maximum sequence length 8192.
A.4 DETAILS FOR DFSDT
DFSDT balances search effectiveness and API-call cost by avoiding child-node sorting while retaining DFS-like coverage and ReACT-like efficiency on simple instructions.
- Pairwise child-node sorting requires approximately O(n log n) OpenAI API calls for n child nodes.
- DFSDT skips child-node sorting and uses pre-order traversal because highest-ranked nodes are often generated first.
- When actions are not retracted, DFSDT degrades to ReACT and matches ReACT’s efficiency for simple instructions.
- DFSDT explores nearly the same nodes as classical DFS, allowing it to handle complex instructions solvable only through DFS.
- DFSDT achieves performance similar to DFS while significantly reducing costs.
A.5 DETAILS FOR TOOLEVAL
ToolEval evaluates tool-use paths with pass rate and win rate, using ChatGPT judgments across solvable and unsolvable instructions and multiple quality dimensions. Its human comparison supports evaluator agreement, while the authors note that tool-use evaluation remains difficult because valid paths and preferences can vary.
- ToolEval uses pass rate and win rate as its two automatic tool-use evaluation metrics.
- Pass-rate evaluation first classifies instructions as solvable or unsolvable and assigns Pass, Fail, or Unsure labels to solution paths.
- For solvable instructions, giving up after extensively trying APIs can receive Pass, whereas giving up after limited attempts or useful results receives Fail.
- For unsolvable instructions, a justified final answer or refusal can receive Pass, while hallucinated false-positive completion receives Fail.
- At least four ChatGPT predictions are majority-voted for each path’s pass-rate and win-rate evaluation.
- Win rate compares solution paths using information richness, factuality, reasoning, milestones, exploration, and API-call cost.
- Human comparison found high agreement with ToolEval, supporting its use as a credible evaluator for pass rate and win rate.
- Tool-use evaluation remains difficult because instructions may have infinitely many valid paths and human experts can disagree about which path is better.