Source-linked AI summary
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li
TL;DR
API-Bank addresses unanswered questions about LLM tool use by benchmarking planning, retrieval, and API calling, while providing training data for tool-augmented models. Experiments show current models have meaningful but inconsistent tool-use abilities, and Lynx improves substantially over Alpaca-7B while remaining behind GPT-4.
Problem
API-Bank addresses limited evidence about how effectively LLMs use external APIs, how to improve that ability, and which obstacles hinder reliable tool use.
Method
The paper introduces API-Bank, a benchmark and training resource covering API planning, retrieval, calling, and response evaluation, and trains Lynx with automatically generated tool-use data.
Results
Lynx improves average performance across three API-use capabilities by 24% over Alpaca-7B, approaches GPT-3.5, and remains 21% behind GPT-4.
Takeaways & Limitations
API-Bank finds that diverse, high-quality training data can enhance tool use, while proper API calls and adherence to API documentation remain significant challenges.
Takeaways & Limitations
API-Bank evaluates only English, fine-tunes only Lynx-7B, and does not report results for its larger commercial online model.
Abstract
from arXiv · showhide
Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.
1 Introduction
API-Bank addresses how effectively LLMs use external APIs, how tool use can be improved, and which obstacles remain. It introduces an executable benchmark, a large training set, and experiments evaluating existing models and Lynx.
- Benchmark motivation: API-Bank evaluates planning, API retrieval, and API calling for tool-augmented LLMs.The benchmark was designed around user requirements collected through interviews and includes these three capabilities.
- Benchmark construction: 73 APIs, 314 manually annotated dialogues, and 753 API calls form the executable evaluation system.The system is intended to assess LLM tool utilization through runnable API interactions.
- Training resource: 2,138 APIs and 1,888 dialogues constitute a comprehensive training set spanning diverse API-use abilities.The training data includes 4,149 API calls and covers three levels of API usage ability.
- Training resource: Multi-agent generation reduces tool-use data annotation cost by 98% compared with human annotation, enabling training of Lynx from Alpaca-7B.Five collaborative agents generate domains, APIs, queries, calls, and responses.
- Experimental findings: 24% average improvement across three API-use capabilities makes Lynx stronger than Alpaca-7B, while a 21% gap from GPT-4 remains.Lynx approaches GPT-3.5 performance but does not match GPT-4.
2 Design Principles of API-Bank
API-Bank’s design principles define tool use around realistic user needs, API-pool size, and the number of calls required per turn. These dimensions yield three graded abilities and emphasize diverse, authentic, executable evaluation data.
- User-informed requirements: Interviews with over 500 users informed API-Bank’s first definitions and data standards for tool-augmented LLM abilities.The requirements were collected because authoritative definitions and benchmarks were unavailable.
- API-pool size: Small API pools allow direct selection, whereas large pools require retrieving potentially suitable APIs under input-length constraints.The distinction is based on whether all API metadata can be supplied to the LLM.
- Calls per turn: Users may require either one API call per turn or autonomous sequences of calls for complex requirements.The benchmark distinguishes patient decomposition across queries from step-by-step execution within one turn.
- Ability grading: Call uses known APIs, Retrieval+Call retrieves and calls one unknown API, and Plan+Retrieval+Call repeatedly plans, retrieves, and calls unknown APIs.The first two expected conditions were merged because their difficulty was considered similar.
- Data quality principles: The benchmark requires domain diversity, API authenticity, API diversity, and evaluation authenticity through functional real-time interaction.Evaluation checks whether executed API calls fulfill user requirements rather than relying only on surface text.
3 Evaluation System of API-Bank
The evaluation system combines realistic APIs, manually annotated dialogues, API Search, and execution-based metrics. It tests increasingly demanding combinations of calling, retrieval, and planning.
- System implementation: 73 commonly used APIs were implemented in a shared framework, including weather forecasting and text-to-image generation.Database APIs use initialized databases, while external-information APIs are controlled for reproducibility.
- System implementation: API Search identifies potentially relevant APIs before calls in the Retrieval+Call and Plan+Retrieval+Call settings.It compares query-keyword embeddings with API metadata to support retrieval when the available APIs are unknown.
- Data annotation: Call dialogues use known APIs, Retrieval+Call decomposes complex requirements into single-API queries, and Plan+Retrieval+Call annotates an unsplit sequential call chain.These procedures create supervision for the benchmark’s three graded abilities.
- Data annotation: 314 retained dialogues contain 753 API calls after 21.5% of 400 annotated dialogues were discarded for annotation issues.Annotation involved multiple annotators checking format, logical consistency, and call reasonability.
- Evaluation metrics: API-call Accuracy measures consistency with annotated database operations and results, while ROUGE-L measures response quality.The system is initialized with default database values before each evaluation.
4 Training Set of API-Bank
API-Bank develops a scalable training set because manual annotation is costly and struggles to produce diverse, authentic API pools. Its multi-agent method decomposes generation into coordinated stages.
- Motivation: Manual annotation costs $8 per dialogue, and human annotators generated only 100 APIs, limiting scalable diversity and authenticity.These constraints motivate automated construction of a large tool-augmented training set.
- Multi-agent generation: A multi-agent method generates tool-use training data rapidly and cost-effectively while targeting API-Bank’s design principles.The method is intended to address the cost and diversity limitations of manual construction.
- Motivation: Self-instruct produced only 5% usable data under the full requirements, while GPT-4 increased the available rate to 25%.The requirements covered domain and API diversity/authenticity plus planning, retrieval, and calling abilities.
- Generation design: The generation pipeline models dependencies among domain, API, query, ability, and API call-response elements.Together, these elements determine the query type, API calls, and appropriate response.
- Generation design: Five agents sequentially generate domains, APIs, ability-matched queries, simulated API calls and responses, and quality checks.Public API examples are supplied during API generation to support simulated API authenticity.
5 Benchmark Analysis
API-Bank combines broad coverage, realistic dialogue settings, and evaluation of three tool-use abilities. Its automatically generated training data achieves a 94% available rate, while the benchmark includes distribution-shift testing.
- 1,008 domains, 2,211 APIs, 2,202 dialogues, and 6,135 turns define the benchmark’s overall scale.
- The benchmark covers Call, Retrieve+Call, and Plan+Retrieve+Call dialogues, with 934, 769, and 499 dialogues respectively.Training and testing dialogues average 2.76 and 2.91 turns, respectively.
- The benchmark evaluates generalization using a distribution-shift evaluation set because its automatically generated training data differs from manually annotated evaluation data.The differences concern domain, API scope, and dialogue content.
- 94% of Multi-agent-generated training samples are available, an 89% improvement over single-agent self-instruct data.Each evaluation instance was reviewed by four annotators, and 100 training samples were randomly evaluated.
- 78% of data filtered by the tester agent failed to follow the design principles as intended, indicating that the tester agent processes data according to those principles.
- API-Bank emphasizes diversity and realism by covering many domains and APIs and supporting multi-turn, multi-call dialogues.The comparison also considers API calls, responses, and three distinct tool-use abilities.
6 Related Work
Related work uses external tools to supplement language-model knowledge and solve tasks through search, question answering, translation, calculation, and other APIs. Prior systems also demonstrate integration with very large API collections.
- External tools can offload exact computation and information retrieval from language-model weights to modules such as Python interpreters and search engines.
- WebGPT and ReAct use search APIs, while Toolformer, ART, ToolkenGPT, and TRICE combine multiple tool types for NLP tasks.
- ChatGPT Plugins and TaskMatrix.AI demonstrate that language models can integrate with thousands to millions of APIs.
7 Experiments
API-Bank experiments show that tool use depends on both instruction tuning and reasoning, while fine-tuning on its diverse training data substantially improves smaller models. Error analysis identifies API selection, call formatting, and parameter handling as persistent challenges.
- Experimental Setup: API-Bank evaluates LLMs across API calling, retrieval, and planning, using models including GPT-3, GPT-3.5, GPT-4, ChatGLM, Alpaca, and Lynx.The benchmark’s evaluation system and model comparisons target progressively more demanding tool-use abilities.
- Main Results: About 20% API-call accuracy from Alpaca and ChatGLM contrasts with GPT-3 Davinci’s nearly absent API-use capability.The authors associate API calling with instruction understanding, which GPT-3’s training omitted through its lack of instruction tuning.
- Main Results: 35 points and 0.44 Rouge-L: GPT-3.5 surpasses Alpaca-7B in API-call correctness and response Rouge-L, but performance drops 21% and 17% in harder retrieval and planning settings.The declines occur in Retrieve+Call and Plan+Retrieve+Call relative to simple API calls.
- Main Results: Nearly 50%: GPT-4 improves over GPT-3.5 in the most difficult Plan+Retrieve+Call setting, while showing similar Retrieve+Call performance.GPT-4 also records a 4-point improvement in API calls compared with GPT-3.5.
- Main Results: 26 points and 0.41 Rouge-L: Lynx outperforms Alpaca-7B in API-call correctness and response Rouge-L, approaching GPT-3.5.Lynx is trained from Alpaca-7B initialization on API-Bank data generated through the Multi-agent strategy.
- Error Analysis: API name mismatch accounts for 61% of Lynx’s errors, while problematic input parameters contribute 32% and GPT-4’s API-search failures account for 68% of its problems.Additional issues include incorrectly formatted calls, hallucinated APIs, missing or invalid parameters, and multiple simultaneous calls that violate prompt instructions.
8 Conclusion
API-Bank is a benchmark for tool-augmented LLMs that evaluates API use and identifies remaining challenges in reliable tool utilization.
- API-Bank comprises 2,202 dialogues involving 2,211 APIs from 1,008 domains.
- Current LLMs demonstrate some ability to use APIs, but reliable service provision remains an open challenge.
- Building diverse, high-quality training data is presented as a promising approach to improving LLM tool use.
- Correct API calls and strict adherence to API documentation remain significant challenges for LLMs.
9 Limitations
API-Bank has scope and reporting limitations involving language coverage, model scale, and an unreported commercial model.
- API-Bank focuses solely on English, with other languages left for future data construction and model evaluation.
- The study fine-tunes only Lynx-7B and does not explore larger-scale models, although Lynx-7B performs comparably to GPT-3.5.
- Results from a commercially viable tool-augmented LLM are not reported or analyzed because of anonymity restrictions.
10 Ethical Statement
The paper reports ethical safeguards for user interviews and the originality of its API-tool implementations.
- Interview participants were informed that their feedback could support product development and publication, without disclosing personal private information.
- All API tools in the dataset are original implementations and do not infringe upon existing ...
A Appendix
The appendix documents evaluation prompts, error categories, dataset statistics, and examples of API calling, retrieval, and planning.
- API requests are generated from prior dialogue context using the format [ApiName(key1='value1', key2='value2', ...)].
- The error taxonomy includes API hallucination, exceptions, invalid or missing parameters, unparseable formats, and absent API calls.
- API evaluation and response evaluation use separate prompts, kept concise to test fundamental model capabilities.
- The test-set appendix includes domain statistics and examples such as mental health support and dental cost estimation.
- The appendix illustrates calling APIs, retrieving and calling APIs, and planning, retrieving, and calling APIs.