Source-linked AI summary
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis A. Lastras
TL;DR
API-BLEND addresses limited and unevenly evaluated data for training and testing LLM API use. It curates and transforms existing datasets with hybrid generation into a 10-dataset corpus covering API detection, slot filling, and sequencing. Models fine-tuned on API-BLEND generalize better to OOD API tasks than other tool-augmented LLMs.
Problem
Training and benchmarking data for API tasks are scarce, while synthetic corpora can lack diversity and existing evaluations may inadequately validate API use.
Source: §1 · pp. 1–2, §2.2 · p. 3Method
Results
Takeaways & Limitations
API-BLEND supports training and systematic benchmarking of tool-augmented LLMs across API detection, slot filling, and sequencing tasks.
Source: §1 · p. 2, §5 · p. 8API-BLEND does not cover environment interactions for API agents and is focused on English API commands.
Source: §6 · p. 9
Abstract
from arXiv · showhide
There is a growing need for Large Language Models (LLMs) to effectively use tools and external Application Programming Interfaces (APIs) to plan and complete tasks. As such, there is tremendous interest in methods that can acquire sufficient quantities of train and test data that involve calls to tools / APIs. Two lines of research have emerged as the predominant strategies for addressing this challenge. The first has focused on synthetic data generation techniques, while the second has involved curating task-adjacent datasets which can be transformed into API / Tool-based tasks. In this paper, we focus on the task of identifying, curating, and transforming existing datasets and, in turn, introduce API-BLEND, a large corpora for training and systematic testing of tool-augmented LLMs. The datasets mimic real-world scenarios involving API-tasks such as API / tool detection, slot filling, and sequencing of the detected APIs. We demonstrate the utility of the API-BLEND dataset for both training and benchmarking purposes.
1 Introduction
API-BLEND addresses scarce API-task data by curating and transforming diverse human-annotated datasets into training and benchmarking resources. Models trained on this diversity show stronger out-of-domain generalization than other tool-augmented approaches.
1 Introduction
- API-task data for training and benchmarking LLMs has been scarce, especially for sequencing alongside API detection and slot filling.Existing datasets often emphasize detection or slot filling, while sequencing remains relatively underrepresented.Source: §1 · pp. 1–2
- Synthetic API data can be inexpensive and large-scale, but sampling bias and limited diversity can cause overfitting to in-distribution APIs and weak OOD generalization.Related datasets have also overlooked human-annotated resources from domains such as digital assistants and semantic parsing.Source: §1 · p. 2
- API-BLEND combines five human-annotated datasets with LLM-assisted generation, producing over 150k training, 17k development, and 11k test instances.Its transformation emphasizes sequencing while also covering API detection and slot filling.Source: §1 · p. 2
- The corpus spans semantic parsing, dialogue, and digital-assistant domains, increasing API-data diversity for training.This cross-domain construction is intended to support broader API generalization.Source: §1 · p. 2
- Models trained on API-BLEND achieve significantly better OOD generalization than other state-of-the-art methods, as illustrated by API-BLEND-LLaMA-2-7B outperforming Lynx and ToolLLaMA-2-7B.The benchmark includes five existing OOD datasets within a 10-dataset collection: five for training and five for OOD testing.Source: Figure 1 · p. 1, §1 · p. 2, Figure 2 · p. 2
2 Related Work
Related work develops general tool-use methods and increasingly broad API datasets, but evaluation remains uneven. API-BLEND is positioned against limitations involving tool necessity, API-call structure, synthetic data, and validation quality.
2.1 Tool-Usage by LLMs
- Tool-use research has progressed from human annotation toward self-supervision and few-shot prompting, while external tools support information access, computation, and program execution.Prompting frameworks have also been studied for cost-effectiveness, performance, and data-generation quality.Source: §2.1 · p. 2
- Many standard benchmarks do not clearly distinguish questions solvable from internal LLM knowledge from those requiring external tool calls.This weakens their ability to evaluate tool use specifically.Source: §2.1 · pp. 2–3
2.2 API Datasets
- API datasets vary in API volume, scenario structure, number of calls, and whether calls are executed against real APIs or simulated by a prompted LLM.Some datasets use single calls, whereas others require multiple calls across tools or categories.Source: §2.2 · p. 3
- Self-supervised API corpora often evaluate generated examples through hallucination rates, separate-LLM judgments, small manual ground-truth sets, or manual inspection.These heterogeneous validation strategies constitute a limitation in evaluating API-use scenarios.Source: §2.2 · p. 3
3 API-BLEND Dataset Curation
API-BLEND curates existing datasets into API-sequencing tasks through language-model assistance, rule-based conversion, and off-the-shelf API data. The resulting collection spans multiple domains and supports training, in-domain evaluation, and OOD testing.
3 API-BLEND Dataset Curation
- API-BLEND maps a natural-language utterance to an ordered sequence of API calls with parameter names and values using three curation approaches.The approaches are language-model-assisted generation, grammar-rule conversion of semantic-parsing and assistant notations, and off-the-shelf API datasets.Source: §3 · p. 3
3.1 Language Model Assisted Generation
- SeqSGD and SeqMultiWoz use prompted generation and filtering to convert multi-domain dialogues into API-sequencing datasets.SeqSGD contains 6.8K train, 1.1K validation, and 1.6K test examples; SeqMultiWoz contains about 6.8K train, 485 validation, and 1K test samples.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
SeqATIS and SeqSNIPS:
- SeqATIS and SeqSNIPS reconstruct API-specific parameters from multi-intent utterances by splitting intents, parsing IOB slots, and merging the results into API sequences.SeqATIS has around 11.5K training examples, while SeqSNIPS has around 40K; their average API sequence lengths are 2.13 and 1.96, respectively.Source: SeqATIS and SeqSNIPS · pp. 4–5, Figure 4 · p. 4
SeqToolQA:
- SeqToolQA adds intermediate API sequences to ToolQA questions because the original dataset provides final answers but not the tool steps required to produce them.The resulting dataset abstracts 17 APIs across 6 domains and contains 358 test examples.Source: SeqToolQA · p. 5
SeqTopV2:
- SeqTopV2 converts semantic-parsing intents and slots into ordered APIs, removes unsupported or ambiguous examples, and retains 94K, 13.5K, and 6K train, development, and test examples.The conversion preserves prerequisites such as CREATE_REMINDER following SEND_MESSAGE.Source: SeqTopV2 · p. 5
ToolBench:
- ToolBench, ToolLLM, API Bank, and ToolAlpaca are used as-is or through test-set transformations for OOD evaluation rather than newly transformed training data.The ToolBench subset covers Home-Search and Booking, while API Bank dialogues are divided into examples containing the conversation and API sequence required thus far.Source: ToolBench · pp. 5–6
4 Experiments and Results
The experiments compare nine open-source models across few-shot testing, target-dataset instruction tuning, and combined-dataset instruction tuning. A shared instruction template asks models to sequence available APIs and slots for each query.
4.1 Baselines
- Nine open-source models are evaluated under few-shot, target-dataset fine-tuning, and combined-dataset fine-tuning settings.The combined setting tests whether one model trained across datasets can perform well on each individual test set and includes OOD evaluation.Source: §4.1 · p. 6
4.2 Instruction Tuning
- The instruction template supplies possible APIs and slots, optional in-context examples, a query, and an answer field requiring the API call order.Three in-context examples are added for non-fine-tuned and OOD evaluation, but removed otherwise.Source: §4.2 · p. 6
4.3 Settings and Parameters:
All models are fine-tuned with QLoRA under controlled sample, epoch, batch, accumulation, and learning-rate settings. Target-dataset training uses 100k samples, whereas combined training runs for two epochs over the accumulated datasets.
4.3 Settings and Parameters:
- QLoRA fine-tunes all models, using 100k training samples for targeted datasets and two epochs over accumulated datasets for combined training.Both settings use batch size 1, gradient accumulation of 8, and learning rate 5e−5.Source: §4.3 · p. 6
4.4 Metrics
The evaluation measures API selection, parameter extraction, and adherence to the required API order using retrieval metrics and longest common subsequence.
4.4 Metrics
- F1 APIs and F1 slots/Parameters compare predicted APIs and parameter values with their gold counterparts, while LCS measures predicted versus gold API-name sequences.The metrics jointly assess API detection, slot filling, and sequencing behavior.Source: §4.4 · p. 6
4.5 In-Distribution Evaluation Results
In-distribution evaluation compares API selection, parameter extraction, and sequence matching across individual-dataset and combined-dataset fine-tuning settings. Combined training improves weighted average performance overall, although dataset-specific trade-offs remain.
Fine-tuning on One Dataset
- More than 90% API-F1 is achieved across single-dataset evaluations, reaching 100% on SeqMultiWoz.Parameter-name and parameter-value detection remains more difficult, especially on SeqSGD.Source: §4.5 · p. 6
No Fine-tuning
- 3-shot LLaMA-2-70B, Falcon-180B, and ToolLLaMA-2-7B often identify APIs but fail to recover the parameters needed to call them.The evaluation covers SeqATIS, SeqSNIPS, SeqSGD, SeqMultiWoz, and SeqTopV2.Source: §4.5 · p. 6
Evaluation Setup
- Table 2 reports API-F1, Parameter-F1, and LCS-F1, with weighted averages calculated by the number of test samples in Table 1.These metrics jointly evaluate API detection, parameter extraction, and sequence matching.Source: Table 2 · p. 7
All Training Datasets
- Combined-dataset fine-tuning achieves better weighted average performance than single-dataset training overall.Performance is lower on SeqATIS and SeqMultiWoz, similar on SeqSNIPS, and better on SeqSGD and SeqTopV2.Source: §4.5 · p. 7
4.6 Out-Of-Distribution Evaluation
Out-of-distribution evaluation tests models on five external suites and shows that API-BLEND-fine-tuned models outperform other API/tool-augmented models overall. Performance varies by benchmark and model, while parameter extraction remains difficult for code-, SQL-, and math-heavy slots.
Evaluation Setup
- The OOD suite includes ToolLLM subcategories G1–G3, API-Bank levels L1–L2, and ToolBench Home Search and Booking subsets.ToolLLM is API-only and therefore has no parameter score in the reported evaluation.Source: §4.6 · p. 7
OOD Results
- API-BLEND-fine-tuned models perform better than other API/tool-augmented models on the OOD evaluation.The evaluation uses ToolLLM, API-Bank, ToolBench, SeqToolQA, and ToolAlpaca.Source: §4.6 · p. 7
- Falcon-40B and StarCoder-15B perform better on ToolLLM, while FLAN-T5-XXL and MPT-30B do well on API-Bank and ToolAlpaca.Falcon-40B outperforms the other models on both ToolBench datasets.Source: §4.6 · pp. 7–8
- SeqToolQA yields low Parameter-F1 because its parameter values include code, SQL, and mathematical functions absent from the training data.Models obtain some API-detection points but struggle to generate these special slot values from few in-context examples.Source: §4.6 · p. 8
4.7 Qualitative Studies
The qualitative study attributes many slot-detection errors to parameter-value normalization and semantically similar parameter names. These mismatches can occur despite correct intent or values, and the authors reserve normalization issues for future work.
4.7 Qualitative Studies
- Parameter names and values are the main observed causes of slot-detection errors in in-distribution and OOD evaluations.The study examines failed outputs and recurring model mistakes.Source: §4.7 · p. 8
4.7.1 Unnormalized Slot-Values
- Different parameter-value formats can produce programmatic mismatches even when human evaluators would consider the outputs equivalent.Examples include repeated dates, shortened locations, and alternative date-function syntax.Source: §4.7.1 · p. 8
- The authors plan to investigate unnormalized parameter-value mismatches further and leave them for future work.The issue includes outputs that represent correct intent and slot values in different formats.Source: §4.7.1 · p. 8
4.7.2 Semantically similar slot-names in API Specification
- Semantically equivalent parameter names can fail exact matching, such as leaving_date versus departure_date or destination versus to_city.The values may be correct while the generated parameter name differs from the gold specification.Source: §4.7.2 · p. 8
5 Conclusion
API-BLEND is a 10-dataset corpus for training and evaluating tool-augmented LLMs, built from real-world data across multiple domains. Models fine-tuned with API-BLEND generalize better on out-of-distribution experiments than other tool-augmented LLMs.
- API-BLEND comprises 10 datasets—five in-distribution and five out-of-distribution—with over 190k instances for training and evaluation.Source: §5 · p. 8
- The corpus is curated from real-world datasets spanning dialog, semantic parsing, digital assistants, and other domains.Source: §5 · p. 8
- Models fine-tuned with API-BLEND generalize better than other tool-augmented LLMs in out-of-distribution experiments.Source: §5 · p. 8
6 Limitations and Risks
API-BLEND does not model environment interactions for API agents and is currently restricted to English API commands. The authors identify embodied-agent settings and multilingual extensions as future directions.