PaperlayerView exact revision on arXiv

Source-linked AI summary

APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets

Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, Caiming Xiong

arXiv:2406.18518v1cs.CLcs.AIcs.LGcs.SE

TL;DR

Function-calling agents need diverse, reliable training data, but existing datasets can be static, noisy, and insufficiently verified. APIGen addresses this gap with structured generation and three-stage verification, producing large datasets whose trained models achieve strong benchmark performance, while currently remaining limited to REST APIs, Python functions, and single-turn calling.

  • Problem

    Existing function-calling datasets are largely static and lack comprehensive verification, creating challenges when models encounter unseen APIs.

  • Method

    APIGen samples APIs and seed examples, generates standardized JSON function-call data, and filters it through format, execution, and semantic verification.

  • Results

    The 6.7B xLAM model ranks 6th on the Berkeley Function-Calling Leaderboard, while the 1.3B model outperforms GPT-3.5-Turbo and many larger models.

  • Takeaways & Limitations

    APIGen shows that verified, diverse data can support strong function-calling performance from models with substantially fewer parameters.

  • Takeaways & Limitations

    The current framework and dataset cover only REST APIs and Python functions and implement only single-turn function calling.

Abstract

from arXiv · show

The advancement of function-calling agent models requires diverse, reliable, and high-quality datasets. This paper presents APIGen, an automated data generation pipeline designed to synthesize verifiable high-quality datasets for function-calling applications. We leverage APIGen and collect 3,673 executable APIs across 21 different categories to generate diverse function-calling datasets in a scalable and structured manner. Each data in our dataset is verified through three hierarchical stages: format checking, actual function executions, and semantic verification, ensuring its reliability and correctness. We demonstrate that models trained with our curated datasets, even with only 7B parameters, can achieve state-of-the-art performance on the Berkeley Function-Calling Benchmark, outperforming multiple GPT-4 models. Moreover, our 1B model achieves exceptional performance, surpassing GPT-3.5-Turbo and Claude-3 Haiku. We release a dataset containing 60,000 high-quality entries, aiming to advance the field of function-calling agent domains. The dataset is available on Huggingface: https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k and the project homepage: https://apigen-pipeline.github.io/

1 Introduction

Function-calling agents depend on training data that is diverse, accurate, and robust to unseen APIs. APIGen addresses this need with verified data generation and releases a large dataset alongside strong model results.

  • Motivation: Current function-calling datasets are often static and insufficiently verified, limiting reliability and adaptation to unseen APIs.Models trained mainly on one API domain may struggle when asked to handle APIs from another domain.
  • APIGen: APIGen generates diverse function-calling data and verifies each example through format, execution, and semantic checks.The framework is designed to support fine-tuning with data reflecting real-world API variability and complexity.
  • Released Dataset: 60,000 entries cover 3,673 APIs across 21 categories, including rarely represented parallel function-calling queries.The dataset is intended for training and testing function-calling models.
  • Results: The 6.7B model ranks 6th on the Berkeley Function-Calling Leaderboard, surpassing GPT-4o and Gemini-1.5-Pro.The 1.3B model outperforms GPT-3.5-Turbo.

2 Related Work

Prior work has developed tool-use agents, agent datasets, and benchmarks, but many datasets remain insufficiently verified. APIGen builds on this landscape by targeting reliable, diverse function-calling data.

  • Tool-use Agents: Tool-use systems such as RestGPT, Toolformer, xLAM, Octopus-v4, NexusRaven, and Gorilla OpenFunctions-v2 enable LLM interaction with APIs or specialized tools.NexusRaven and Gorilla OpenFunctions-v2 are described as strong open-source function-calling models without released training datasets.
  • Agent Datasets: Agent datasets span tasks including embodied environments, web shopping, browsing, knowledge graphs, operating systems, databases, and API dialogues.Examples include AgentInstruct, APIBank, Toolalpaca, ToolBench, AgentOhana, and Lumos.
  • Agent Datasets: Most existing agent datasets were not rigorously verified and commonly contain noisy data.This limitation motivates stronger quality control for function-calling dataset construction.
  • Benchmarks: Benchmarks evaluate agent abilities across web interaction, reasoning, decision making, function calling, and tool usage.AgentBoard and ToolEval are examples of benchmarks covering multi-turn tools and functional calling.

3 APIGen Framework

APIGen combines structured generation, diversity-oriented sampling, and hierarchical verification to construct scalable function-calling datasets. Its pipeline filters malformed, non-executable, and semantically misaligned examples before recycling accepted data.

  • Framework Design: APIGen targets data quality, diversity, and collection scalability through verification, sampling, prompt templates, and a unified modular format.The framework is designed to incorporate diverse API sources through format conversion.
  • 3.1 Data Generation Overview: The generation workflow samples APIs and seed QA pairs, formats them as standardized JSON, selects a prompt template, and produces JSON function calls.The prompt steers the LLM toward query-answer pairs matching the generation objective.
  • 3.1 Data Generation Overview: Standardized JSON enables required-field checks, function-call parsing, argument validation, and integration of Python or REST API sources through converters.Invalid fields, unknown arguments, and hallucinated functions are discarded.
  • 3.2 Multi-Stage Data Verification: The format checker validates JSON structure and fields, the execution checker runs calls against backends, and the semantic checker assesses alignment with the query objective.Semantic verification considers function choice, arguments, call count, execution errors, and result relevance.
  • 3.2 Multi-Stage Data Verification: Examples passing all three verification stages are treated as high-quality and added back to the seed dataset for future generation.This creates a feedback path between verification and subsequent diverse data generation.
  • 3.3 Methods to Improve Dataset Diversity: APIGen promotes diversity through query styles, API sampling, example sampling, and randomized numbers of sampled APIs and examples.Its four query categories are simple, multiple, parallel, and parallel multiple.
  • 3.3 Methods to Improve Dataset Diversity: APIGen provides a large-scale, high-quality dataset for parallel-related function calling, which existing public training data largely lacks.The authors describe this as the first such dataset to their knowledge.

4 Dataset Preparation and Collection

The dataset collection process filters and cleans executable, documented APIs before generating verified function-calling data. The resulting corpus spans 3,673 APIs across 21 categories, with multi-stage checks removing low-quality examples.

  • Dataset preparation: The pipeline begins by selecting real-world APIs that are executable and supported by thorough documentation.APIs were primarily sourced from ToolBench, which contains REST APIs from RapidAPI Hub.
  • Dataset preparation: Cleaning removes APIs with malformed documentation, missing parameters, inaccessible endpoints, execution errors, or noisy descriptions.The procedures include documentation filtering, endpoint testing with example parameters, and docstring regeneration.
  • Dataset API sources: 3,673 executable APIs were retained: 3,539 REST APIs with good documentation and 134 documented Python functions.The Python functions cover mathematics, finance, and data management, among other fields.
  • Dataset API sources: The APIs were consolidated into 21 distinct categories spanning fields including technology, social sciences, education, and sports.The consolidation merged semantically overlapping categories such as Finance and Financial.
  • Collection setup and dataset details: Data generation samples combinations of APIs, seed data, and prompt templates across several base language models, targeting 40,000 data points per model.Table 1 reports verified counts and filtering at each verification stage.
  • Collection setup and dataset details: Format, execution, and semantic checks filter malformed, non-executable, or query-misaligned examples before release.The semantic checker removes outputs whose function calls or execution results fail to address the user’s query.
  • Collection setup and dataset details: Stronger generator models have higher pass rates, whereas smaller models more often produce data that cannot be executed.The authors therefore recommend strict verification when weaker models are used for data generation.
  • Collection setup and dataset details: Approximately 60,000 high-quality examples generated by the two strongest models are released across the 3,673 APIs and 21 categories.Each released data point was verified using real-world APIs and covers the query styles described in the paper.

5 Experiments

The experiments evaluate APIGen-generated data through BFCL benchmarking, model comparisons, and ablations of filtered data. Results show strong performance from both model sizes and benefits from multi-stage data verification.

  • Experiment Setup: Two models, xLAM-1B (FC) and xLAM-7B (FC), are trained from DeepSeek-Coder bases using the xLAM training pipeline.The experiments compare these models with GPT-4, Claude-3, Gemini, Llama3, Mixtral, OpenFunctions-v2, Command R+, and others.
  • Benchmark: BFCL evaluates function-calling across programming languages and application domains, including parallel and multiple-function calls in 2,000 test cases.Its metrics include AST evaluation for syntactic accuracy and executable function evaluation for operational correctness.
  • Experiment Results Analysis: 6th-place xLAM-7B surpasses GPT-4o, GPT4-Turbo-FC, Llama3-70B, multiple Claude-3 models, and other strong models on the BFCL leaderboard.Table 2 ranks models using overall accuracy, a weighted average across evaluation categories.
  • Experiment Results Analysis: 24th-place xLAM-1B outperforms Claude-3 Haiku, Command-R-Plus, DBRX-Instruct, Mistral-large, and GPT-3.5-Turbo-0125.Both xLAM models improve on complex parallel and multiple function-calling scenarios; the DeepSeek-Coder-v1.5 base ranks 45th.
  • Experiment Results Analysis: Adding stage-2 execution-checker or stage-3 semantic-checker failures to training harms BFCL performance, especially for the smaller model.The ablation simulates training without the full verification process, supporting APIGen’s filtering of low-quality data.

6 Conclusion

APIGen generates reliable and diverse function-calling datasets through multi-stage verification. The framework and released dataset support smaller models and remain limited to REST APIs, Python functions, and single-turn calling.

  • APIGen generates reliable and diverse function-calling datasets by leveraging a multi-stage verification process.
  • The approach shows that high-quality data can be as important as model size for achieving strong function-calling performance.
  • Smaller models achieve competitive results and significantly enhanced function-calling capabilities with the released approach and dataset.
  • APIGen currently supports only REST APIs and Python functions, with generation limited to single-turn function calling.

A.1 Dataset Documentation and Intended Uses

The dataset is intended for training and evaluating function-calling agents and is distributed under a stated licensing and maintenance plan. The authors also provide a responsibility statement concerning rights in dataset collection, distribution, and use.

  • The dataset is intended for training and evaluating function-calling agents using diverse query-answer pairs with verified function calls.
  • The dataset is available through the project homepage and Huggingface under the Creative Commons Attribution 4.0 License.
  • The authors state that they bear responsibility for the datasets and believe their collection, distribution, and use violate no rights.

A.2 JSON Data Format and Examples

The appendix defines a JSON format that pairs a query with available tools and verified answers. Examples illustrate both a single weather call and parallel calls for multiple mathematical intentions.

  • A.2.1 Dataset Structure: The dataset JSON structure contains query, tools, and answers keys.The query states the problem, tools describe callable functions, and answers record selected tools with arguments.
  • A.2.1 Dataset Structure: Each tool specifies a name, description, and parameter object containing parameter types, descriptions, and required-status fields.
  • A.2.1 Dataset Structure: Each answer identifies the tool used and provides an arguments object containing the values passed to that tool.
  • A.2.2 Example Data: The simplest example represents a current-weather query for Palo Alto using one weather API call.
  • A.2.2 Example Data: The weather example defines location as a required string and units as an optional string, then calls the API with Palo Alto and Celsius.
  • A.2.2 Example Data: The parallel function-calling example contains multiple intentions and multiple parallel tool calls in its answers.
  • A.2.2 Example Data: The answers encode the mathematical calls and their arguments as separate entries for the two tools.

A.3 Human Evaluation of Dataset Quality

A human evaluation assessed the quality of generated data through manual inspection of 600 samples. The results indicate that the verification checkers effectively filter low-quality examples.

  • Three human evaluators manually inspected 600 released samples for parameter accuracy and appropriate numbers of API calls.
  • 28 of 600 inspected samples had minor issues, while approximately 95.3% were judged to be of very high quality.
  • The format checker validates JSON structure and required fields, while the execution checker verifies successful backend execution of generated calls.
  • Together with semantic verification, the checkers form a robust process that filters low-quality data points.

B Dataset Generation and Experiment Details

APIGen generates diverse parallel function-calling query–answer pairs with structured prompts and validates them through format, execution, and semantic checks.

  • Dataset Generation: Generated queries may combine several independent requests, requiring an answer list whose length matches the number of requests.The instructions require each request to be addressed effectively in JSON-formatted function calls.
  • Dataset Generation: APIGen prompts an LLM to generate diverse query–answer pairs for specified functions, using sampled examples and API descriptions.The template emphasizes practical scenarios, clear queries, plausible parameter values, and varied difficulty.
  • Dataset Generation: The generation instructions enforce strict JSON output containing a query and an answers list with API names and arguments.No additional text is permitted, supporting consistent downstream extraction.
  • Dataset Generation: Batching multiple query–answer pairs in one inference reduces token usage and generation costs.The template uses a number field to request multiple pairs simultaneously.
  • Verification: A semantic checker evaluates whether function calls and execution results align with the query, rejecting incorrect calls, arguments, counts, relevance, or failed executions.The checker returns a JSON pass decision and may use multiple LLMs to increase verification credibility.

B.3 Model Training

The authors train xLAM-1B and xLAM-7B on APIGen-generated data, extending the dataset with relevance-detection examples for unsuitable tools or missing arguments.

  • Model Training: The authors train xLAM-1B (FC) and xLAM-7B (FC) using the APIGen-generated dataset.The training pipeline mainly follows AgentOhana and uses eight NVIDIA A100 40GB GPUs.
  • Relevance Detection: Relevance-detection scenarios include tools that cannot solve the query and tools missing key arguments.Examples contrast a weather query with a house-price tool or with a weather tool lacking the required date argument.
  • Relevance Detection: For unsuitable or incomplete tools, the correct output is an empty tool call or a concise refusal explanation.This teaches the model to avoid producing unsupported function calls.
  • Relevance Detection: 8,000 relevance-detection data points are created by discarding called tools or dropping required parameters, then relabeling the answer.These examples target cases where available tools are irrelevant or insufficiently specified.
  • Training Prompt: Training fills sampled queries and available tools into a prompt template that asks the model to predict one or more tool calls in a specified JSON format.The template also requires correct parameter types and an empty tool_calls list when no call is needed.
  • Training Configuration: The models are trained with a 5 × 10^-6 learning rate for four epochs using AdamW, BF16, a 2048-token cutoff, and gradient accumulation.The setup uses a per-device batch size of six, cosine scheduling, and 50 warmup steps.
Loading 2406.18518v1…