Source-linked AI summary
ToolACE: Winning the Points of LLM Function Calling
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, Enhong Chen
TL;DR
Existing function-calling data is difficult to collect and annotate, while prior synthetic pipelines can lack coverage and accuracy. ToolACE automatically synthesizes diverse APIs and dialogs through self-evolution and multi-agent generation, then verifies them with rule- and model-based checks. Models trained on this data achieve state-of-the-art performance, including with only 8B parameters.
Problem
Existing function-calling data is challenging to collect and annotate, while current tool-augmented LLMs often use limited-diversity, simple, or inaccurate data.
Method
ToolACE combines self-evolving API synthesis, self-guided multi-agent dialog generation, and dual-layer rule- and model-based verification.
Results
ToolACE-trained models achieve state-of-the-art function-calling performance, with 8B-parameter models significantly outperforming existing open-source LLMs and competing with latest GPT-4 models.
Takeaways & Limitations
ToolACE provides an automated pipeline for generating accurate, complex, and diverse tool-learning data tailored to the target LLM’s capabilities.
Takeaways & Limitations
The data-generation process is intended to match the capabilities of the LLM being tuned, so data usefulness depends on model-specific complexity alignment.
Abstract
from arXiv · showhide
Function calling significantly extends the application boundary of large language models, where high-quality and diverse training data is critical for unlocking this capability. However, real function-calling data is quite challenging to collect and annotate, while synthetic data generated by existing pipelines tends to lack coverage and accuracy. In this paper, we present ToolACE, an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data. ToolACE leverages a novel self-evolution synthesis process to curate a comprehensive API pool of 26,507 diverse APIs. Dialogs are further generated through the interplay among multiple agents, guided by a formalized thinking process. To ensure data accuracy, we implement a dual-layer verification system combining rule-based and model-based checks. We demonstrate that models trained on our synthesized data, even with only 8B parameters, achieve state-of-the-art performance on the Berkeley Function-Calling Leaderboard, rivaling the latest GPT-4 models. Our model and a subset of the data are publicly available at https://huggingface.co/Team-ACE.
1 INTRODUCTION
ToolACE addresses limited diversity, complexity, and accuracy in existing function-calling data with an automated pipeline that synthesizes APIs and dialogs, verifies their quality, and improves generalization. Experiments report competitive performance from models with only 8B parameters.
- Research gap: Real-world function calling requires diverse APIs, complex or ambiguous instructions, precise API selection, and accurate parameter configuration.These requirements include parallel or dependent tool use and multi-turn interactions.
- Research gap: Existing tool-augmented LLMs mainly target simple, limited-diversity function calls, often relying on public APIs and single-turn queries.Dependent, parallel, and multi-turn interactions are comparatively neglected.
- ToolACE: ToolACE is an automated pipeline that synthesizes accurate, diverse, and complex function-calling data with awareness of the target model’s capabilities.Its framework combines tool self-evolution synthesis, self-guided dialog generation, and dual-layer verification.
- ToolACE: Tool Self-Evolution Synthesis generates tools across domains through speciation, adaptation, and evolution rather than relying only on public APIs.The process begins with pretraining data and continually updates the API pool with varied data types and constraints.
- ToolACE: Self-Guided Dialog Generation uses the target LLM as a complexity evaluator and multi-agent interactions to produce varied function-calling dialogs.The evaluator dynamically adjusts samples judged too simple or too complex, while rule and model checkers verify data quality.
- Results: With 8B parameters, ToolACE-trained models significantly outperform existing open-source LLMs and are competitive with the latest GPT-4 models on BFCL and APIBank.The reported evaluations use two widely adopted function-calling benchmarks.
2 DATA GENERATION PIPELINE
ToolACE generates tool-learning data through evolving API synthesis, model-calibrated multi-agent dialogs, and dual-layer verification. Its complexity measure uses the target model’s loss, while experiments link higher loss to harder API-selection settings.
- Pipeline overview: ToolACE deploys agents to recursively synthesize diverse APIs, collaboratively construct dialogs with appropriate complexity, and reflect on data quality.The framework comprises Tool Self-evolution Synthesis, Self-Guided Dialog Generation, and Dual-Layer Validation.
- Tool Self-Evolution Synthesis: Tool Self-Evolution Synthesis builds an API pool through speciation, adaptation, and evolution, using a hierarchical context tree to vary domains and functionalities.Evolution mutates functionalities, parameters, constraints, parameter types, and returned results.
- Tool Self-Evolution Synthesis: The TSS module generates diverse API documentation, including nested types such as lists of lists and lists of dictionaries.This expands the structural variety available for function-calling data generation.
- Self-Guided Dialog Generation: Self-Guided Dialog Generation uses the target LLM to evaluate complexity and dynamically adjust dialogs that are too simple or too complex.Three simulated agents—user, assistant, and tool—role-play conversations involving single, parallel, dependent, or non-tool-use cases.
- Data complexity evaluation: The target model’s loss HM(x, y) measures data complexity, with higher loss indicating samples that are harder for that model to learn.Here x is the input query and y is the tokenized response.
- Data complexity evaluation: Loss is generally positively correlated with candidate-API count, APIs utilized, and dissimilarity between user queries and API descriptions.These factors correspond to more difficult selection, greater query complexity, and more demanding reasoning.
- Dual-Layer Validation: Dual-Layer Validation combines rule verification and model verification, with the results overseen by human experts.The system exploits the strict format requirements of executable function calls to verify synthesized data.
3 EXPERIMENT
ToolACE is evaluated through benchmark comparisons, ablations, scaling studies, backbone comparisons, and tests of broader model capabilities. The experiments examine function-calling performance, data complexity and diversity, verification, scalability, and generalization.
- Overall Performance: ToolACE-8B outperforms most API-based and open-source models in BFCL AST and Exec categories.It also maintains substantial advantages over all open-source models on API-Bank.
- Overall Performance: ToolACE-8B shows comparable API-Bank performance with GPT-4-series models while outperforming all open-source models.The API-Bank comparison is presented as evidence for the effectiveness of the synthesized training data.
- Overall Performance: 85.37% relevance and 83.81% irrelevance scores show ToolACE-8B’s balance between detecting relevant and irrelevant function calls.The model consistently outperforms similarly sized xLAM-7b-fc-r across all reported categories.
- Ablation on Complexity: ToolACEmedium achieves slightly higher overall and tool-use accuracy than ToolACEeasy and ToolACEhard on BFCL.The result supports using an intermediate complexity level rather than data that is too simple or overly complex.
- Ablation on Diversity: Training-data diversity positively correlates with overall model accuracy, with particularly pronounced gains in relevance detection.The study constructs low-, medium-, and high-diversity subsets by sampling APIs from 6, 14, and 30 context-tree clusters.
- Scaling and Backbones: Larger Qwen-1.5-xB-Chat models perform better on functional calling, and fine-tuning produces substantial gains across tested backbones.The backbone study evaluates Qwen1.5-7B-Chat, LLaMA-3-8B-Instruct, and LLaMA-3.1-8B-Instruct.
- General Capabilities: ToolACE-8B substantially improves over xLAM-7B-fc-r across most general-capability benchmarks, especially MMLU, GSM8K, and CommonSenseQA.Compared with raw LLaMA-3.1-8B-Instruct, it has negligible degradation on some benchmarks while improving functional calling.
4 RELATED WORK
Prior tool-learning methods either use tools without additional training or fine-tune models with tool-augmented data. Existing data-synthesis approaches often repurpose other-domain data or rely on public APIs, yielding limited interaction coverage.
- Tool Learning: Tuning-free methods provide in-context tool descriptions and examples, whereas tool-augmented tuning trains models to use external tools.The related-work taxonomy distinguishes methods by whether they require additional model training.
- Data Synthesis: Existing tool-use datasets often rely on repurposed data or publicly available APIs and primarily produce single-turn instructions with basic interactions.These approaches provide less coverage of complex tool-use scenarios than ToolACE’s combined synthesis, dialogue, and verification approach.
- Data Synthesis: ToolACE combines tool synthesis, dialogue generation, and verification into a more comprehensive data-generation approach.This distinguishes it from methods that mainly modify existing data or depend on public APIs.
5 CONCLUSION
ToolACE is an automated pipeline for generating accurate, complex, and diverse synthetic APIs and function-calling dialogues. The paper concludes that models trained on this data can achieve strong tool-learning performance, while broader capability enhancement remains limited by model and corpus scale.
- Conclusion: ToolACE uses self-evolution synthesis and self-guided generation to curate accurate, complex, and diverse synthetic APIs and dialogues.The pipeline is presented as an automated approach for enhancing LLM function-calling capabilities.
- Conclusion: ToolACE demonstrates state-of-the-art performance with smaller models trained on its synthesized data.The conclusion frames the results as advancing tool-augmented AI agents and setting new benchmarks for tool learning.
- Conclusion: The constructed API context tree represents a subtree of APIs in the Entertainment domain.The figure provides an example of the hierarchical organization used in API construction.
- Conclusion: ToolACE’s rule checker evaluates API definition clarity, function-call executability, dialogue correctness, and data-sample consistency.These four aspects define the documented rule-checking scope.
C.1 BENCHMARKS
BFCL evaluates function calling across diverse languages, domains, and complex scenarios, while API-Bank tests planning, retrieval, and API calling. The section also illustrates parallel, multiple, and relevance-focused function-calling cases.
- BFCL: BFCL contains 4,951 test cases spanning single-turn and multi-turn function-calling scenarios, including multiple, parallel, multi-turn, and multi-step calls.It includes 3,951 single-turn and 1,000 multi-turn cases focused on dynamic, real-world scenarios.
- BFCL: BFCL evaluates function calls through AST, executable-function, irrelevance, relevance, and overall-accuracy metrics.AST evaluation checks functions and parameters; executable evaluation compares executed outputs; overall accuracy averages sub-category accuracies.
- API-Bank: API-Bank contains 314 tool-use dialogues with 753 API calls and assesses calling, retrieval-plus-calling, and planning-plus-retrieval-plus-calling.Its data include 363 single calls and 122 multiple calls, with accuracy calculated as correct predictions divided by total predictions.
- Illustrative cases: Parallel-calling examples train parameter extraction and repeated correct-function invocation for multiple requested events.The illustrated query requests Theatre, Dance, and Music event information for separate dates.
- Illustrative cases: Multiple-calling examples train function discrimination by requiring selection of the correct tools for several information requests.One case selects live-event, NBA-statistics, and team-media functions from five available tools.
- Illustrative cases: Irrelevance-detection examples train the assistant to decline queries when available functions cannot address the requested task.One scenario contrasts cinema-showtime requests with tools for event details or book characters.
E.1 COMPARISON WITH OTHER TRAINING DATA
Compared with models trained on ToolLLM and xLAM data under matched data and backbone conditions, ToolACE training data produces consistently stronger BFCL performance across categories.
- Matched comparison: Using 25,000 samples and LLaMA-3.1-8B-Instruct for each comparison, ToolACE training data consistently outperforms ToolLLM and xLAM across BFCL categories.The comparison controls both training-data amount and base model before evaluation on BFCL.
- Observed weaknesses: xLAM-trained models show relatively poor irrelevance detection, likely because the dataset lacks diverse cases where available tools cannot solve the task.This explanation is presented as a likely reason for the observed weakness.
- Observed weaknesses: ToolLLM-trained models demonstrate weak BFCL generalization, consistent with their primary focus on multi-step and dependent cases.The reported comparison indicates that this narrower case emphasis does not transfer strongly across the benchmark.
E.2 ABLATION ON VARIOUS TYPES OF DATA
Ablations show that parallel and multi-type samples are especially important for concurrent tool use, irrelevance detection, and multi-turn behavior, whereas nested and dependent samples have smaller direct effects.
- Data-type ablations: Removing parallel-execution data significantly impairs concurrent invocation of multiple tools and lowers performance on Non-live AST and execution tasks.These tasks rely heavily on parallel tool usage.
- Data-type ablations: Removing multi-type samples reduces irrelevance-detection accuracy to 6.99% and impairs multi-turn function-calling behavior.In some multi-turn tests, models must ask clarifying questions instead of calling functions.
- Data-type ablations: Removing nested and dependent samples has a relatively minor effect on BFCL tool use because few tests require nested arguments and almost none require dependent usage.Including these data types nevertheless contributes to diversity and slight overall-performance improvements.
E.3 ABLATION ON COMPLEXITY EVALUATOR
The complexity-evaluator ablation compares self-evaluation with independent Qwen evaluators while training LLaMA-3.1-8B-Instruct, and reports more accurate guidance from the model being trained.
- Complexity evaluation: The self-guided complexity evaluation method uses the model being trained as its own evaluator, with an independent Qwen1.5-7B-Chat model used for validation.The independent evaluator was selected for comparable size and fairness.
- Complexity evaluation: The model-being-trained evaluator provides more accurate guidance than the independent evaluator and leads to improved results.The comparison is summarized in the complexity-evaluator ablation table.
G FINETUNING VS IN-CONTEXT LEARNING
ToolACE fine-tuning outperforms few-shot in-context learning on the reported BFCL comparisons. Fine-tuning also solves an example that few-shot prompting fails, while using fewer input tokens.
- Given 3 shots, LLaMA-3.1-8B-Instruct fails to generate correct arguments for a simple example.
- The fine-tuned model solves the example successfully in a zero-shot setting, whereas few-shot prompting consumes substantially more tokens.
- Few-shot in-context learning underperforms fine-tuning on BFCL and even falls short of the zero-shot setting.
H LIMITATIONS
The paper identifies limits in complexity-based sampling and general capabilities, while examples illustrate benchmark formats and zero-shot versus few-shot behavior. ToolACE remains specialized despite strong function-calling performance.
- Limitations: Complexity evaluation limits scalability as model size and training-sample count increase, and non-uniform sampling may bias learning toward easier examples.
- Limitations: ToolACE still lags behind GPT-4 in capabilities beyond functional calling, leaving simultaneous improvement across capabilities unresolved.
- Benchmark examples: BFCL and API-Bank examples identify the blue text as the portion evaluated.
- Examples: Figure 15 presents a zero-shot case solved by a model fine-tuned with ToolACE data.
- Examples: Figure 16 presents a few-shot case where the model hallucinates tool calls instead of correctly calling tools for the test sample.