Source-linked AI summary
ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedback
Min Zeng, Yuzhou Liu, Zhenyu Cao, Hanxiu Chen, Heng Li, Caiquan Liu, Yafei Wen, Xiaoxin Chen
TL;DR
Tool-use data synthesis is difficult because existing generate-then-filter methods provide limited quality control and can produce inefficient, imbalanced data. ToolLoop addresses this with three-stage generation and dynamic self-feedback, achieving strong BFCL performance with 11K examples and generalizing to ACEBench. The paper’s evaluation remains bounded by static benchmarks and verifier-related limitations.
Problem
Existing synthetic tool-use methods rely on one-shot generation followed by filtering, while high-quality data linking user instructions to tool selection remains scarce.
Method
ToolLoop decomposes synthesis into function-name ground truths, backward-derived user queries, and forward-generated tool calls, with dynamic self-feedback at each stage.
Results
ToolLoop achieves 86.40% BFCL accuracy with 11K examples, while its overlap-filtered Isolate variant reaches 86.07% and the method generalizes to ACEBench.
Takeaways & Limitations
The results support constructing tool-use data through iterative generate-verify-refine rather than generating a single query–answer pair and filtering only at the end.
Takeaways & Limitations
Static evaluation provides no real-environment feedback for timeout errors, malformed API responses, cascading failures, or execution-driven iterative refinement.
Abstract
from arXiv · showhide
High-quality tool-use data is critical for training language models to interact effectively with external tools. However, existing synthetic approaches typically follow a generate-then-filter paradigm with static post-hoc verification, often yielding inefficient data with imbalanced feature distributions. We propose ToolLoop, a closed-loop framework that decomposes synthesis into three progressive stages: (1) sampling function name combinations as ground truth; (2) backward derivation of user queries; and (3) forward derivation of tool calls. At each stage, dynamic self-feedback iteratively guides the model toward high-quality generation, realizing a transition from generate-then-filter to generate-verify-refine. On the Berkeley Function Calling Leaderboard (BFCL), a 4B parameter model trained with our 11K synthetic examples achieves 86.40% accuracy in non-reasoning mode, while an Isolate variant that removes BFCL-overlapping candidate functions still reaches 86.07\%. Cross-benchmark evaluation on ACEBench further demonstrates strong generalization, with 72.1% overall accuracy using only 18.3% of baseline training data.
1 Introduction
ToolLoop addresses scarce high-quality tool-use data by replacing one-shot generate-then-filter synthesis with decomposed generation and dynamic self-feedback. The framework is evaluated on BFCL and ACEBench, using a compact synthetic dataset to achieve strong downstream performance.
- Tool-use datasets remain scarce because they must accurately connect user instructions with appropriate tool selections.
- ToolLoop decomposes synthesis into three stages and uses dynamic self-feedback to guide iterative optimization.The framework generates function combinations, derives user queries, and instantiates tool calls.
- ToolLoop is evaluated comprehensively on BFCL and ACEBench to assess downstream tool-use performance.
- 11K synthetic examples provide the basis for ToolLoop’s practical evaluation across four representative function-calling scenarios.
2 Related Work
Prior work advances synthetic instruction and tool-use data through generation, filtering, refinement, and automatic evaluation. ToolLoop is situated within this broader progression toward scalable synthetic data and LLM-based assessment.
- Synthetic-data research progressed from instruction generation and filtering toward more diverse task and agent-data synthesis.Examples include Self-Instruct, Stanford Alpaca, Persona Hub, SynthAgent, and APIGen-MT.
- APIGen automated generation of verifiable function-calling data, while later methods extended synthesis to multi-turn agent trajectories.
- LLM-based assessment is presented as a scalable alternative to traditional evaluation protocols in many scenarios.
3 Methods
ToolLoop constructs tool-use data through scenario-aware function sampling, backward query derivation, and forward tool-call generation. Dynamic, stage-specific validation repairs errors before samples advance or are discarded.
- Scope of Category: ToolLoop covers Simple, Multiple, Parallel, and Parallel Multiple scenarios spanning single-function, selection, and parallel invocation patterns.These categories range from one required function to multiple candidate functions invoked repeatedly in parallel.
- Function Sampling: Candidate functions are sampled by embedding function descriptions, clustering them with K-means, and using an LLM to select semantically related functions.
- Three-Stage Generation: Synthesis proceeds through ground truth generation, user query derivation, and tool-call instantiation.Each stage uses an intermediate representation before the next stage begins.
- Ground Truth Generation: Ground truths enforce parallelizability, coherence, and scenario-specific invocation cardinality.Parallel and Parallel Multiple cases contain 2–4 invocations, while Simple and Multiple cases contain one target invocation.
- Query Derivation: Backward query derivation enforces exact function-call requirements, signature-consistent parameters, completeness, and natural conversational phrasing.
- Tool-Call Instantiation: Forward generation produces structured tool calls that satisfy function schemas and accurately fulfill the query.The representation follows the OpenAI function-calling format.
- Dynamic Self-Feedback: Validation combines semantic LLM verification, deterministic rules, and AST parsing, with stage-specific feedback used for targeted regeneration.Failed samples receive actionable refinement prompts and may be retried up to three times before discard.
4 Experiments
ToolLoop is evaluated on BFCL and ACEBench using a 4B base model under non-reasoning settings, with comparisons spanning baselines, ablations, and leakage control. It achieves strong BFCL performance with 11K examples and generalizes to ACEBench, while showing weaker personalized-selection performance.
- BFCL Results: 86.40% overall BFCL accuracy is achieved by ToolLoop-4B with 11K synthetic examples, exceeding APIGen-4B by 3.29 points and ToolMind-4B by 2.87 points.ToolLoop-4B-Isolate reaches 86.07% after removing BFCL-overlapping candidate functions.
- BFCL Results: 91.29% non-live accuracy surpasses APIGen-4B’s 89.90%, with ToolLoop reaching 96.50% on Multiple and 94.50% on Parallel_Multiple.ToolLoop-4B-Isolate reaches 97.00% on Multiple.
- BFCL Results: 81.50% live accuracy exceeds APIGen-4B’s 76.31% and ToolMind-4B’s 77.57%, although Live-Parallel_Multiple favors APIGen-4B by 79.17% to 83.33% on a 24-instance split.The authors identify high sampling uncertainty for this small split and avoid a strong category-level conclusion.
- Ablation Study: ToolLoop’s ablation reaches 86.40% overall, compared with 82.56% for final filtering and 79.97% without feedback under matched training settings.The comparison isolates iterative verifier-guided refinement from discarding invalid outputs.
- ACEBench Generalization: 72.1% overall ACEBench accuracy surpasses APIGen-4B by 5.1 points while using 18.3% of its training data, and surpasses ToolMind-4B by 1.9 points using one fifth of its data.ToolLoop also leads on Atom at 84.0% and Similar API at 78.0%.
- ACEBench Generalization: ACEBench Profile remains a limitation: all fine-tuned models underperform the base model, although ToolLoop scores 60.0% versus 54.0% for APIGen-4B and ToolMind-4B.Single Turn is another improvement direction, where ToolLoop reaches 66.5% but trails ToolMind-4B’s 69.5%.
5 Analysis
The analysis examines refinement cost, retry patterns, discarded samples, and verifier reliability. It finds that extra computation is concentrated in a small subset of difficult intermediate outputs, especially backward query derivation, while automated verification remains useful but incomplete.
- Synthesis Efficiency: 18.1% of samples require at least one retry in Stage 2, the highest refinement demand among ToolLoop’s three synthesis stages.Samples requiring two or three retries account for 1.72% in Stage 1, 8.05% in Stage 2, and 4.25% in Stage 3.
- Synthesis Efficiency: 33.41M total tokens are consumed to synthesize 11,024 retained examples, with parallel examples accounting for the largest share of cost.The accounting includes 26.13M input tokens and 7.28M output tokens.
- Refinement Patterns: Stage 2 needs more retries because it translates symbolic function plans into natural-language queries, where fluent outputs can still omit parameters or misrepresent the target function set.Stage 3 has relatively few retries once the query aligns with the ground truth.
- Refinement Patterns: Backward query derivation is the main semantic bottleneck because mismatches between natural-language intent and function sequences are difficult for rule-based checks alone.ToolLoop addresses these inconsistencies before later tool-call generation stages.
- Refinement Patterns: 280 samples remain invalid after three retries, including 192 repeated semantic-verification failures, 83 deterministic-validation failures, and 5 failures of both types.Most discarded cases involve persistent semantic inconsistencies rather than isolated formatting errors.
- Verifier Reliability: 94% human agreement with Qwen-Max supports its use as a semantic verifier, but rule-based and AST checks remain necessary for JSON validity, argument names, data types, and schema conformance.The authors describe the manual annotation as an overall reliability check rather than a complete validation of every error category or stage.
6 Conclusion
ToolLoop synthesizes tool-use data through decomposed generation and dynamic self-feedback, shifting from generate-then-filter toward generate-verify-refine. Evaluations report strong BFCL performance and the best overall ACEBench score among data-centric methods, suggesting transfer beyond the primary benchmark.
- Results: 86.40% BFCL accuracy was achieved by a 4B model trained on only 11K ToolLoop examples in non-reasoning mode.The BFCL-overlap-filtered Isolate variant still reached 86.07%.
- Method: ToolLoop decomposes synthesis into ground truth generation, user query derivation, and tool call instantiation, with dynamic self-feedback guiding refinement.This implements a generate-verify-refine process rather than single-pass generation followed by filtering.
- Results: ToolLoop achieved the best overall ACEBench score among data-centric methods, suggesting that synthesized data can transfer beyond the primary benchmark.
- Takeaway: The reported value of ToolLoop lies in improving internal consistency across synthetic examples rather than simply scaling supervision.The framework jointly verifies intended function sequences, derived user intent, and executable tool calls.
Limitations
The evaluation does not include real executable tool environments, limiting verification of how well the synthesized data prepares models for practical tool-use failures and execution-driven refinement. The verifier design also leaves open the possibility of correlated semantic biases.
- Environment scope: Without real environment feedback, the study cannot verify preparedness for timeout errors, malformed API responses, or cascading ground-truth failures.Future work is directed toward interactive testbeds with executable tool environments.
- Evaluation scope: Static evaluation protocols do not assess iterative refinement based on execution results, a capability common in real-world tool use.
- Verifier scope: Qwen-Max serves as the semantic verifier across all three synthesis stages, so verifier-specific or correlated semantic biases are not ruled out.The reported 94% agreement with human annotations does not eliminate this limitation.
Ethics Statement
The study focuses on synthetic tool-use data generation and uses publicly available API specifications and benchmarks without collecting personally identifiable information or conducting user studies.
- Scope: The work focuses on synthetic data generation for tool use in language models.
- Data and participants: The study does not collect personally identifiable information or conduct user studies.
- Materials: All experiments use publicly available API specifications and benchmarks.
A The distribution of synthetic data
The ToolLoop dataset contains 11,024 synthetic examples distributed across four query categories. Simple and parallel instances make up most of the dataset, while multiple and parallel-multiple instances form smaller portions.
- Distribution: 11,024 total examples are distributed across simple, parallel, multiple, and parallel-multiple query categories.
- Distribution: 40.4% of examples are simple and 33.0% are parallel.These categories contain 4,453 and 3,634 examples, respectively.
- Distribution: 16.2% of examples are multiple and 10.5% are parallel-multiple.These categories contain 1,783 and 1,154 examples, respectively.
B Prompt Design
ToolLoop’s prompt design operationalizes decomposed generation through three stages: parallel ground-truth sampling, backward user-query derivation, and forward function-calling data generation. Each stage uses dynamic self-feedback to support iterative refinement.
- Three-stage prompt design: The prompt templates cover Stage 1 ground-truth sampling, Stage 2 backward user-query derivation, and Stage 3 forward tool-call derivation.Figures 5, 6, and 7 provide the corresponding stage-specific templates.
- Dynamic self-feedback: Each stage incorporates a History of Feedback section that guides iterative refinement through dynamic self-feedback.The feedback mechanism is included in the complete prompts to support reproducible implementation of the decomposed process.
- Stage 1: Ground Truth Sampling: Stage 1 selects functions that can be invoked simultaneously to satisfy a hypothetical user request.The setting is single-turn parallel calling, with no tool responses available at call time.
- Stage 2: Backward Derivation of User Query: Stage 2 generates a natural user query requiring the specified function sequence.This stage performs reverse reasoning from the intended function sequence to the user request.
- Stage 3: Forward Derivation of Tool Calls: Stage 3 generates OpenAI standard-format function-calling data from the user query and available functions.The prompt frames this as forward generation of function-calling data.