PaperlayerView exact revision on arXiv

Source-linked AI summary

LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls

Kangning Zhang, Wenxiang Jiao, Kounianhua Du, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu

arXiv:2511.09148v2cs.CLcs.AIcs.LG

TL;DR

Static, costly, and error-prone tool-use data pipelines leave training data disconnected from a model’s changing needs. LoopTool closes this loop by diagnosing weaknesses, refining labels, and expanding challenging samples. Its 8B model surpasses its 32B generator and achieves state-of-the-art results among similarly sized models on BFCL-v3 and ACEBench.

  • Problem

    Static tool-use data pipelines separate generation from training, while costly closed-source systems and noisy annotations constrain the quality and practicality of data synthesis.

    Source: §1 · pp. 1–2

  • Method

    LoopTool iteratively probes model capabilities, uses an open-source judge to correct labels, and expands verified failures into diverse challenging samples.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    LoopTool’s 8B model surpasses its 32B generator and achieves state-of-the-art performance among 8B-scale open-source models on BFCL-v3 and ACEBench; its BFCL-v3 overall accuracy is 74.93%.

    Source: §1 · p. 2, §5.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    The results support iterative, model-aware data evolution as a way to amplify tool-use performance beyond the model generating and evaluating its data.

    Source: §1 · p. 2, §6 · p. 10

    LoopTool is an offline iterative framework, so training-data evolution cannot occur concurrently with model training.

    Source: §6 · p. 10

Abstract

from arXiv · show

Augmenting Large Language Models (LLMs) with external tools enables them to execute complex, multi-step tasks. However, tool learning is hampered by the static synthetic data pipelines where data generation and model training are executed as two separate, non-interactive processes. This approach fails to adaptively focus on a model's specific weaknesses and allows noisy labels to persist, degrading training efficiency. We introduce LoopTool, a fully automated, model-aware data evolution framework that closes this loop by tightly integrating data synthesis and model training. LoopTool iteratively refines both the data and the model through three synergistic modules: (1) Greedy Capability Probing (GCP) diagnoses the model's mastered and failed capabilities; (2) Judgement-Guided Label Verification (JGLV) uses an open-source judge model to find and correct annotation errors, progressively purifying the dataset; and (3) Error-Driven Data Expansion (EDDE) generates new, challenging samples based on identified failures. This closed-loop process operates within a cost-effective, open-source ecosystem, eliminating dependence on expensive closed-source APIs. Experiments show that our 8B model trained with LoopTool significantly surpasses its 32B data generator and achieves new state-of-the-art results on the BFCL-v3 and ACEBench benchmarks for its scale. Our work demonstrates that closed-loop, self-refining data pipelines can dramatically enhance the tool-use capabilities of LLMs.

1 INTRODUCTION

Static tool-use data pipelines separate generation from training, leaving data misaligned with evolving model capabilities and vulnerable to noisy labels. LoopTool closes this loop by diagnosing weaknesses, refining labels, and expanding failure cases.

1 INTRODUCTION

  • Static pipelines generate data before training, wasting capacity on mastered cases while leaving harder, underrepresented cases unresolved.The model does not guide data generation, creating a persistent mismatch between learning needs and fixed training data.Source: §1 · p. 1
  • LoopTool starts from a seed corpus and couples GRPO post-training with capability probing to identify mastered, borderline, and failed cases.The seed corpus combines synthesized tool specifications with multi-agent dialogue generation.Source: §1 · p. 2
  • JGLV uses an open-source judge to correct label errors, while EDDE turns verified failures into structurally similar, diverse training samples.The framework uses Qwen3-32B for generation and evaluation, avoiding dependence on costly closed-source APIs.Source: §1 · p. 2
  • LoopTool’s 8B model surpasses its 32B generator and achieves state-of-the-art performance among similarly sized models on BFCL-v3 and ACEBench.The framework uses an open-source, self-contained data generation and refinement process.Source: §1 · p. 2

2 RELATED WORK

Related work covers tool-augmented LLMs, synthetic tool-use data, and reinforcement learning for tool-use optimization. LoopTool combines these lines in an iterative process guided by model diagnostics and error correction.

Tool-Augmented Large Language Models

  • Tool-augmented LLM research spans API invocation, knowledge-base interaction, code execution, and multimodal processing, with benchmarks evaluating varied tool-use capabilities.BFCL and ACEBench assess areas including tool selection, argument generation, multi-step reasoning, and multi-turn calling.Source: §2 · pp. 2–3

Synthetic Data Generation for Tool Use

Reinforcement Learning for Tool-Use Optimization

  • LoopTool embeds GRPO in an interleaved train–generate loop, exposing the model to prior failures and progressively refined supervision.Related work describes GRPO as showing strong performance in tool-use settings (Zhang et al., 2025).Source: §2 · p. 3

3 AUTOMATED TOOL-AUGMENTED DIALOGUE CONSTRUCTION

The data-construction stage builds a diverse seed dataset from curated and synthetic APIs and simulated tool-use dialogues. It combines hierarchical API synthesis with multi-agent generation and two-stage verification.

3 AUTOMATED TOOL-AUGMENTED DIALOGUE CONSTRUCTION

  • The initial construction stage creates a diverse, high-quality seed dataset by combining curated APIs with simulated multi-agent conversations.The paper positions this stage as a foundation for later iterations rather than its core innovation.Source: §3 · p. 3

3.1 HIERARCHICAL DUAL-TREE GUIDED API SYNTHESIS

  • The Hierarchical Dual-Tree method synthesizes APIs by combining a sampled topical path from the Context Tree with structural constraints from the Constraint Tree.Rule-based validation then checks conformity and semantic coherence.Source: §3.1 · pp. 3–4

3.2 MULTI-AGENT TOOL-USE DIALOG GENERATION

  • Four simulated roles—Planner, User, Assistant, and Tool—produce turn-by-turn dialogues, with real executable backends used in some domains.Generated dialogues pass rule-based checks of syntax, parameters, types, and schemas, followed by Qwen3-32B evaluation of contextual appropriateness, logical consistency, and user-intent alignment.Source: §3.2 · p. 4

4 ITERATIVE MODEL TRAINING AND DATA AUGMENT

LoopTool couples model training with iterative data refinement, using capability diagnosis, label verification, and failure-driven expansion to adapt supervision to model weaknesses.

4 ITERATIVE MODEL TRAINING AND DATA AUGMENT

  • LoopTool integrates GRPO training, Greedy Capability Probing, Judgement-Guided Label Verification, and Error-Driven Data Expansion into a closed loop that targets weaknesses and refines supervision.The cycle repeatedly assesses model capabilities and updates the training data.Source: §4 · p. 4

4.1 GRPO TRAINING FOR TOOL CALLING

  • GRPO training represents each tool-call turn with available tools, dialogue context, and a reference call, then optimizes the policy using rewards for candidate responses.The training format supports multi-turn conversations with sequential tool calls; the reward is binary and rule-based.Source: §4.1 · pp. 4–5

4.2 GREEDY CAPABILITY PROBING

  • Greedy Capability Probing decodes each training instance, provisionally marks matching calls as mastered, and sends mismatches to label verification.It also retains selected high-perplexity cases, which indicate low confidence and proximity to a decision boundary.Source: §4.2 · p. 5

4.3 JUDGEMENT-GUIDED LABEL VERIFICATION

  • JGLV uses a judge to distinguish model errors from annotation errors, replacing incorrect labels with model predictions and discarding cases where both calls are wrong.For cases judged both correct, only high-perplexity examples are retained for the next round.Source: §4.3 · p. 5

4.4 ERROR-DRIVEN DATA EXPANSION

  • EDDE turns verified failures into structurally similar, diverse hard samples, preserving the core challenge while varying scenarios and filtering generated samples through rule-based and LLM-based evaluation.The next-round dataset merges identified sources and a small untrained subset of the initial seed data before GRPO training.Source: §4.4 · p. 6

5 EXPERIMENTS

Experiments evaluate LoopTool on BFCL-v3 and ACEBench, then examine its iterative data process, component ablations, model-size scaling, and non-tool generalization. Results show strong benchmark performance and gains from iterative refinement across the reported settings.

5.2 OVERALL PERFORMANCE ANALYSIS

  • LoopTool-8B reaches 74.93% overall accuracy on BFCL-v3, ranking third overall and beating Qwen3-8B by +8.59 points; it scores 73.4% on ACEBench, +6.3 over Qwen3-8B.It is among 8B-scale open-source leaders on both benchmarks and surpasses the 32B Qwen3 model used as generator and judge.Source: Table 1 · p. 6, Table 2 · p. 7, §5.2 · p. 7

5.1 EXPERIMENT SETUP

  • Evaluation uses Qwen3-8B under pure RL fine-tuning on BFCL-v3 and ACEBench, with official evaluation scripts and average accuracy across categories.Ablations examine individual modules; implementation uses GRPO, with details reported in the setup passage.Source: §5.1 · pp. 6–7

5.3.1 ITERATIVE DATASET DISTRIBUTION

  • Across iterations, the dataset gradually reduces untrained seed samples while adding newly generalized queries and increasingly challenging examples.The initial seed dataset contains 28k tool-call samples.Source: §5.3.1 · pp. 7–8, Table 3 · p. 7

5.3.2 PERFORMANCE ANALYSIS OF ITERATIVE TRAINING FRAMEWORK

  • Iterative training yields consistent BFCL-v3 accuracy gains, whereas seed-only training improves less, plateaus by Iteration 2, and declines after Iteration 3.The authors attribute the static-data decline to overfitting and growing mismatch with the model’s evolving inference behavior.Source: §5.3.2 · p. 8, Figure 2 · p. 8

5.4 ABLATION STUDY

  • Ablations show that removing high-PPL samples lowers accuracy, especially in Multi-Turn cases, while skipped label verification degrades results and EDDE variants improve mastery of original hard cases.Replacing EDDE samples with high-PPL cases sustains performance close to the full configuration; retraining on original error seeds alone is insufficient.Source: Table 4 · p. 8, §5.4 · p. 9, Figure 3 · p. 9

5.5 SCALING PERFORMANCE WITH MODEL SIZE

  • The 8B model gains +1.80 BFCL-v3 points in Iteration 2, compared with +0.70 for the 0.6B model.Larger backbones score higher initially and after refinement; the authors relate this trend to discovering correct tool-use trajectories during GRPO rollout exploration.Source: §5.5 · p. 9, Figure 4 · p. 9

5.6 GENERALIZATION ABILITY EVALUATION

  • LoopTool-8B matches or surpasses vanilla Qwen3-8B across six non-tool benchmarks, including gains of +1.40 on IFEval, +3.84 on LiveCodeBench, and +1.20 on Math-500.The reported results also show gains on both AIME sets.Source: Table 5 · p. 9, §5.6 · p. 10

6 CONCLUSION AND LIMITATION

LoopTool combines data synthesis, label refinement, and GRPO-based post-training in a model-aware closed loop using an open-source model for generation and judgment. The framework remains offline and serial, and the authors identify online or parallelized variants as future work.

6 CONCLUSION AND LIMITATION

  • LoopTool evolves data between training rounds rather than concurrently, and each round waits for the previous one to finish.The authors identify online or streaming variants and parallelized iteration schemes as future work.Source: §6 · p. 10

A THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • The authors used an open-source language model as judge and data generator, while general-purpose LLMs were used only for English-language editing.The authors retain responsibility for the paper’s concepts, experiments, analyses, and claims.Source: A THE USE OF LARGE LANGUAGE MODELS… · p. 14

B.1 BENCHMARKS

The appendix describes the evaluated benchmarks, points to the iterative GRPO configuration, and identifies the LoopTool algorithm and hierarchical subtree examples.

B.1 BENCHMARKS

  • BFCL-v3 covers function-calling tasks across languages and domains, using AST, execution, multi-turn state and response, irrelevance, and relevance metrics.The benchmark includes 4,951 test instances: 3,951 single-turn and 1,000 multi-turn cases.Source: §B.1 · p. 14
  • ACEBench divides tool-use evaluation into Normal, Special, and Agent categories, covering basic use, ambiguous or incomplete instructions, and multi-agent interactions.Its described measures include AST-based normal evaluation, problem identification for Special, and end-to-end and process accuracy for Agent.Source: §B.1 · p. 15

B.2 HYPER-PARAMETERS

  • Table 6 presents the configuration for iterative GRPO training.Source: Table 6 · p. 15

C THE ALGORITHM OF LOOPTOOL

D THE EXAMPLE OF HIERARCHICAL DUAL SUBTREES

E THE TRAINING SAMPLE FOR GRPO

The appendix illustrates GRPO training samples and the prompt used for label verification. Its procedure trains a policy, probes its capabilities, verifies mismatches, expands error seeds, and updates the next-round dataset.

E THE TRAINING SAMPLE FOR GRPO

  • GRPO trains the policy on the current dataset, after which GCP probes seed cases and retains high-PPL samples among correctly predicted cases.The predicted calls are sent to JGLV for evaluation.Source: E THE TRAINING SAMPLE FOR GRPO · p. 16
  • JGLV obtains a judgment from Qwen3-32B for mismatched cases, distinguishing prediction errors, label errors, cases where both are correct, and cases where both are wrong.Source: E THE TRAINING SAMPLE FOR GRPO · p. 16
  • EDDE constructs an error-seed set, generates k diversified samples per seed, and validates the generated set with rule-based and LLM-based evaluation.Source: E THE TRAINING SAMPLE FOR GRPO · p. 16
  • The next iteration selects an untrained seed subset and constructs its dataset using the update procedure in Equation 5.The appendix also presents the GRPO instruction prompt and single-turn and multi-turn sample examples.Source: E THE TRAINING SAMPLE FOR GRPO · pp. 15–16

F THE LABEL VERIFICATION PROMPT

G THE ERROR GENERATION PROMPT AND NEW ERROR SAMPLES

The appendix shows EDDE prompts and a generated sample, alongside examples of trees, GRPO samples, and JGLV materials.

G THE ERROR GENERATION PROMPT AND NEW ERROR SAMPLES

H THE LEARNING CURVES IN ITERATIVE LEARNING

The Prompt For Judgement-Guided Label Verification (JGLV)

  • Figure 10 shows the JGLV prompt for the judgement model, with red text marking placeholders.Source: Figure 10 · p. 18

The PRED_WRONG Example Identified by JGLV

The REF_WRONG Example Identified by JGLV

The Generated Single-Turn GRPO Sample from EDDE

Figure 15 depicts an EDDE sample generated according to an error in the model response.

The Generated Single-Turn GRPO Sample from EDDE

  • Figure 15 shows a new sample EDDE generated according to an error in the model response.Source: Figure 15 · p. 20

Related papers

Loading 2511.09148v2…