PaperlayerView exact revision on arXiv

Source-linked AI summary

iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Use

Yirong Zeng, Xiao Ding, Yuxian Wang, Weiwen Liu, Wu Ning, Yutai Hou, Xu Huang, Duyu Tang, Dandan Tu, Bing Qin, Ting Liu

arXiv:2501.09766v5cs.CLcs.AIcs.LG

TL;DR

Synthetic tool-use data becomes less effective as it scales, particularly for complex scenarios where models often make localized parameter errors. iTool combines MCTS-based response-path exploration with iterative preference optimization to target those deficiencies. It improves performance over the base model and baseline, while the authors report computational and validation limits.

  • Problem

    Synthetic-data fine-tuning shows decaying gains in complex tool-use scenarios, often leaving localized parameter deficiencies in model responses.

  • Method

    iTool uses MCTS to diversify response paths, constructs fine-grained preference pairs, and iteratively applies preference optimization to correct wrong fragments.

  • Results

    iTool achieves approximately 13% better performance than the base model and a 6.5-point improvement over SFT on the Multi-turn metric.

  • Takeaways & Limitations

    Iterative fragment-level correction improves tool-use ability in complex scenarios and enables an 8B model to outperform larger open-source models while competing with top-tier closed-source models.

  • Takeaways & Limitations

    The method requires substantial computation, was not validated on 30B or 70B models, and was tested on only one synthetic tool-use dataset.

Abstract

from arXiv · show

Augmenting large language models (LLMs) with external tools is a promising approach to enhance their capabilities, especially for complex tasks. Synthesizing tool-use data through real-world simulations is an effective way to achieve this. However, our investigation reveals that training gains significantly decay as synthetic data increases. The model struggles to benefit from additional synthetic data, which fails to endow it with advanced tool-use capabilities in complex scenarios Moreover, we discovered that the above limitation usually manifests as a fragment deficiency (i.e., parameter errors) in response. To this end, we propose an iterative reinforced fine-tuning strategy designed to alleviate this limitation. This strategy involves: (1) enhancing the diversity of response for synthetic data through path exploration of Monte Carlo Tree Search. (2) iteratively pinpointing the model's deficiency by constructing fine-grained preference pairs, and then improving it by preference optimization algorithms for targeted improvement. The experiments show that our method achieves 13.11% better performance than the same-size base model. It achieves an improvement of 6.5% in complex scenarios compared to the baseline, and it also outperforms larger open-source and closed-source models.

1 Introduction

Tool-use models can handle simple scenarios but struggle in complex real-world applications, and additional synthetic data yields diminishing training gains. iTool addresses this limitation through diverse response exploration and iterative preference-based correction of response fragments.

  • Motivation: Complex applications remain difficult for LLMs despite prompt-based success on simple tool-use scenarios.The challenging settings include long contexts and extensive toolsets.
  • Motivation: Synthetic tool-use data produces diminishing training gains as its scale increases, especially in complex scenarios.The model struggles to benefit from additional synthetic data under supervised fine-tuning.
  • Problem diagnosis: The limitation commonly appears as a small response fragment containing parameter-name or parameter-value errors.The surrounding response can remain consistent with the ground truth while the parameter fragment differs.
  • Proposed approach: iTool combines MCTS-based path exploration for response diversity with iterative reinforced fine-tuning to correct wrong response fragments.It constructs fine-grained preference pairs and applies preference optimization, preceded by easy-to-hard warm-up SFT.
  • Contribution: iTool achieves approximately 13% better performance than the base model and improves tool use in complex scenarios while using an 8B-parameter model.The method also outperforms larger open-source models and competes with top-tier closed-source models.

2 Problem Statement and Analysis

The task requires selecting and executing tools for user queries, but synthetic-data fine-tuning exhibits diminishing gains as data increases. Analysis attributes the problem mainly to parameter errors concentrated in small response fragments, motivating targeted correction.

  • 2.1 Task Overview: Tool use maps a query and candidate tools to a sequence of tool-selection and execution actions.The model policy generates actions from the initial task state, query, and tool set.
  • 2.2 Preliminary Study: The evaluation spans simple Non-live and complex Live, Multi-turn, and Hallucination tool-use scenarios using BFCL.Non-live represents simple tools, whereas Live represents more complex settings such as multiple parallel tools.
  • 2.2 Preliminary Study: Performance gains decline significantly as synthetic training data increases, with improvement dropping after 30% of the data is used.The analysis identifies insufficient data diversity as one key factor in the decay.
  • 2.2 Preliminary Study: Bad cases are highly concentrated in Parameter Value and Parameter Name errors, which usually occupy only a small response fragment.Parameter Value errors mismatch the ground truth, while Parameter Name errors fail to identify a parameter from the query.
  • 2.2 Preliminary Study: The analysis motivates exploring diverse response paths with MCTS and iteratively optimizing the identified erroneous fragments.The proposed strategy targets fragment-level deficiencies to alleviate training-gain decay.

3 Method

iTool combines easy-to-hard warm-up training with iterative reinforcement learning that targets complex tool-use examples and fragment-level response deficiencies. It refreshes a replay buffer, uses MCTS to generate fine-grained preferences, and applies preference optimization to update the policy.

  • Overall architecture: iTool combines warm-up training with iterative reinforcement learning to improve tool use in complex scenarios.The method first performs warm-up training, then repeatedly samples difficult data, constructs preferences, and updates the policy.
  • Warm-up training: Easy-to-hard warm-up training divides data into easy, medium, and hard subsets using toolset size, toolset length, and required tool calls.The model is fine-tuned sequentially on the three difficulty levels.
  • Complex-data sampling: The replay buffer retains high-complexity examples by measuring generation perplexity and sampling the top 10% most uncertain data each iteration.Sample complexity is represented by generation perplexity, which reflects generation uncertainty.
  • MCTS-based preference generation: Step-wise MCTS converts response generation into discrete actions and uses PUCT-guided selection, expansion, rollouts, and backup to obtain step-level preferences.The search uses outcome correctness and self-evaluation in its reward, while fine-grained fragments can represent operations such as argument assignments.
  • MCTS-based preference generation: MCTS selects highest- and lowest-value steps as preferred and dispreferred responses, producing negative trajectories with fine-grained deficiencies for data diversification.The prefix path is treated as the question, and action values determine the preference pair.
  • Iterative preference optimization: SimPO updates the policy from MCTS-derived step-level preferences without a reference model, after which complex-data sampling and optimization repeat iteratively.The objective uses a length-normalized reward difference scaled by β, with γ enforcing a target margin between preferred and dispreferred responses.

4 Experiments

Experiments show that iTool improves tool-use performance across benchmarks, complex scenarios, base models, and dataset types, with gains tied to its warm-up and iterative reinforcement components.

  • 4.2 Overall Performance: 5.27 points: iTool-8B outperforms xLAM-8x22b-r in overall BFCL accuracy and surpasses most larger open-source and closed-source models in BFCL.It also shows consistent advantages on API-Bank and BFCL compared with comparable-scale open-source models.
  • 4.3.1 Module Ablation: 6.5 points: iTool improves over SFT on Multi-turn, while gaining 12.5 points over the Base model.Warm-up and iterative reinforcement contribute 2.3 and 4.2 points individually on Multi-turn.
  • 4.3.1 Module Ablation: Easy-to-hard warm-up training produces gradual gains, with notable improvements from Base to easy and from medium to hard.The results support a curriculum process in which easier tool-use patterns benefit harder scenarios.
  • 4.3.2 Deeper Ablation: 3 iterations: the model performs best around this count, especially on Multi-turn and Live, whereas replacing MCTS reduces performance.Excessive iterations can cause overfitting and reduce data diversity.
  • 4.3.3 Base Model Analysis: 18%: iTool improves over the base model on Llama-3.2-3B, while reaching 63.22% on Qwen2.5-7B.The method maintains stable performance across the evaluated base models.
  • 4.3.4 Training Gain Analysis: 30%: SFT training benefits flatten beyond this data proportion, while iTool's Live gains continue rising more steeply as data scale increases.Across datasets, gains are larger on synthetic data (+4.42 to +6.49) than on non-synthetic data (+2.17 to +3.65).

5 Related Work

Related work advances tool use through prompting, execution frameworks, synthetic-data tuning, and preference optimization. iTool builds on these directions by combining synthetic tool-use training with iterative reinforcement learning and preference-based updates.

  • Tuning-free Tool Use: Prompt manipulation and execution frameworks, including ReAct and RestGPT, seek to unlock inherent tool-use capabilities without tuning.These approaches are limited by the user-defined tools available in prompts.
  • Tuning-based Tool Use: Synthetic-data tuning methods expand toolsets and improve data synthesis efficiency, including ToolLlama, ToolACE, BUTTON, and xLAM.These methods motivate studying how synthetic data scaling affects tool-use performance.
  • Reinforcement Learning: Reinforcement learning aligns language models with human intentions through reward signals, while DPO simplifies offline preference optimization and improves training stability.Later work extends preference optimization to iterative settings by updating policies or regenerating preference pairs.

6 Conclusion

The paper studies advanced tool use with synthetic data, identifies training decay, and proposes iterative reinforced fine-tuning to correct response deficiencies through preference optimization.

  • The proposed iterative reinforced fine-tuning strategy continuously pinpoints wrong response fragments and addresses these deficiencies through preference optimization.

7 Limitation

The study’s approach remains constrained by computational demands, limited validation on larger models, and evaluation on only one synthetic tool-use dataset.

  • Monte Carlo Tree Search requires substantial computational resources to generate fine-grained preference data.The authors report approximately 7 hours on 8 V100 GPUs per iteration after applying parameter constraints.
  • The method was not validated on larger 30B or 70B base models because of limited computing resources.
  • The synthetic tool-use analysis tested only a single dataset, so additional public datasets are needed to strengthen the conclusions.

A.1 Descriptions of error types

The appendix categorizes tool-use errors into parameter, tool-count, tool-name, code-syntax, and other error types.

  • Parameter errors include incorrect values or types, unrecognized parameter values, and incorrect parameter counts or missing required parameters.
  • Tool-use errors also include calling the wrong number of tools or an incorrect or nonexistent tool name.
  • Code syntax errors occur when a tool call violates Python, Java, or JavaScript syntax.
  • The category “Other” covers errors not included in the listed error types.

B.1 More Implementation Details

Implementation uses parameter-efficient fine-tuning, specified training settings, and bounded MCTS search with confidence-based preference-data filtering.

  • The SFT warm-up uses LoRA with rank=16 and alpha=32, while the experiments use LLaMA-Factory and select the best validation checkpoint.Training can be completed within 28 hours on 8 NVIDIA Tesla V100-SXM2-32GB GPUs.
  • MCTS confidence scores average four policy-model response samples using weights 1.0, 0.1, -1.0, and -2.0 for options A, B, C, and D.
  • Preference pairs are filtered by candidate-step similarity, Q-value difference, and accepted-sample quality thresholds.Excluded pairs exceed 95% similarity, have Q-value differences below 0.1, or have accepted-sample Q-values below 0.3.
  • MCTS overhead is controlled by limiting tree depth, child-node width, simulation steps, and related search parameters.

B.2 Cost Analysis

The cost analysis finds that iTool improves BFCL-Live and Multi-turn accuracy over SFT at increased computational cost, while preference-optimization experiments identify SimDPO as the strongest variant.

  • B.2 Cost Analysis: 3.30% higher BFCL-Live accuracy and 6.46% higher Multi-turn accuracy over SFT came with a 2.8× increase in time cost.The analysis used BFCL-Live and Multi-turn accuracy alongside time costs on an 8×32G V100 configuration.
  • B.2 Cost Analysis: The gains in complex Multi-turn scenarios, where complexity is highest, support favorable cost-effectiveness for practical deployment.
  • B.2 Cost Analysis: The controlled SFT study tested Llama3.2-3B-Instruct, Llama3.1-8B-Instruct, and Qwen2.5-32B-Instruct with incrementally scaled training subsets on BFCL-Live.Figure 9 summarizes the study’s focus on whether training-gain dynamics generalize across model sizes.
  • B.2 Cost Analysis: Preference optimization comparisons included DPO, SimPO, IPO, and ORPO, initialized from the same SFT checkpoints for fair comparison.The methods differ in reward parameterization, reference-model use, or joint SFT training objectives.
  • B.2 Cost Analysis: SimDPO achieved the best performance, while preference algorithms showed no significant gaps except for ORPO.The comparison used optimally searched hyperparameter settings, with results shown in Figure 10.

C Case Analysis

The case analyses show how MCTS-derived preference pairs distinguish correct parameter handling from formatting errors and hallucinated values in tool-use responses.

  • C Case Analysis: One preference pair contrasts a correctly parameterized “Get Trending Result” call with a response that is improperly formatted and assigns timeframe the value 1.The rejected response also omits necessary parentheses, illustrating multiple tool-call errors.
  • C Case Analysis: Another MCTS-sampled pair rewards identifying missing reviews and metrics rather than hallucinating values for the requested functions.The query lacks details required by reviewAnalytics.extractSentiment and social-Trends.fetchTrendingProducts.
  • C Case Analysis: The case-analysis materials include a hyperparameter-search table and a self-evaluation prompt associated with Eq. 7.
Loading 2501.09766v5…