Source-linked AI summary
Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning
Yifei Chen, Guanting Dong, Zhicheng Dou
TL;DR
TIR models can use external tools but often call them inefficiently or overthink after tool results, making efficient and accurate reasoning difficult. Tool-Light analyzes entropy changes to guide self-evolved data construction and combines SFT with self-evolved DPO. Across 10 datasets, it improves overall TIR performance, tool-call effectiveness, and inference simplicity.
Problem
TIR models often make excessive or insufficient tool calls and overthink after low-quality tool results, leaving efficient and accurate tool use unresolved.
Method
Tool-Light uses entropy-guided and vanilla sampling to construct preference data, followed by SFT and self-evolved DPO training.
Results
Across 10 challenging reasoning datasets, Tool-Light achieves strong overall performance while improving tool-call effectiveness and inference simplicity.
Takeaways & Limitations
Tool-Light provides a framework for efficient and accurate multi-tool reasoning supported by entropy analysis, self-evolved data, and two-stage training.
Abstract
from arXiv · showhide
Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to improve their internal reasoning ability by integrating external tools. However, models employing TIR often display suboptimal behaviors, such as insufficient or excessive tool usage and overthinking after tool calls. The challenge of incentivizing LLMs to perform TIR efficiently and accurately, while stabilizing the reasoning process, remains an open question. In this paper, we start by exploring the impact of tool calls on model reasoning from the perspective of information entropy. Our findings indicate that tool call results lead to a distinct change in the information entropy of subsequent reasoning, with the overall entropy of the reasoning chain varying based on the number of tool calls. Building on these insights, we propose Tool-Light, a framework designed to encourage LLMs to perform TIR efficiently and accurately. Our framework includes dataset construction and multi-stage fine-tuning. For dataset construction, we employ continuous self-evolved sampling using the fine-tuned model, integrating both vanilla sampling and entropy-guided sampling. Besides, we establish strict criteria for selecting positive-negative pairs during sampling. The training process involves a two-stage approach, comprising Supervised Fine-Tuning (SFT) and Self-Evolved Direct Preference Optimization (DPO). Experimental results on 10 datasets demonstrate the effectiveness of Tool-Light, significantly improving the model's efficiency in executing TIR tasks.
1 INTRODUCTION
Tool-Integrated Reasoning improves access to external capabilities but still suffers from inefficient tool use and unstable post-call reasoning. Tool-Light addresses these issues through entropy-guided self-evolved data construction and two-stage training, with improvements reported across 10 challenging datasets.
- Research challenge: TIR models commonly make excessive or insufficient tool calls and may overthink or enter analysis paralysis after low-quality tool results.These behaviors are collectively described as incorrect tool calls.
- Research gap: Existing multi-tool research often emphasizes tool overuse while neglecting tool underuse and the effect of tool results on subsequent reasoning.
- Entropy perspective: Tool-call results cause subsequent output entropy to fluctuate, and reasoning paths with fewer tool calls tend to have lower overall entropy when multiple correct paths exist.
- Proposed framework: Tool-Light combines entropy-guided and direct sampling with SFT and self-evolved DPO to improve TIR reasoning and tool-call effectiveness.Entropy-guided sampling branches from high-entropy positions, while the training pipeline includes Pre-Aligned DPO Training and Self-Evolved DPO Alignment.
- Experimental scope: 10 challenging reasoning datasets demonstrate Tool-Light's effectiveness, with improved efficiency and accuracy of tool calls while maintaining stable reasoning performance.
2 RELATED WORK
Prior TIR work uses fine-tuning, internal signals, metacognitive strategies, or reinforcement learning to guide tool use, while self-evolved methods improve models through generated data or adaptation. Tool-Light combines these directions around multi-tool reasoning and self-generated training data.
- Tool-Integrated Reasoning: TIR research applies fine-tuning, metacognitive strategies, internal signals, and reinforcement learning to improve models' tool use.
- Tool-Integrated Reasoning: Existing approaches include IKEA and SMART for knowledge-boundary guidance, Self-DC for internal behavioral signals, and Search Wisely and OTC for reinforcement-learning optimization.
- Self-Evolved Methods in LLMs: Self-evolution methods improve models through parameter-updating or in-context learning, including self-generated training data, environmental experience, prompts, and memory mechanisms.
- Self-Evolved Methods in LLMs: Tool-Light's self-evolution centers on continuously generating better training data for parameter updates.
3 PRELIMINARIES
The preliminaries define multi-tool TIR and examine its information-entropy dynamics using sampled reasoning chains. The pre-experiment finds characteristic entropy changes after tool results and lower entropy in chains using fewer tool calls.
- Problem Definition: Multi-TIR enables a model to autonomously call multiple tools while completing complex tasks from an input instruction.
- Pre-Experiment: The entropy experiment rolls out ten Search-R1 reasoning chains per sample, groups them by tool-call count, and records average entropy at each step.
- Pre-Experiment: After receiving a tool-call result, output entropy initially rises, fluctuates, and then drops sharply before the next tool call.
- Pre-Experiment: For the same sample, lower-entropy reasoning chains tend to use fewer tool calls, with the difference becoming more evident as reasoning progresses.
- Pre-Experiment: The entropy findings guide the framework's subsequent data-sampling strategy.
- Tool Design: The explored tools are a code interpreter, which returns compilation results or errors, and a search tool, which retrieves relevant local or web content.
4 METHOD
Tool-Light constructs TIR preference data with vanilla and entropy-guided sampling, then trains models through SFT and iterative self-evolved DPO alignment. Its sampling selects challenging trajectories and preference pairs that favor accurate reasoning with fewer tool calls and lower entropy.
- 4 METHOD: Tool-Light combines dataset construction with a two-stage TIR training pipeline consisting of SFT followed by pre-aligned and self-evolved DPO alignment.The self-evolved DPO stage repeatedly resamples data and retrains the model until performance converges.
- 4.1 DATASET CONSTRUCTION: The source dataset retains samples where the SFT model’s direct inference answer is incorrect, targeting examples considered more challenging for reasoning improvement.The model is trained on existing SFT data, then evaluated without external tools; only mismatched golden and inferred answers are retained.
- 4.1 DATASET CONSTRUCTION: Vanilla sampling generates multiple TIR paths per source question, while entropy-guided sampling branches from the top-k highest-entropy reasoning steps to increase output diversity.Entropy is computed over early token positions within each reasoning step, and the maximum average entropy identifies the sequence length and branching locations.
- 4.1 DATASET CONSTRUCTION: Entropy-guided sampling collects main and branch chains while reducing idealized sampling complexity from O(mn) to O(n log m).Here, m is the number of rollouts and n is the average sequence length.
- 4.2 TWO-STAGE TIR TRAINING PARADIGM: Preference construction favors correct trajectories with minimal tool calls and low entropy, pairing them against incorrect trajectories that use more tools or exhibit less efficient behavior.Vanilla criteria additionally separate hard and easy samples using ≤40% and ≥70% correct trajectories, with a 2:1 hard-to-easy ratio to reduce unnecessary tool calls and excessive reasoning.
- 4.2 TWO-STAGE TIR TRAINING PARADIGM: Pre-aligned DPO produces an intermediate model, which then resamples data under a second criterion for self-evolved DPO alignment and repeated iterative improvement.The second criterion distinguishes problems that remain difficult from those already mastered by the intermediate model.
5 EXPERIMENTS
Tool-Light is evaluated on 10 mathematical-reasoning and knowledge-intensive datasets using correctness, Efficiency, and Necessity metrics. Results show benefits from external tools and multi-round DPO, alongside improved tool-use efficiency, shorter reasoning sequences, and lower entropy.
- 5.2 MAIN RESULT: External tools substantially improve prompting-based mathematical reasoning, but provide smaller gains on knowledge-intensive tasks, improving only on MuSiQue.The results indicate that untrained models do not use external tools effectively.
- 5.2 MAIN RESULT: Multi-tool training generalizes better across mathematical and knowledge-intensive tasks than single-tool training, whose performance varies by task type.Search-R1 performs well on knowledge-intensive tasks but poorly on mathematical reasoning, whereas ToRL shows the opposite pattern.
- 5.2 MAIN RESULT: Tool-Light achieves strong cross-task performance, reaching optimal results on four mathematical-reasoning datasets and ranking among the top two across knowledge-intensive tasks.The framework also outperforms most GRPO-trained baselines in average performance when using DPO alone.
- 5.3 QUANTITATIVE ANALYSIS: Tool-Light achieves the best Efficiency and Necessity scores among compared methods, while using shorter output sequences than Tool-Star.It also outperforms Tool-Star in result correctness and reasoning effectiveness, indicating reduced overthinking.
- 5.3 QUANTITATIVE ANALYSIS: Tool-Light produces lower-entropy output distributions than existing TIR models, consistent with its entropy-guided sampling strategy.The training process incorporates learning from low-entropy path outputs.
- 5.3 QUANTITATIVE ANALYSIS: Performance, Efficiency, and Necessity peak after two self-evolved DPO training loops and decline with further loops.The analysis attributes this pattern to initially sufficient positive-negative pairs followed by degradation as evolution proceeds.
6 CONCLUSION
The paper proposes Tool-Light to improve the efficiency and accuracy of tool-integrated reasoning through entropy-guided data construction and multi-stage training.
- Tool-Light combines entropy-guided sampling with data construction and training improvements for efficient and accurate TIR.The framework targets overall performance, tool-call effectiveness, and inference simplicity.
A MORE DETAILS OF MAIN RESULTS
Tool-Light improves progressively across its training stages, with self-evolved DPO outperforming pre-aligned DPO alone before later gains begin to converge.
- Performance improves after SFT and increases further through multiple self-evolved DPO alignment rounds.
- Self-evolved DPO alignment outperforms pre-aligned DPO alone, while additional evolution rounds yield progressively smaller gains.The authors attribute the slower improvement to difficulty generating further training-beneficial samples.
- The results are reported for Qwen2.5-7B-Instruct across Tool-Light’s different training stages in Table 3.
B IMPACT OF DIFFERENT DATA RATIOS
Training performance depends strongly on the mixture of vanilla and entropy-guided data, with the study selecting a 13:7 ratio after weighing performance, tool use, and efficiency.
- The experiment mixes vanilla and entropy-guided samples at ratios from 1:7 to 7:1 while retraining the first DPO round.
- Increasing the vanilla-data proportion improves model performance and tool-call counts, but Efficiency does not improve monotonically.The authors associate the rising tool-call tendency with better problem solving but relate excessive calls to lower reasoning efficiency.
- The selected data ratio for the experiments is 13:7.
- Figure 6 presents metric variation as the data ratio changes, including Efficiency and LLM-as-Judge performance.
C CASE STUDY
The case study compares Tool-Light with Tool-Star on mathematical reasoning examples, emphasizing fewer tool calls on one problem and proactive external-tool use on a harder problem.
- On the same MATH500 problem, Tool-Light reaches the correct answer with fewer tool calls than Tool-Star.
- On a more difficult AIME24 problem, Tool-Light proactively calls an external tool during its reasoning trajectory.
- Another displayed example asks how many integer values satisfy a square-root interval, with the shown solution concluding 8.
- One displayed example asks for the maximum real part of a complex expression under |z| = 4.