PaperlayerView exact revision on arXiv

Source-linked AI summary

ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration

Yifei Chen, Guanting Dong, Zhicheng Dou

arXiv:2601.06860v2cs.AI

TL;DR

TIR agents often optimize answer accuracy without adequately calibrating tool-use behavior, leading to ineffective trajectories. ET-Agent combines a Self-evolving Data Flywheel with two-phase Behavior Calibration Training to expand exploration and progressively calibrate actions. Experiments report improvements across correctness, efficiency, reasoning conciseness, and tool execution, while the study remains limited to local Wikipedia retrieval and faces scaling difficulty for larger models.

  • Problem

    Existing TIR training overlooks behavioral alignment, leaving agents prone to redundant and insufficient tool calls while effective trajectory exploration remains unresolved.

  • Method

    ET-Agent iteratively refines trajectories through a Self-evolving Data Flywheel, then applies exploration fine-tuning and iterative behavior-calibration reinforcement learning.

  • Results

    ET-Agent improves behavioral efficiency, reasoning conciseness, and execution success rates while maintaining high accuracy across the reported evaluation.

  • Takeaways & Limitations

    ET-Agent provides a framework for progressively calibrating TIR behavioral patterns toward effective trajectories while preserving inference correctness.

  • Takeaways & Limitations

    Experiments are restricted to local Wikipedia retrieval, and applying ET-Agent to larger models is relatively difficult.

Abstract

from arXiv · show

Large Language Models (LLMs) can extend their parameter knowledge limits by adopting the Tool-Integrated Reasoning (TIR) paradigm. However, existing LLM-based agent training framework often focuses on answers' accuracy, overlooking specific alignment for behavior patterns. Consequently, agent often exhibits ineffective actions during TIR tasks, such as redundant and insufficient tool calls. How to calibrate erroneous behavioral patterns when executing TIR tasks, thereby exploring effective trajectories, remains an open-ended problem. In this paper, we propose ET-Agent, a training framework for calibrating agent's tool-use behavior through two synergistic perspectives: Self-evolving Data Flywheel and Behavior Calibration Training. Specifically, we introduce a self-evolutionary data flywheel to generate enhanced data, used to fine-tune LLM to improve its exploration ability. Based on this, we implement an two-phases behavior-calibration training framework. It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors. Further in-depth experiments confirm the superiority of \ourmodel{} across multiple dimensions, including correctness, efficiency, reasoning conciseness, and tool execution accuracy. Our ET-Agent framework provides practical insights for research in the TIR field. Codes can be found in https://github.com/asilverlight/ET-Agent

1 Introduction

ET-Agent addresses ineffective TIR behavior by combining expanded action-space exploration with progressive calibration toward effective trajectories. The framework targets behavioral correctness alongside answer accuracy, improving efficiency, conciseness, and execution success.

  • Motivation: Existing TIR training often prioritizes answer accuracy while overlooking behavioral correctness, producing redundant and insufficient tool calls.Redundant interactions can hinder practical application, while limited calibration leaves broader behavioral errors unresolved.
  • Motivation: The authors categorize TIR errors into improper tool-use and flawed reasoning logic through prior-work synthesis and quantitative preliminary analysis.The framework is motivated by a comprehensive investigation of erroneous behavioral patterns.
  • Approach: The Self-evolving Data Flywheel iteratively augments previous trajectories to broaden coverage of the tool-use action space.Training on enhanced samples is intended to improve the agent’s exploration ability.
  • Approach: Behavior Calibration Training first uses Action Space Exploration Fine-tuning, then alternates Group-wise Pareto Sampling with Curriculum RL Training.The two phases progressively calibrate actions toward optimal and standardized behavioral patterns.
  • Results: Experiments on six challenging tasks show best overall performance across correctness, efficiency, reasoning conciseness, and tool execution success dimensions.The reported improvements maintain high accuracy while substantially improving behavioral efficiency, reasoning conciseness, and execution success rates.

2 Related Work

Prior TIR work improves tool use through inference-time scaling, supervised or preference-based training, and reinforcement learning. ET-Agent differs by comprehensively calibrating behavioral patterns with on-policy training rather than focusing mainly on efficiency.

  • TIR Methods: Existing TIR methods generally use inference-time scaling or training-based optimization with SFT, DPO, or RL.These approaches equip LLM-based agents with tool-integrated reasoning ability.
  • Behavior Calibration: Several methods target behavioral efficiency, while Tool-Light also studies insufficient tool calls and reasoning redundancy using DPO.The supplied related-work passage identifies redundant tool-call mitigation as a limited focus of prior work.
  • ET-Agent: ET-Agent uses on-policy training to progressively calibrate TIR behavioral patterns more comprehensively.This positions the framework beyond approaches focused primarily on behavioral efficiency.
  • Agentic Reinforcement Learning: Agentic reinforcement learning has expanded TIR across domains and produced frameworks adapted to branching strategies and tool use.Examples include methods that optimize information-entropy-based branching or design training frameworks for tool use.

3 Preliminary Experiment

The preliminary study identifies four recurring TIR errors spanning tool use and reasoning logic. Its analyses show redundancy, aborted executions, premature closure, and substantial trajectory diversity as calibration targets.

  • Improper Tool-Use: Improper tool-use errors comprise redundant tool calls and aborted tool execution caused by improperly formulated calls.Examples include calls that add no useful information, null queries, and defective code.
  • Flawed Reasoning Logic: Flawed reasoning logic comprises insufficient tool calls and erroneous reasoning processes arising from defective planning or cognition.Insufficient calls can lead to incorrect results when additional tool use is needed.
  • Improper Tool-Use: Redundant tool calls are prevalent across both analyzed tasks, indicating that uncalibrated agents struggle to invoke tools efficiently.The study uses GPT-4o to identify redundant calls in model outputs.
  • Improper Tool-Use: Most aborted-execution trajectories contain one error, while only a small number contain repeated errors.This pattern motivates improving tool-call triggering accuracy through training.
  • Flawed Reasoning Logic: A considerable number of incorrect trajectories can be modified after continued reasoning, revealing a premature-closure problem caused by difficulty invoking additional tools.Figure 3(a) examines additional calls required to modify incorrect outputs.
  • Trajectory Diversity: Correct outputs still show substantial disparities in tool-call frequency for identical questions, confirming that TIR has an extensive action space.Without calibration, agents can hardly explore an effective reasoning trajectory.

4 The Proposed Method: ET-Agent

ET-Agent combines a self-evolving data flywheel with two-phase behavior-calibration training to expand trajectory exploration and steer TIR agents toward effective behavioral patterns.

  • 4.1 Self-Evolving Data Flywheel: The Self-Evolving Data Flywheel iteratively refines correct trajectories, reflects on incorrect ones, and reintegrates enhanced paths into an expanded training dataset.Correct trajectories receive redundant-step or global refinement, while incorrect trajectories undergo self-correction or hint injection.
  • 4.1 Self-Evolving Data Flywheel: The flywheel initializes multiple outputs per question, separates correct and incorrect rollouts, and repeats enhancement for R iterations.The resulting dataset broadens action-space coverage and supports exploration during fine-tuning.
  • 4.2 Behavior Calibration Training: Behavior Calibration Training first performs Action Space Exploration Fine-tuning, then alternates Group-wise Pareto Sampling with Curriculum RL Training.This sequence expands trajectory diversity before reinforcement learning calibrates actions toward optimal patterns.
  • 4.2.1 Action Space Exploration Fine-tuning: Quality controls remove incorrect, malformed, or failed trajectories before rejection-sampling fine-tuning, reducing aborted tool execution and format errors.The resulting trajectory diversity supplies variance for subsequent group-wise calibration.
  • 4.2.2 Group-wise Pareto Sampling: Pareto Sampling selects non-dominated trajectories using correctness and behavioral dispersion, retaining diverse high-contribution samples through frontier ranking and crowding distance.Correctness dispersion measures variation in trajectory correctness, while behavioral dispersion measures variation in tool invocation counts.
  • 4.2.3 Curriculum RL Training: Curriculum RL uses format, correctness, tool-behavior, and thinking-process rewards to calibrate behavior while preserving correctness and encouraging concise reasoning.The aggregate reward favors trajectories that are correct, efficient, and well formatted.

5 Experiments

Experiments evaluate ET-Agent on six mathematical- and knowledge-intensive tasks using correctness, efficiency, behavioral, and training-dynamics analyses. ET-Agent generally outperforms baselines and improves multiple dimensions of tool-integrated reasoning.

  • Experimental Setup: Evaluations cover AIME24, AMC23, MATH500, 2WikiMultiHopQA, Bamboogle, and MuSiQue, using search and code-execution tools.Correctness uses LLM-as-Judge for mathematical reasoning and F1 for knowledge-intensive tasks.
  • Main Results: ET-Agent achieves the best correctness and efficiency performance in most evaluated tasks.The main results report broad gains across the six-task evaluation.
  • Main Results: AutoTIR reaches correctness 52.3 on MSQ but has efficiency 12.0, below ET-Agent’s 15.3.The comparison illustrates a mismatch between answer quality and tool-use efficiency in existing methods.
  • Main Results: On average, ET-Agent records correctness 60.1 and efficiency 46.0, outperforming baselines especially on AIME24, AMC23, and 2Wiki.The paper reports a leading position across the remaining datasets as well.
  • Tool-use Behavioral Patterns Analysis: ET-Agent achieves state-of-the-art results across five behavioral metrics, including conciseness, successful execution, and reasoning length.These metrics assess redundant calls, tool-call reliability, and reasoning-chain compactness.
  • Tool-use Behavioral Patterns Analysis: The RFT stage broadens action-space exploration, while RL produces a more compact distribution indicating convergence toward calibrated behavior.The analysis samples ten trajectories per query across the base Instruct, RFT, and RL stages and visualizes them with t-SNE.
  • Training Dynamics and Ablation Study: ET-Agent outperforms Vanilla ARPO in both efficiency and reward during RL, while removing any major component degrades correctness or efficiency.Ablations identify the data flywheel, Pareto sampling, and reward design as important components; fixing σ can cause reward hacking.

6 Conclusion

ET-Agent calibrates TIR behaviors by first expanding action-space exploration and then steering exploration toward optimal trajectories. Experiments report effective behavioral patterns alongside inference correctness, while evaluation remains limited by local Wikipedia retrieval and difficulty scaling to larger models.

  • ET-Agent first enhances action-space exploration, then calibrates that exploration toward optimal trajectories.The pipeline is designed to comprehensively calibrate TIR behaviors.
  • ET-Agent exhibits effective behavioral patterns while ensuring inference correctness.
  • Experiments are restricted to local Wikipedia retrieval because of resource constraints.The authors identify integration of live web search as future work.
  • Applying ET-Agent to larger models is relatively difficult.Scaling the framework to larger architectures is proposed as future work.

A.1.1 Training Datasets

The training-dataset section combines mathematical, knowledge-intensive, and deep-search resources with datasets and baselines for evaluating reasoning and tool-integrated search. It describes resources spanning competition mathematics, multi-hop question answering, and search-oriented agent training.

  • Training data is composed as shown in Table 3, covering mathematical-reasoning, knowledge-intensive, and deepsearch task categories.The table abbreviates these categories as MR, KI, and DS.
  • 2WikiMultiHopQA, HotpotQA, MuSiQue, and TriviaQA evaluate multi-hop or open-domain question answering through linked evidence and reasoning chains.MuSiQue contains approximately 25,000 questions with logical depths of 2 to 4 hops.
  • ASearcherLRM35k addresses data scarcity and short interaction turns by constructing a task library for long-range search reasoning.
  • WebShaper constructs complex search and reasoning tasks using a formal-driven approach based on mathematical set theory.
  • AMC23 evaluates competition-level mathematical reasoning using real 2023 AMC 10 and AMC 12 examination questions.
  • MATH500 measures competition-level mathematical reasoning across seven mathematical subfields.Its design addresses evaluation cost while retaining discriminative problems.
  • Search-o1, Search-R1, Research, WebThinker, WebSailor, and ToRL represent search- and tool-integrated reasoning baselines or related frameworks.Their described approaches include agentic search, reinforcement learning, autonomous web exploration, and trial-and-error tool use.

C.1 Preliminary Experiments Details

Preliminary experiments use mathematical and knowledge-intensive task sets to analyze insufficient tool calls and differences in tool-call counts across sampled trajectories.

  • The mathematical task set includes AIME, Numina-Math, and OpenR1-Math-220k.
  • The knowledge-intensive task set includes 2WikiMultihopQA, HotpotQA, MuSiQue, and TriviaQA.
  • 20.8% of failures in the insufficient-tool-call analysis can be corrected after continued reasoning.
  • The tool-call-number analysis samples 16 trajectories for each query.

C.2 Training Details

Training uses Qwen2.5-7B-Instruct for action-space exploration fine-tuning and VERL with ARPO for reinforcement learning. The experiments use specified optimization settings and four NVIDIA A800 GPUs.

  • Action-space exploration fine-tuning selects 10.5k data for RFT using Qwen2.5-7B-Instruct.Training uses a 5e-6 learning rate, batch size 32, and three epochs.
  • Reinforcement learning uses VERL and the ARPO algorithm across three curriculum-learning rounds.The setup uses batch size 64, three epochs, maximum output length 4096 tokens, and learning rate 1e-6.
  • All reinforcement-learning experiments are completed using 4 NVIDIA A800 GPUs.

C.3 Evaluation Details

Evaluation uses fixed sample, tool, call-limit, and output-length settings across the tested task types.

  • Each test dataset contributes its first 500 samples for evaluation.
  • Mathematical-reasoning tasks use Google Search, while knowledge-intensive tasks use local retrieval matching training settings.
  • Code compiler and search-engine calls are each capped at 6, with 4096 tokens allowed per reasoning round.

D Complete Results of Behavioral Pattern Indicators

ET-Agent is reported to outperform most baselines across the behavioral-pattern benchmarks and metrics, with the section presenting conciseness, execution, and reasoning-length evaluations alongside analysis prompts.

  • Complete benchmark results: ET-Agent outperforms most baseline methods on all reported metrics and stably surpasses them in average performance.
  • Behavioral-pattern indicators: The section includes conciseness, successful execution, and reasoning-length metrics, with M500 denoting MATH500.
  • Behavior analysis: The evaluation materials define redundant and ineffective tool calls and require binary yes-or-no trajectory judgments.
  • Trajectory calibration: Additional prompts identify the earliest redundancy or flawed step and request concise analyses, corrected steps, or refined reasoning while preserving trajectory structure.

F Case Study of Error Behavior Patterns

The case study provides representative examples of four TIR error patterns, with erroneous locations marked in red and examples organized across Tables 7–10.

  • Case study: Representative examples illustrate four distinct TIR error patterns identified in the case study.
  • Case study: Red text marks the erroneous locations in the example trajectories.
  • Error-pattern examples: Tables 7–10 cover redundant tool calls, aborted tool execution, insufficient tool calls, and erroneous reasoning processes.
Loading 2601.06860v2…