PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning

Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, Li Du

arXiv:2507.21836v1cs.CL

TL;DR

Existing tool-integrated reasoning approaches often rely on rigid tool-use patterns that can undermine core language abilities. AutoTIR uses reinforcement learning to let models choose whether and which tools to invoke, and reports superior overall performance and generalization across tasks.

  • Problem

    Many tool-integrated reasoning methods rely on fixed tool-use patterns, while learning tool-use strategies can degrade language understanding and instruction following.

    Source: Introduction · pp. 1–2

  • Method

    AutoTIR uses reinforcement learning with action and output rewards to learn whether and which tools to invoke and to encourage accurate final answers.

    Source: Introduction · p. 2, Reward Design · p. 3

  • Results

    AutoTIR achieves superior overall performance and generalizes well across knowledge-intensive, mathematical, and general-domain benchmarks.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    The results highlight reinforcement learning's potential as a foundation for generalizable and scalable reasoning in language models.

    Source: Introduction · p. 2

    The formalization uses a search engine for local databases and a sandboxed code interpreter, though it can accommodate more tools.

    Source: Problem Formalization · p. 3

Abstract

from arXiv · show

Large Language Models (LLMs), when enhanced through reasoning-oriented post-training, evolve into powerful Large Reasoning Models (LRMs). Tool-Integrated Reasoning (TIR) further extends their capabilities by incorporating external tools, but existing methods often rely on rigid, predefined tool-use patterns that risk degrading core language competence. Inspired by the human ability to adaptively select tools, we introduce AutoTIR, a reinforcement learning framework that enables LLMs to autonomously decide whether and which tool to invoke during the reasoning process, rather than following static tool-use strategies. AutoTIR leverages a hybrid reward mechanism that jointly optimizes for task-specific answer correctness, structured output adherence, and penalization of incorrect tool usage, thereby encouraging both precise reasoning and efficient tool integration. Extensive evaluations across diverse knowledge-intensive, mathematical, and general language modeling tasks demonstrate that AutoTIR achieves superior overall performance, significantly outperforming baselines and exhibits superior generalization in tool-use behavior. These results highlight the promise of reinforcement learning in building truly generalizable and scalable TIR capabilities in LLMs. The code and data are available at https://github.com/weiyifan1023/AutoTIR.

Introduction

Reasoning-oriented post-training strengthens LLMs, while external tools extend reasoning with knowledge retrieval and computation. AutoTIR addresses the challenge of choosing whether and which tool to invoke while preserving language and instruction-following capabilities.

Introduction

  • TIR extends language-model reasoning with external tools that provide real-time knowledge or precise computation for tasks requiring factual precision or symbolic manipulation.The introduction describes retrieval and code execution as examples of tools that can support more complex queries and reasoning chains.Source: Introduction · p. 1
  • Existing TIR methods often use fixed tool patterns, and tool-use learning can degrade core language understanding and instruction following.The authors identify the missing ability to decide whether to invoke tools and which tool to call as the source of these challenges.Source: Introduction · pp. 1–2
  • AutoTIR uses reinforcement learning to let models choose whether and which tool to invoke, with action and output rewards guiding tool decisions and answers.Its action reward encourages appropriate invocation and discourages redundant use; its output reward promotes responses better than a reference model.Source: Introduction · p. 2
  • AutoTIR shows consistent performance improvements over baselines across knowledge-intensive, mathematical, and general reasoning tasks, with efficient, context-aware tool use.The authors attribute the gains primarily to learning a generalizable tool-invocation strategy that balances tool-integrated reasoning with language modeling and instruction following.Source: Introduction · p. 2

Related Work

Related work describes reinforcement learning with verifiable rewards as a way to train reasoning models, and surveys reinforcement-learning approaches to tool use. It also identifies limits in generalization and preserving instruction following as challenges for tool-integrated reasoning.

Reinforcement Learning with Verifiable Rewards

Tool-Integrated Reasoning

Methodology

AutoTIR uses pure reinforcement learning to let models decide whether and which tool to invoke. It formalizes tool-integrated reasoning as iterative steps and trains the policy with action and answer rewards.

Methodology

  • AutoTIR uses pure reinforcement learning to enable autonomous decisions about whether and which tool to invoke across diverse tasks.The method contrasts this with prior work using predefined tools for specific domains.Source: Methodology · p. 2

Problem Formalization

  • Each reasoning step combines language reasoning, a tool call, and its result; no-tool steps use empty call/result fields, and generation ends at an answer or context limit.The formalization allows a search engine and code interpreter, while noting that more tools could be accommodated.Source: Problem Formalization · pp. 2–3

Reward Design

  • The total reward is a weighted sum of action and output rewards, targeting both tool-choice decisions and accurate final answers.The action reward guides tool invocation strategy; the output reward assesses the final answer based on the reasoning process.Source: Reward Design · p. 3
  • Correct tool choices are rewarded, wrong tool selections are penalized even when the answer is correct, and open-domain tool use is left unrestricted for exploration.Search is encouraged for complex knowledge tasks and code execution for math with massive numerical calculation; open-domain exploration is intended to promote efficiency and preserve core language capabilities.Source: Reward Design · p. 3
  • The output reward requires a boxed answer and evaluates correctness with task-specific functions, including F1 for QA, binary reward for math, and instruction-adherence scoring.The IF score grants 1 for satisfied instructions and 0 otherwise.Source: Reward Design · pp. 3–4

Inference with Multiple Tools.

  • At inference, the model iterates through reasoning, optional tagged tool calls, and returned results, but can skip tools and answer directly.Tool execution results are masked during loss computation and do not contribute to gradient updates.Source: Inference with Multiple Tools · p. 4

RL Learning Algorithm

  • GRPO trains the policy from groups of rollouts, using normalized advantages and a KL-divergence penalty to constrain policy shifts.The reference model is an instruction model; ε controls the permissible policy-update range and β regulates KL-penalty magnitude.Source: RL Learning Algorithm · p. 4

Experiments

Experiments assess AutoTIR across ten benchmarks and report stronger cross-domain performance alongside generalizable tool use. Ablations examine the roles of tool access, instruction-following data, action penalties, and free exploration.

Experiment Setup

  • AutoTIR is trained on Qwen2.5-7B-Instruct with tool-use, answerable-without-tools NQ, and instruction-following data, then tested across knowledge, math, open-domain, and instruction-following benchmarks.Evaluation uses EM for mathematical reasoning and open-domain QA, Accuracy for LogiQA, SAcc for IFEval, and TS and TP as auxiliary tool-use metrics.Source: Experiment Setup · pp. 4–5

Main Results

  • 46.01 average score across three domains; AutoTIR outperforms baselines on knowledge and math while retaining instruction following and generalizing to out-of-distribution tests.The reported average covers benchmarks from knowledge-intensive, mathematical, and open-domain reasoning and instruction-following tasks.Source: Table 1 · p. 5, Main Results · p. 6

Ablation Analysis

  • Removing tools harms all benchmarks, omitting instruction-following data cuts IFEval from 51.02 to 13.12 SAcc, and removing action penalties degrades knowledge and instruction-following performance.The study describes the no-tools decline as especially pronounced in knowledge-intensive and mathematical domains.Source: Ablation Analysis · p. 6
  • Forcing tools on prior-selected open-domain instructions lowers IFEval, GSM8K, and MATH500 performance; the authors favor free exploration because open-domain tool need lacks a clear standard.The paper notes that human priors may not always help given the complexity of open-domain instructions and differences between human and model reasoning.Source: Ablation Analysis · pp. 6–7

Scalability of Performance

  • During training, action and output rewards and response length rise; the authors interpret longer outputs alongside reward gains as increasingly complex reasoning.The paper presents these trends as indicating that the model is learning a tool invocation strategy and adapting to harder reasoning problems.Source: Scalability of Performance · p. 7, Figure 3 · p. 7

Tool-Utilization Efficiency Analysis

  • AutoTIR shows comparable or higher tool selection and productivity on knowledge and math tasks, while baseline tool-use strategies are task-specific and weaken on open-domain tasks.Tool selection is not calculated on open-domain instructions because there is no clear standard for deciding whether tools should be invoked.Source: Tool-Utilization Efficiency Analysis · p. 7

Tool Advantage Analysis

  • Tool gains are modest on GSM8K and MATH500 but substantial on AIME; tool-free AutoTIR also beats Qwen2.5-7B-Instruct, and results suggest better balance than other RL baselines.The paper describes tool advantage as varying with task difficulty and says AutoTIR aims to invoke tools where they offer the greatest advantage.Source: Tool Advantage Analysis · p. 8

Conclusion

AutoTIR uses reinforcement learning to replace rigid tool-use patterns with context-aware decisions about whether and which tools to invoke. Across knowledge-intensive, mathematical, and general-domain benchmarks, it achieves superior overall performance and generalizes across tasks.

Conclusion

  • AutoTIR achieves superior overall performance and significantly outperforms baselines across knowledge-intensive, mathematical, and general-domain benchmarks.Its learned tool-use policy adapts to varying task demands for more efficient, context-aware integration.Source: Conclusion · p. 8
  • AutoTIR jointly rewards answer correctness and structured-output adherence while penalizing incorrect tool usage.The framework enables autonomous decisions about whether and which tools to invoke.Source: Conclusion · p. 8
  • AutoTIR shifts from static invocation patterns to adaptive learned strategies as a foundation for future tool-augmented LLMs.The paper characterizes this foundation as scalable and generalizable.Source: Conclusion · p. 8

System Prompt Template for AutoTIR

AutoTIR uses control tags to trigger tool invocation and requires adherence to a predefined output format. Its two prompt templates support switching between tool-assisted and standalone reasoning modes.

System Prompt Template for AutoTIR

  • Special control tags such as </search> and </code> trigger tool invocation, and the LLM must follow the predefined output format.The output-format requirement is intended to ensure proper policy execution.Source: System Prompt Template for AutoTIR · p. 11
  • Two prompt templates enable switching between tool-assisted reasoning for complex retrieval or code tasks and standalone reasoning for simpler tasks such as GSM8K.Standalone mode bypasses tool use and prioritizes inference speed.Source: System Prompt Template for AutoTIR · p. 11
  • Avoiding unnecessary tool calls on simple problems reduces latency when tool overhead outweighs the benefits.The paper describes this dual-mode design as optimizing efficiency.Source: System Prompt Template for AutoTIR · p. 11

Test-Time Scaling Analysis

AutoTIR’s test-time tool use can add computational overhead, particularly when invocation decisions are based on problem category rather than individual-sample difficulty. The analysis motivates an adaptive switch for straightforward problems.

Test-Time Scaling Analysis

  • The current approach can invoke tools inefficiently because it selects them by problem category rather than single-sample difficulty.This can lead to unnecessary computational overhead.Source: Test-Time Scaling Analysis · p. 11
  • For simpler tasks such as GSM8K and MATH500, AutoTIR without tools achieves significantly faster inference than AutoTIR with tools.The comparison highlights the computational cost of tool invocation where performance gains are minimal.Source: Test-Time Scaling Analysis · p. 11
  • AutoTIR with tools generates longer responses across tasks, including 1.5k tokens for MuSiQue versus 0.1k without tools.The longer outputs contribute to higher generation costs and slower inference.Source: Test-Time Scaling Analysis · p. 11
  • An adaptive tool-integrated reasoning switch could bypass calls for straightforward problems, improving inference speed without compromising performance.The paper presents this as a way to optimize compute resources and practical deployment efficiency.Source: Test-Time Scaling Analysis · p. 11

Scaling Analysis on the RL Training Curve

During training, AutoTIR’s response length and format reward increase, while validation performance trends positively across nearly all monitored benchmarks through 250 steps.

Scaling Analysis on the RL Training Curve

  • Validation scores show a positive trend across nearly all five benchmarks as training progresses up to 250 steps.The datasets span knowledge-intensive question answering, mathematics, and general instruction following; dev scores generally improve rapidly at first.Source: Scaling Analysis on the RL Training… · p. 11
  • AutoTIR’s response length generally increases during training, which the paper interprets as growing long-chain-of-thought reasoning capability.Response length counts output tokens excluding retrieval results and is interpreted as test-time reasoning cost.Source: Scaling Analysis on the RL Training… · p. 11
  • Format reward improves with training steps, indicating that AutoTIR increasingly generates responses that follow predefined formats.Overall reward improvement accompanies learning an action strategy for whether and which tools to use across task types.Source: Scaling Analysis on the RL Training… · p. 11

System Prompt Template for AutoTIR

Figure 5 presents the system prompt template for training and inference. Training metrics rise steeply at first, then stabilize or continue improving more gradually across domains.

Implementation Details

The implementation uses Qwen2-7B-instruct with GRPO and a Wikipedia-backed search tool. The section also identifies the implementation table and figures covering inference prompts, training metrics, and test-time comparisons.

  • Experiments use Qwen2-7B-instruct, GRPO in verl, e5-base-v2 over a Wikipedia 2018 corpus, and FlashRAG-reproduced baselines on 8 NVIDIA A800-80G GPUs.Key parameter settings are detailed in Table 5.Source: Implementation Details · p. 12, Table 5 · p. 12
  • Figure 6 presents the system prompt template for text-based inference.Source: Figure 6 · p. 13
  • Figure 7 covers training-stage performance metrics across datasets, while Figure 8 presents a test-time comparison.Source: Figure 7 · p. 13, Figure 8 · p. 13
Loading 2507.21836v1…