Source-linked AI summary
Acting Less is Reasoning More! Teaching Model to Act Efficiently
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, Heng Ji
TL;DR
Existing tool-integrated reasoning RL methods often optimize final correctness without controlling costly or unnecessary tool use, which can hinder internal reasoning. The paper proposes OTC-PO, a reward-based RL framework that jointly targets correctness and minimal tool calls, instantiated as OTC-PPO and OTC-GRPO. Across multiple benchmarks, it reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4% while maintaining comparable accuracy.
Problem
Existing TIR RL approaches optimize final answer correctness but do not model optimal tool use, leading to excessive calls, higher costs, and cognitive offloading.
Method
OTC-PO uses a tool-integrated reward that ranks tool-call counts against an estimated question–model-specific optimum and jointly optimizes correctness and tool efficiency.
Results
Across search and code tasks, OTC-PO reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4% while maintaining comparable answer accuracy.
Takeaways & Limitations
Tool productivity provides a finer-grained objective for training TIR agents to preserve answer performance while reducing unnecessary tool interactions.
Takeaways & Limitations
The optimal number of tool calls is assumed to vary by question and model and is unknown in advance, so OTC-PO approximates it from observed correct trajectories.
Abstract
from arXiv · showhide
Tool-integrated reasoning (TIR) augments large language models (LLMs) with the ability to invoke external tools during long-form reasoning, such as search engines and code interpreters, to solve tasks beyond the capabilities of internal reasoning. While reinforcement learning (RL) has shown promise in training such agents, most of existing approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use. This often leads to excessive tool calling, incurring high computational costs and hindering the development of internal reasoning capabilities - a phenomenon known as \textit{cognitive offloading}. To this end, we propose Optimal Tool Call-controlled Policy Optimization (OTC-PO), a simple yet effective RL-based framework that encourages models to produce accurate answers with minimal tool calls. Our method introduces a tool-integrated reward that jointly considers answer correctness and corresponding tool use behavior of model to reach that answer. To validate the effectiveness, we introduce the metric of \textit{tool productivity}, defined as the ratio between the number of correct answers and the total number of tool calls across all test cases. This metric reflects how efficiently tool usage contributes to successful task completion, with higher values indicating smarter and more autonomous reasoning. We instantiate this framework within both Proximal Policy Optimization (PPO) and Group Relative Preference Optimization (GRPO), resulting in OTC-PPO and OTC-GRPO. Experiments with Qwen-2.5 and Qwen-Math across multiple QA benchmarks show that our approach reduces tool calls by up to 68.3\% and improves tool productivity by up to 215.4\%, while maintaining comparable answer accuracy.
1 Introduction
Tool-integrated reasoning expands LLM problem-solving through external tools, but correctness-only RL can encourage unnecessary tool use and cognitive offloading. OTC-PO addresses this by jointly optimizing answer correctness and minimal tool calls.
- Tool-integrated reasoning connects LLMs to search engines, code interpreters, and other external capabilities beyond internal knowledge-only reasoning.
- Correctness-only RL for TIR can encourage indiscriminate tool usage, increasing computational and temporal overhead while hindering internal reasoning.
- OTC-PO rewards correct answers reached with the minimal number of tool calls, treating excessive tool use as suboptimal.
- OTC-PO combines answer correctness with tool efficiency through a tool-integrated reward and defines tool productivity as task benefit relative to tool usage cost.
- OTC-PPO and OTC-GRPO instantiate OTC-PO within PPO and GRPO while reducing tool-call costs and preserving most accuracy across in-domain and out-of-domain evaluations.
2 Related Work
Prior work has explored tool-integrated reasoning efficiency through prompting and supervised fine-tuning, but these approaches often depend on confidence mechanisms, curated datasets, or complex prompt engineering.
- Existing efficiency studies examine tool-use costs in tool-integrated reasoning through prompting engineering and supervised fine-tuning.
- Self Divide-and-Conquer uses an LLM confidence score to decide whether to call tools during reasoning and targets effectiveness–efficiency trade-offs in retrieval-augmented generation.
- SMART fine-tunes models on a designed dataset to call tools only when knowledge lies outside their inherent parametric knowledge.
- Most existing approaches still rely on complex prompt engineering or annotated datasets, limiting their adaptability.
3 Methodology
The methodology reframes tool-integrated reasoning as jointly optimizing answer correctness and the minimal tool-call cost. OTC-PO implements this objective through tool-aware rewards and PPO or GRPO variants that adapt to question–model-specific tool requirements.
- 3.1 Task Definition: Tool-integrated reasoning trajectories interleave model reasoning, optional tool calls, and returned observations before producing a final answer.Reasoning steps without tool usage are also represented, with empty tool-call and observation fields.
- 3.1 Task Definition: The task objective is to generate a correct answer while minimizing trajectory cost, measured by the number of tool calls.This shifts optimization beyond answer correctness toward behavioral efficiency during reasoning.
- 3.2 Tool-integrated Reinforcement Learning: OTC-PO augments conventional RL rewards with tool-use behavior and can be instantiated within PPO and GRPO.The framework is designed to preserve accuracy while reducing tool-interaction costs.
- OTC-PPO: OTC-PPO applies a decreasing reward as tool calls increase, with a smooth constant controlling how quickly additional calls are penalized.The authors note that tracking the minimum observed call count for later optimization can require longer training because of poor PPO sampling efficiency.
- OTC-GRPO: OTC-GRPO estimates the optimal call count from the minimum number of calls among correct trajectories for each question and updates it across epochs.Its mapping gives the highest reward when actual calls equal the estimated optimum and lower reward when they exceed or fall short.
- Tool-integrated Reward Design: The final tool-integrated reward multiplies conventional reward by a tool-efficiency coefficient, so efficiency is emphasized only when the primary task objective is satisfied.For an incorrect answer, the conventional correctness reward is zero and disables the tool-efficiency influence, mitigating reward hacking.
4 Experiments
The experiments evaluate OTC-PO on search and code tasks using accuracy, tool-call cost, and tool productivity, with comparisons against established baselines. Results report comparable accuracy alongside fewer tool calls and improved productivity, including stronger effects as model size increases.
- Datasets and Baselines: The evaluation uses NQ and HotpotQA for search, ToRL code data, and baselines including SFT, Base-RL, RAG, IRCoT, Search-R1, and ToRL.
- Evaluation Metrics: Tool productivity measures correctly answered questions per unit of tool call, complementing exact match with reasoning-cost information.
- Search as Tool: OTC-PO achieves comparable performance with much lower tool-call counts and higher tool productivity than corresponding Search-R1 baselines.
- Search as Tool: 68.3% fewer tool calls and 215.4% higher tool productivity are reported for OTC-PPO relative to Search-R1-PPO on search-required tasks.
- Training: Training analyses show comparable results with fewer tool calls, shorter responses, faster optimization, and delayed early collapse for GRPO.
- Code as Tool: OTC-PO improves tool productivity on code tasks without substantially sacrificing accuracy, with greater productivity gains as model size increases.
5 Analysis
OTC-PO reduces unnecessary tool use while preserving answer effectiveness across training, inference, and out-of-domain evaluation. Case-study evidence attributes this efficiency to more informative searches or greater reliance on internal reasoning.
- Training: OTC-PO achieves comparable results with fewer tool calls and shorter responses, while making training optimization faster and more efficient.The method also reduces financial and computational overhead from real-time tool interactions during training.
- Inference: OTC-PO matches baseline answers with fewer tool calls in most cases, exceeding 80% of cases as model size increases.The analysis reports no significant effectiveness loss under the same reinforcement-learning setting.
- Inference: OTC-PPO has a lower gap between MA and LA, while AE comprises roughly 80% of MA cases for 3B models and exceeds 95% for 7B models.These results indicate efficiency gains while retaining effectiveness.
- Out-of-domain evaluation: OTC-PO achieves the highest TP across evaluated models and benchmarks, with OTC-GRPO outperforming OTC-PPO on TP as model size increases.OTC-PPO also outperforms Search-R1-PPO in EM for Qwen2.5-7B-Base with fewer tool calls.
- Case study: Search-R1 makes redundant, information-poor searches, whereas OTC-PPO uses richer queries and OTC-GRPO relies on internal reasoning when external tools are unnecessary.The case study frames unpenalized tool use as cognitive offloading.
6 Conclusion and Future Work
The paper reframes agentic reinforcement learning to optimize both answer correctness and tool-use behavior. OTC-PO uses tool-integrated rewards to encourage optimal tool calls and support tool productivity without sacrificing benchmark accuracy.
- Conclusion: The paper redefines the agentic-RL objective to optimize tool-use behavior alongside final answer correctness.This reframing targets effectiveness and efficiency in tool-integrated reasoning.
- Conclusion: OTC-PO is an RL framework that rewards correct answers reached with optimal tool calls through a tool-integrated reward.The framework is intended to preserve accuracy while reducing tool-interaction costs.
- Conclusion and future work: The paper reports that extensive external-tool reliance hinders internal reasoning, whereas minimizing tool calls fosters its development.Future work extends the framework to more complex tasks, broader tool sets, and longer-horizon reasoning.
- Reward design: The reward illustrations show decreasing OTC-PPO reward with more tool calls, while OTC-GRPO gives highest reward along the optimal-call diagonal.For OTC-PPO, a correct trajectory using no tools receives the highest reward.
D.2 Theoretical Justification
The theoretical justification argues that the tool-integrated reward preserves correctness optimization while ranking correct trajectories by tool efficiency. The reward design also limits tool-reward influence when answers are wrong.
- Theoretical justification: The analysis treats the response as a tool-integrated reasoning trajectory paired with a final answer and reuses prior justification for the reward design.The stated assumptions are retained under this response formulation.
- Accuracy preservation: The population-level maximizer under the tool-integrated reward remains as accurate as the maximizer under correctness-and-format reward.This is the theoretical accuracy-preservation result reported in the justification.
- Reward behavior: For correct answers, the tool reward assigns higher scores to trajectories using fewer tools because correctness and format rewards are fixed and positive.This establishes the intended efficiency ranking among correct trajectories.
- Reward behavior: For wrong answers, the correctness-dependent component becomes zero, disabling the tool reward’s influence and reducing reward hacking.When format is positive, fewer tools still receive more reward because they incur less cost.
E Analysis of Search as Tool
Comparisons between Qwen2.5-7B-Base and Qwen2.5-7B-Instruct show that model initialization affects tool-use behavior. Base models generally perform better, while Instruct models use more tools but still remain below Search-R1 in tool calls.
- Model comparison: Instruct models tend to use more tools than Base models, although their tool-call counts remain much lower than Search-R1.The passage attributes this tendency to memorized patterns from supervised fine-tuning.
E.2 The Effects of α
The experiments examine how α and C affect training behavior and tool use. Larger α does not necessarily improve results, while OTC-GRPO maintains more stable tool-call counts as C changes.
- E.2 The Effects of α: Larger α does not consistently improve performance and can make the desired behavior harder to learn.A single tool call may cause large reward fluctuations, especially when C is relatively low.
- E.2 The Effects of α: Increasing C leads ToRL-GRPO to make more tool calls when rewards consider only final correctness.OTC-GRPO instead maintains more stable tool-call counts as C increases.
- E.2 The Effects of α: OTC-GRPO produces larger tool-productivity improvements as C increases and delays earlier reward collapse observed in GRPO.The same pattern is also observed when search is used as the tool.
F.2 Additional Results
Additional evaluation on the MATH dataset supports the conclusions from the main experiments and indicates generalization of the method.
- F.2 Additional Results: The method’s conclusions also hold on MATH when evaluated following ToRL.The authors characterize this result as demonstrating great generalization capability.
G More Examples
Additional examples show OTC-PO solving questions with no tool calls or fewer, more accurate searches. Compared with Search-R1 and ToRL-GRPO, it relies more on internal reasoning while reducing external tool use.
- No Tool Call Required: OTC-GRPO can produce a correct answer through internal reasoning without external tool calls, unlike Search-R1’s extensive unnecessary calls.This behavior is illustrated by an example in which the question concerns the Bellagio resort’s US city.
- Less Tool Calls with Accurate Search: OTC-PO can use fewer tool calls while performing more accurate searches to reach a correct answer.An example concerns identifying the first-person science-fantasy young-adult series Animorphs.
- More Examples: OTC-GRPO and OTC-PO examples show correct answers obtained with reduced external tool use.The examples include both no-tool solutions and solutions using fewer calls.