PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning

Qisheng Su, Shiting Huang, Zhen Fang, Ziyan Chen, Zehui Chen, Feng Zhao

arXiv:2604.05404v2cs.PFcs.SE

TL;DR

Tool-integrated reasoning creates hardware costs that token and toolcall counts do not capture, because toolcalls can evict KV-Cache and long responses expand the decode context. The paper proposes PTE, a hardware-aware metric that unifies prefill and decode costs, then validates it against latency across hardware and benchmarks. PTE aligns better with wall-clock latency, identifies four inefficiency patterns, and finds that higher-cost trajectories tend to have lower correctness, although these observations are correlational and the validation scope is limited.

  • Problem

    Existing TIR efficiency metrics do not capture the asymmetric hardware costs of prefill and decode, including KV-Cache eviction and context growth from long tool responses.

  • Method

    PTE unifies internal reasoning and external tool-use costs by expressing memory-bound decode cost in equivalent compute-bound prefill tokens.

  • Results

    PTE aligns better with wall-clock latency than token-count metrics, maintains model rankings across hardware profiles, and experiments identify four TIR inefficiency patterns.

  • Takeaways & Limitations

    Higher-PTE trajectories tend to have lower reasoning correctness, indicating that using more tools does not necessarily improve answer quality.

  • Takeaways & Limitations

    PTE omits API latency, abstracts hardware efficiency through γ, and has been validated only on specific tasks and models.

Abstract

from arXiv · show

In real-world Tool-Integrated Reasoning (TIR) scenarios, where LLMs interleave reasoning with external tool calls, a major source of inefficiency is that the toolcalls create pauses between LLM requests and cause KV-Cache eviction, forcing recomputation. Also, the long, unfiltered response returned by external tools inflates the KV-Cache, so each decode step spends more time loading the growing cache and thus becomes steadily slower as context length increases. However, existing efficiency metrics like token counts and toolcall counts fail to capture the real model inference latency. To address this, we introduce PTE (Prefill Token Equivalents), a hardware-aware TIR-efficiency metric that unifies internal reasoning and external tool-use costs while explicitly accounting for non-reusable KV-Cache and long-tool-response scenarios. Validation in a high-concurrency industrial setting indicates that PTE aligns significantly better with wall-clock latency than standard token counts, while maintaining consistent efficiency rankings across diverse hardware profiles. We conduct extensive experiments across five TIR benchmarks, quantify their PTE costs, and identify four inefficiency patterns that appear in TIR. We also discover that trajectories with higher PTE costs tend to have lower reasoning correctness, indicating that simply using more tools does not improve the quality of the answer.

1 Introduction

TIR efficiency is difficult to measure because toolcalls evict KV-Cache and long responses increase later decode costs, while token and toolcall counts miss these asymmetric hardware costs. The paper introduces PTE to unify reasoning and tool-use costs, validates its latency alignment, and identifies recurring inefficiency patterns and a negative association between cost and correctness.

  • Motivation: Toolcalls can evict KV-Cache, while long, unfiltered responses inflate context and increase the cost of subsequent decoding.These effects make decode increasingly expensive as accumulated context grows.
  • Motivation: Existing TIR efficiency metrics based on token counts or toolcall counts fail to reflect true hardware-level inference latency.They do not unify the asymmetric costs of compute-bound prefill and memory-bound decode.
  • PTE: PTE unifies internal reasoning and external tool-use costs by pricing memory-bound decode in units of one compute-bound prefill token.The metric explicitly accounts for non-reusable KV-Cache and long-tool-response scenarios.
  • Validation: PTE aligns significantly better with wall-clock latency than token-count metrics in a high-concurrency industrial setting and preserves model rankings across hardware profiles.The validation evaluates both latency alignment and ranking consistency across diverse devices.
  • Empirical findings: Experiments across five TIR benchmarks identify four inefficiency patterns: Confirmatory Tool Usage, Tool-Mixing, Lack of Tool Priors, and Tool Format Collapse.The study quantifies PTE costs for thousands of trajectories.
  • Empirical findings: Higher PTE costs tend to co-occur with lower reasoning correctness, while PTE and token counts expose different cost trajectories as context accumulates.The paper characterizes this relationship as correlational rather than causal.

2 Related Work

Tool-augmented LLM benchmarks have progressed from API selection and plan decomposition toward execution-based and complex multi-step TIR evaluation. However, existing efficiency measures still rely on token counts that do not capture hardware-level latency.

  • Benchmark evolution: Early tool-augmented LLM benchmarks emphasized API selection and plan decomposition, often abstracting away execution.Examples include BFCL, Webshop, ToolBench, and T-Eval.
  • Benchmark evolution: Later datasets introduced execution-based metrics such as API success rate, while newer TIR benchmarks cover complex multi-step tasks.The newer benchmarks include web browsing and domain-specific math and code reasoning.
  • Efficiency evaluation: Existing efficiency measures based on token counts fail to capture hardware-level latency.Related approaches also optimize inference through exploration, routing, or stopping strategies.

3 PTE: A First-Principles Efficiency Metric

PTE is a hardware-aware metric that expresses TIR inference cost in equivalent prefill-token units. It models phase-specific compute and memory costs, including KV-cache access and architectural differences.

  • Inference phases: Prefill processes input tokens in parallel and is compute-bound, whereas decode generates tokens sequentially and is constrained by HBM bandwidth for model weights and KV-cache.KV-cache retrieval cost grows linearly with cumulative sequence length.
  • PTE definition: PTE unifies compute-bound prefill and memory-bound decode costs into equivalent input-token units for a complete reasoning trajectory.The metric explicitly accounts for non-reusable KV-cache and long tool responses.
  • Cost modeling: The decode memory-access volume is modeled as SKV = 4 · nlayers · dmodel bytes under FP16 precision and converted into equivalent compute using Hardware Operational Intensity.
  • Hardware and architecture: γ represents the relative penalty of memory-bound operations compared with prefill and is refined for GQA and MLA architectures.For GQA, γ is scaled by Hkv/Hq; for MLA, dmodel is replaced with compressed dimensions.
  • Hardware and architecture: γ is a static property of a model-hardware pair, so PTE incorporates deployment-specific computational scaling rather than serving as a model-independent constant.

4 Validation: Fidelity and Robustness

The validation tests whether PTE reflects physical latency and remains robust across hardware. PTE tracks wall-clock time much better than raw token counts and preserves model-efficiency rankings across devices.

  • Validation criteria: PTE is validated against physical fidelity to real-world latency and robustness across hardware architectures.
  • Wall-clock fidelity: r = 0.9253: PTE strongly correlates with wall-clock time, while raw token counts correlate only weakly at r = −0.3750.The token-count result used N = 100 and p = 0.2558; PTE used N = 100 and p < 10^-4.
  • Hardware robustness: ρ > 0.95: model-efficiency rankings remain highly consistent across devices despite hardware scaling factors α ranging from 0.18× to 1.00×.The sensitivity analysis varied devices with different memory-to-compute ratios.

5 Experimental Setup

The experimental framework measures TIR efficiency across five benchmarks spanning mathematical reasoning and information seeking. It standardizes model evaluation, tool definitions, and token-level logging.

  • Framework: The study evaluates TIR efficiency differences using a comprehensive framework covering diverse tasks and models.
  • Benchmarks: Five benchmarks target mathematical reasoning and information seeking, including MATH500, AIME 2024/2025, SimpleQA, and WebInstruct-Verified.MATH500 and AIME use Python; SimpleQA uses Search and Visit; WebInstruct-Verified adds Python.
  • Statistical caveat: Step-level statistics are descriptive because trajectory steps are serially dependent, so their p-values are not formal hypothesis tests.
  • Models and controls: The evaluation uses tool-capable open-source models with identical system prompts and tool definitions within a unified vLLM framework.
  • Tools: The framework provides Search, Visit, and Python tools through Serper, Jina, and an open-source Python sandbox.
  • Logging: At each turn, the framework records prefill tokens, decoded tokens, and cumulative sequence length for evaluation.These statistics support the PTE cost calculation.

6 Results and Analysis

Across five benchmarks, PTE exposes large efficiency differences and task-specific reasoning behaviors that token counts can obscure. The analysis identifies distinct inefficiency patterns, including delayed tool use, tool mixing, weak tool priors, and tool-format failures.

  • Efficiency vs. Accuracy Landscape: Models with similar accuracy can have PTE values spanning an order of magnitude or more.
  • Task-Specific TIR Abilities: TIR capability is highly specialized by task and tool type rather than being a general skill.Qwen2.5-72B excels on SimpleQA but performs poorly on MATH500 and AIME with Python.
  • Different TIR Behavioral Patterns across Models: Qwen-2.5 models exhibit a “first-step effect,” delaying the first tool invocation on challenging datasets after allocating substantial initial reasoning tokens.On SimpleQA, they instead suppress reasoning to trigger early toolcalls.
  • Different TIR Behavioral Patterns across Models: +16.7% accuracy on AIME25 costs Qwen3-235B-Thinking 1.8x PTE relative to Qwen3-235B-Instruct.
  • Different TIR Behavioral Patterns across Models: On SimpleQA, the same thinking model drops 3.4% in accuracy while PTE grows by 4.2×, indicating severe over-thinking.The passage concludes that thinking mode is beneficial only when task difficulty justifies the extra compute.
  • Confirmatory Tool Usage: Confirmatory tool usage makes early overthinking increasingly expensive because delayed tool use accumulates context and raises the cost of later steps.Non-reusable KV-Cache and long tool responses make each subsequent step more expensive.
  • Tool-mixing: Most WebInstruct-Verified models rarely mix search+visit and Python, while DeepSeek-V3.1-Terminus consistently alternates between toolsets.Its tool-mixing trajectories have significantly higher PTE without an obvious accuracy gain over peers.
  • Lack of Tool Priors and Tool Format Collapse: Limited Python-tool priors can reduce accuracy and increase PTE, while fixed-schema training makes Tongyi-Deepresearch vulnerable to tool-format collapse.Small syntactic changes, such as renaming a tool or changing a query list, can break calls and harm accuracy and efficiency.

7 From Efficiency to Intelligence

Correct trajectories generally use less PTE than incorrect trajectories across models and benchmarks. The association persists after controlling for problem difficulty, but the evidence is correlational rather than causal.

  • Correct trajectories consistently exhibit lower PTE than erroneous trajectories across most models and benchmarks.Erroneous trajectories often include repeated toolcalls and extended intermediate reasoning.
  • The efficiency gap persists independently of problem hardness in difficulty-stratified analyses.
  • Unsuccessful trajectories often expand context through unhelpful tool outputs and reconciliation steps, whereas correct trajectories are more compact.
  • The negative PTE–correctness relationship is correlational and may reflect uncertainty, redundancy, or ineffective tool utilization rather than PTE causing errors.PTE may therefore serve as a coarse-grained diagnostic signal for inefficient trajectories.

8 Conclusion

The paper introduces PTE as a unified hardware-aware view of reasoning and tool-use costs. Experiments show that TIR costs vary widely and that lower PTE is associated with higher trajectory accuracy.

  • PTE unifies reasoning and tool-use costs by modeling prefill–decode asymmetry.
  • Experiments across models and tasks find TIR costs spanning orders of magnitude, largely driven by four inefficiency patterns.
  • The experiments find a correlation between low PTE and high trajectory accuracy.

9 Limitations

The paper’s limitations concern omitted real-world latency, simplified hardware modeling, and restricted empirical coverage. The generality of the link between PTE and reasoning quality remains unresolved.

  • PTE omits real-world costs such as API latency and simplifies architectural efficiency through the γ parameter.The abstraction may not capture all hardware optimizations or runtime dynamics.
  • Empirical validation covers specific tasks and models, so broader generalizability—especially the link between low PTE and high-quality reasoning—requires further study.

10 Ethical Considerations

The paper presents PTE as a hardware-aware efficiency metric and examines its behavior across hardware, models, and TIR benchmarks. It reports robustness of efficiency rankings across hardware profiles while noting that γ measures scaling behavior rather than absolute latency.

  • PTE unifies reasoning and tool-use costs by modeling prefill–decode asymmetry in tool-integrated reasoning.
  • The H100 is used as reference hardware, with HOI anchoring γ and α scaling γ relative to the H100 baseline.
  • 0.46× is the reported γ scaling for H200 relative to H100, reflecting its higher memory bandwidth.
  • α ranges from 0.18× to 1.00× across tested hardware profiles, while model efficiency rankings remain highly consistent.
  • γ measures how total inference cost scales with sequence length rather than absolute per-token latency.
  • The evaluation covers five TIR benchmarks, including mathematical reasoning, factual question answering, and tool-assisted retrieval settings.

E Quantitative Analysis of Inefficiency Patterns

The quantitative analysis measures the cost and frequency of distinct TIR inefficiency patterns and relates PTE to correctness. Inefficient trajectories generally cost more, with differences persisting after controlling for problem difficulty.

  • 1.77× to 2.42× are the measured PTE cost multipliers for identified inefficiency patterns.Frequency and cost were measured within the dominant model-task setting for each pattern.
  • Incorrect trajectories exhibit significantly higher PTE than correct trajectories at every analyzed difficulty level.
  • 793% is the largest reported PTE gap, occurring at Level 1 where failures are often associated with reasoning inefficiencies.
  • 75% less PTE is used by GPT-OSS to correctly solve a Level-5 problem than to fail on a Level-4 problem.The reported PTE values are 24,302 for the correct trajectory and 97,183 for the failure.
  • r = −0.040 and p = 0.002 indicate a significant negative PTE–accuracy association after controlling for difficulty.
  • PTE achieves r = 0.925 correlation with wall-clock latency, exceeding token-count and commercial-pricing baselines.
  • Confirmatory tool usage, tool-mixing, lack of tool priors, and tool format collapse are documented as distinct inefficiency patterns.
Loading 2604.05404v2…