PaperlayerView exact revision on arXiv

Source-linked AI summary

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

Yiwei Yang, Haoxiang Zhang, Bingbing Wen, Yao Lu, Yuchen Wu, Lei Zhang, Julian McAuley, Pan Lu, Bill Howe

arXiv:2609.16268v1cs.CL

TL;DR

RL-trained tool-using agents can learn superficial prompt cues instead of reasoning about whether a tool is needed, and final-answer accuracy may not reveal these shortcuts. The paper tests controlled cue interventions and a decision-level tool-necessity reward; shortcuts track task competence and cue–tool alignment, while the reward suppresses them without sacrificing task performance.

  • Problem

    RL optimization may lead agents to select tools from superficial prompt features correlated with reward rather than from the task’s intrinsic tool requirements.

    Source: §1 · p. 1

  • Method

    The authors inject causally irrelevant, tool-correlated cues into controlled factual and math tasks, evaluate counterfactual tool selection, and test an LLM-judged tool-necessity reward.

    Source: §1 · p. 2

  • Results

    Shortcuts formed only for reliably learned tools and were amplified by semantic cue–tool alignment; spurious search calls reached +39.2 on GSM8K for [WEB INDEX 742].

    Source: §1 · p. 2, §5.2 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    A dense tool-necessity reward suppresses cue-driven tool use without sacrificing task accuracy, supporting explicit supervision of tool-selection decisions.

    Source: §1 · p. 2

    The study uses a controlled synthetic setting, and validating the findings in more realistic training environments remains future work.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.

1 Introduction

RL-trained tool-using agents may learn to select tools from superficial prompt cues instead of task requirements, and final-answer accuracy can miss this behavior. This work studies when these shortcuts arise and tests a decision-level reward to reduce them.

1 Introduction

  • Shortcut formation is strongly associated with task competence: across tested conditions, cue–tool shortcuts form only for tools the agent has learned to use reliably.The introduction suggests that improving tool capability may also increase susceptibility to shortcut learning.Source: §1 · p. 2
  • Search-semantic cues induced up to +39.2% spurious search calls on math tasks, while code-semantic cues produced no analogous effect despite identical group imbalance.The reported asymmetry tracks whether the agent had learned the associated task reliably.Source: §1 · p. 2
  • A swapped-cue experiment found that code-semantic cues paired with the well-learned factual task produced only marginal spurious search use, with ∆Search ≤3.5%.The paper reports that semantic alignment between cue and tool amplifies the effect.Source: §1 · p. 2
  • A dense tool-necessity reward uses an LLM judge to evaluate each tool call and suppresses spurious use without sacrificing task accuracy.The reward provides per-decision feedback independent of surface cues.Source: §1 · p. 2
  • The study uses a controlled synthetic setting to isolate shortcut formation and identifies validation in more realistic training environments as future work.The authors frame the work as revealing a failure mode in RL-trained agents, not as validating it in naturalistic settings.Source: §1 · p. 2

2 Problem Formalization

The section defines tool-use shortcuts as cue-driven selection of a functionally inappropriate tool and operationalizes them through counterfactual tool-call rates. It frames tool selection as an intermediate policy decision, distinct from final-answer prediction.

2.1 Spurious Correlations

  • Group robustness characterizes a spurious attribute as correlated with the label but not causally related to the task, and compares average risk with worst-case group risk.The gap between these risks quantifies reliance on the shortcut in the described framework.Source: §2.1 · p. 2
  • Figure 1 illustrates a search-semantic cue learned alongside web search for factual questions, then triggering unnecessary search when appended to a math question requiring code.The counterfactual example depicts cue-driven selection replacing reasoning about tool need.Source: Figure 1 · p. 3

2.2 Spurious Correlations in Tool Selection

  • For tool-using agents, the spurious correlation lies in the intermediate decision of which tool to select, rather than in the final prediction..Source: §2.2 · p. 3
  • An agent selects a tool according to its policy and receives a binary reward; the selected tool’s functional utility is its task utility.The formulation separates tool-selection probability from the utility of using that tool for the task.Source: §2.2 · p. 3
  • Unnecessary tool calls can persist under rewards based only on final-answer correctness, because an agent may ignore the tool result and still answer correctly.This is why the operationalization measures tool-selection behavior directly.Source: §2.2 · p. 3
  • Tool spurious correlation means a cue raises selection probability for an inappropriate tool that provides no advantage on the underlying task.The definition does not require the unnecessary call to actively harm reward; an agent may ignore its result and still answer correctly.Source: §2.2 · p. 3
  • Counterfactual groups compare each question with and without the cue, isolating the cue’s effect on tool selection.A positive ∆ToolY-N when the tool is not needed indicates cue-driven selection consistent with the definition.Source: §2.2 · p. 3

3 Synthetic Dataset

The paper constructs controlled synthetic datasets that vary injected cues while holding dataset composition fixed. It evaluates cue-driven tool selection through counterfactual variants of questions, including held-out benchmarks.

3 Synthetic Dataset

  • The controlled dataset design fixes composition and varies the injected cue to isolate its relationship with tool-selection behavior.Training combines 500 Natural Questions examples with 500 DeepMath-103k examples, for 1,000 total.Source: §3 · p. 4, §3.1 · p. 4

3.1 Training Set Construction

  • Training pairs factual questions from Natural Questions with web search and math questions from DeepMath-103k with Python for computation.The balanced 1,000-example training set keeps compute manageable across controlled runs.Source: §3.1 · p. 4

3.2 Cue Design

  • Cues are causally irrelevant to tool necessity; search-semantic examples include [WEB INDEX 742], [REFERENCE], and the <answer> format instruction.Code-semantic examples include a Jupyter cell header, a Python shebang, and [CODE].Source: §3.2 · p. 4

3.3 Counterfactual Evaluation Groups

  • Counterfactual evaluation holds each test question fixed while comparing cue-present and cue-absent variants; their tool-call-rate difference measures cue-driven selection.The evaluation also uses held-out factual 2Wiki and math GSM8K benchmarks with counterfactual variants.Source: §3.3 · pp. 4–5

4 Method

The method adds explicit feedback to tool-selection decisions, which standard final-answer rewards leave unsupervised. An LLM judge estimates whether each call is necessary, and unnecessary calls receive a per-step penalty.

4.1 Training with Tool-Necessity Reward

  • Standard RL training rewards only the final answer, leaving tool-selection decisions unsupervised and unpenalized when the answer is correct.The dense reward is introduced to provide explicit feedback on each tool-selection decision.Source: §4.1 · p. 5
  • A tool call is necessary when a competent model cannot answer correctly without it; convenience calls, such as trivial arithmetic, are unnecessary.Source: §4.1 · p. 5
  • A GPT-5 Nano judge estimates necessity using the original question, tool type, and tool input, assessing need for the tool rather than input plausibility alone.For example, web search on a math problem is judged unnecessary regardless of whether the search query is well formed.Source: §4.1 · p. 5
  • The total trajectory reward combines task reward with a per-step penalty for unnecessary tool calls.The penalty is applied only to unnecessary calls, with α set to 0.5.Source: §4.1 · p. 5

5 Experimental Results

Across the tested conditions, cue-driven tool use was strongest when cues aligned semantically with a tool the agent had learned to use reliably. Tool-necessity rewards suppressed this behavior without reducing task accuracy, while shortcut strength varied with cue–task correlation.

5.1 Experimental Setup

  • Agents were trained with GRPO in VERLTOOL, emitting Python and search calls through structured tags.Tool results return as context for subsequent generation.Source: §5.1 · p. 5

5.2 Main Results: Shortcuts Are Selectively Learned

  • ∆Search reached +39.2 on GSM8K for [WEB INDEX 742], while code-semantic cues produced no meaningful Python-call increase on NQ, revealing an asymmetry despite identical group imbalance.The main results report search-cue effects on math tasks and limited effects from code cues on factual tasks.Source: Table 1 · p. 6, §5.2 · p. 6, Table 2 · p. 6

5.3 Task Competence and Shortcut Formation

  • NQ validation accuracy rose from ∼45% to ∼75%, while DeepMath stayed near ∼45–50%; shortcuts appeared only for tools learned reliably in the tested conditions.The search tool was learned on NQ and developed a spurious cue association, whereas the Python tool was not learned on DeepMath and no analogous association emerged.Source: §5.3 · p. 7, Figure 2 · p. 7

5.4 Disentangling Task Competence from Semantic Alignment

  • With competence present but semantic alignment absent, code cues produced at most +3.5% ∆Search, whereas search cues on poorly learned math produced no meaningful Python use (∆Py ≤3.2%).Together, the swapped-cue results suggest competence enables shortcut formation while semantic alignment modulates its strength.Source: §5.4 · p. 7, Table 3 · p. 8, Table 4 · p. 8

5.5 Tool-Necessity Reward Reduces Spurious Tool Use

  • Tool-necessity rewards reduced cue-driven tool selection to near zero across conditions while preserving or improving accuracy relative to the no-cue baseline.The reported result supports explicit supervision of tool-selection decisions without a task-performance trade-off.Source: §5.5 · p. 8

5.6 Effect of Cue–Task Correlation Strength

  • Spurious search use increased with cue–task correlation, emerging clearly at 80%+ and largely vanishing at 70% and below.Whether this threshold generalizes across cues and training scales remains open.Source: §5.6 · p. 8, Table 5 · p. 9

6 Related Work

Prior work studies spurious correlations and RL-trained tool-use policies, while this paper identifies cue-driven tool selection as a failure mode when rewards focus on final answers. Its tool-necessity reward complements approaches aimed at reducing excessive calls for efficiency.

6.1 Spurious Correlations

6.2 LLM Agents and Tool Use

  • RL tool-use approaches commonly reward only final answers, leaving tool-selection decisions unsupervised; this work targets cue-driven selection rather than call-cost efficiency (Wang et al., 2025b).Source: §6.2 · p. 9
  • The tool-necessity reward complements efficiency-focused methods by targeting cue-driven spurious use rather than reducing unnecessary calls for cost savings (Wang et al., 2025b).Source: §6.2 · p. 9

7 Conclusion

The conclusion links shortcut formation to task competence, cue–tool semantic alignment, and cue–task correlation strength. It reports that a dense tool-necessity reward suppresses shortcuts without sacrificing task performance, while calling for validation at larger scale and in naturalistic training.

  • Shortcut formation largely disappears below ∼80% cue–task correlation, and a dense tool-necessity reward eliminates shortcuts without sacrificing task performance.Source: §7 · p. 9
  • Shortcut formation is shaped by task competence and semantic alignment between the cue and tool, coupling capability with shortcut vulnerability.Source: §7 · p. 9
  • The findings come from the studied settings; validating them at larger scale and in naturalistic training remains future work.Source: §7 · p. 9

Ethics Statement

The study uses controlled synthetic datasets built from public benchmarks and involves no human subjects. The authors note potential LLM-judge bias and possible deployment costs, latency, and reliability effects from unaddressed shortcuts.

  • The study examines RL-trained tool-using agents with controlled synthetic datasets built from publicly available benchmarks; no human subjects were involved.Source: Ethics Statement · p. 10
  • The tool-necessity reward relies on GPT-5 Nano as an LLM judge, whose inherent biases mean it is a practical design choice rather than a definitive standard.Source: Ethics Statement · p. 10
  • If left unaddressed in deployed agents, shortcut behaviors could lead to unnecessary computational costs, increased latency, and degraded reliability.Source: Ethics Statement · p. 10

A.1 Base Model

The experiments use Qwen2.5-7B-Instruct, a 7B-parameter decoder-only transformer instruction-tuned through SFT and RLHF. RL training starts from that checkpoint without additional supervised fine-tuning.

  • The base model is Qwen2.5-7B-Instruct, a 7B-parameter decoder-only transformer instruction-tuned via SFT and RLHF by Alibaba.Source: §A.1 · p. 13
  • No additional supervised fine-tuning is applied to the base model.Source: §A.1 · p. 13
  • RL training starts directly from the instruction-tuned checkpoint.Source: §A.1 · p. 13

A.2 Training Hyperparameters

Cue-condition experiments share hyperparameters, and the appendix describes separate GPT-5 Nano judges for answer correctness and tool necessity.

A.2 Training Hyperparameters

  • Each cue condition is trained in a separate run using the same hyperparameters.This keeps training settings consistent across cue conditions.Source: Table 6 · p. 13

A.3 Judge Prompts

  • Both training judges use GPT-5 Nano: one evaluates answer correctness and the other tool necessity.The answer judge supplies the task reward; the necessity judge supplies the necessity reward.Source: §A.3 · pp. 13–14
  • The answer-correctness judge assigns a binary task reward by comparing the final answer with ground truth after normalization.It accepts equivalent numeric, textual, mathematical, or multiple-choice answers and returns false when no clear final answer is present.Source: §A.3 · p. 13
  • The tool-necessity judge marks a call necessary only when a competent model cannot answer correctly without that tool; convenience uses do not qualify.It judges necessity for the original question, not whether the tool input looks reasonable on its own.Source: §A.3 · p. 14

Related papers

Loading 2609.16268v1…