Source-linked AI summary
Tunable Tool-Call Rates in LLM Agents via Representation Steering
Yuqi Chen, Vincent Siu, Yang Liu, Dawn Song, Chenguang Wang
TL;DR
LLM agents often mismanage tool calls, incurring unnecessary cost or answering tool-dependent questions incorrectly. This paper uses a training-free residual-stream direction to control call propensity at inference time, achieving knowledge-selective tool use and nearly doubling open-domain QA accuracy while tracing a cost–accuracy Pareto frontier.
Problem
LLM agents both under-use tools on factual questions they cannot answer from memory and over-use tools that add cost without improving correctness, motivating lightweight inference-time control without retraining or prompt rewriting.
Method
The method extracts a single residual-stream direction from the model’s tool-calling preference signal and adds it during inference to bidirectionally adjust tool-use propensity without training or prompt changes.
Results
0.29 to 0.56 open-domain QA accuracy: live tool execution shows that steering traces a cost–accuracy Pareto frontier while induced calls concentrate on questions the model cannot answer.
Takeaways & Limitations
The direction provides continuous, knowledge-selective, tool-general control over whether agents call tools while leaving tool selection largely undisturbed.
Takeaways & Limitations
The method controls whether a tool is called, not the quality of tool execution; strong positive steering can also corrupt tool-call formatting.
Abstract
from arXiv · showhide
Deciding whether to call a tool is a core competence of an LLM agent, and a costly one to get wrong: needless calls add latency, accrue cost, and may trigger irreversible side effects, while missing calls leave the model confidently wrong on questions it could only answer through tool-calls. Models manage this balance poorly, both over-using and under-using tools. Existing methods such as post-training and prompt engineering are expensive and difficult to modify at inference time. We show that whether an instruction-tuned model calls a tool can be controlled by a single linear direction in its residual stream, extracted without any training from the model's own tool-use preference signal and turned into an inference-time intervention with no prompt change. Adding the direction with strength $α$ moves the call rate monotonically from near $0\% $ to over $90\%$ while keeping calls well-formed. The steering works in both directions: dialing it down suppresses calls, and dialing it up induces new calls that land precisely on the questions the model cannot answer from its own knowledge. We also show that the direction generalizes to unseen tools with strength comparable to each tool's own direction and without favoring any specific tool choice. With live tool execution, a single sweep of the steering traces a cost/accuracy Pareto frontier and nearly doubles open-domain QA accuracy ($0.29 \! \rightarrow \! 0.56$); the same recipe transfers across a diverse range of models spanning dense, MoE, and multimodal architectures, without any training. Our code is publicly available at https://github.com/YuqiChen4188/Steering-Tool-Use-Propensity.
1 Introduction
The paper targets poor balance between under- and over-use of tools by introducing a lightweight inference-time control mechanism. A single representation-space direction enables bidirectional, continuous tool-use steering without retraining or prompt changes.
- Motivation: Tool-use mistakes trade off reliability against latency, cost, and potentially irreversible side effects.Skipped calls can produce confidently wrong answers, while needless calls may not improve correctness.
- Motivation: Existing fine-tuning and retrieval-based approaches make tool-use selectivity expensive or difficult to customize at inference time.
- Contribution: A single residual-stream direction, identified with difference-of-means, controls whether models call tools across different models.The intervention encourages or limits tool use without training or prompt modification.
- Contribution: The direction is knowledge-selective and tool-general, inducing calls on questions the model cannot answer while transferring to held-out tools without substantially changing tool choice.
- Contribution: 0.29 to 0.56: live tool execution raises open-domain QA accuracy while steering traces a cost–accuracy Pareto frontier.The reported improvement occurs at roughly one search call per question.
2 Method
The method estimates tool-calling propensity from the model’s native tool-call token, extracts a difference-of-means residual direction, and adds it during inference. A scalar coefficient provides bidirectional control, while projection interventions test whether the direction carries the decision.
- Pipeline: The training-free pipeline measures tool-use propensity, extracts a multi-tool difference-of-means direction, and adds it during inference.It uses forward passes and an inference-time intervention rather than model training.
- Propensity proxy: The tool-exclusive token t⋆ shared across available tools provides a tokenization-clean signal for whether the model will call anything.Its first-position log-probability is computed from a single forward pass without generation.
- Direction extraction: Queries are divided into high- and low-propensity pools, whose residual-stream mean difference defines the steering vector at each layer.The high-propensity pool mixes question types, yielding a general call-propensity direction rather than a tool-specific one.
- Inference-time steering: At inference, adding αvℓ to each residual position encourages or suppresses tool use depending on the coefficient’s sign.Negative α suppresses calls, α = 0 preserves the base model, and positive α encourages calls.
- Intervention analysis: Projection interventions clamp or remove the residual component along the normalized direction to test whether that projection carries the tool-use decision.
- Tool generalization: The multi-tool direction is applied unchanged to held-out tools and compared with separately extracted per-tool vectors after norm rescaling.The held-out set includes translation, weather, unit conversion, email, SQL, and stock lookup.
3 Experiments
The experiments test whether training-free representation steering can control tool use across tasks, tools, and model architectures while improving the cost–accuracy trade-off. Steering monotonically changes call rates, targets questions needing external knowledge, improves live-search accuracy, and generalizes across models.
- Main results: Tool-call rates rise monotonically from near zero to 0.79–1.0 as α increases from −2 to +3 across search, calculator, and Python environments.Calls remain well-formed across this range except that the highest-baseline environment begins producing malformed calls at α=+3.
- Main results: Positive steering concentrates induced search calls on low-popularity PopQA questions that the model cannot answer, rather than on popular questions it already knows.Without steering, accuracy is 29% and search calls are nearly absent; first-token call tendency is nearly flat across popularity deciles.
- Main results: Accuracy rises from 0.29 with no searches to 0.56 at about 1.1 searches per question on live-search PopQA, tracing a cost–accuracy Pareto frontier.Combining steering with a search-heavy prompt reaches 0.58.
- Generalization across models: Across dense, MoE, and multimodal models, α=−2 suppresses every model’s call rate to 0.00 despite baseline rates ranging from 0.07 to 0.83.The evaluation covers Qwen3-4B-Instruct-2507, Qwen3-8B, Qwen3-30B-A3B-Instruct-2507, Gemma-4-E4B-it, and gpt-oss-20b.
- Generalization across models: Steering primarily changes whether a call occurs while preserving model-dependent routing patterns across search, calculator, and Python.Qwen models approach balanced routing at α=+2, whereas Gemma and gpt-oss rarely select Python; large positive α can produce invalid or unrecognized tool names.
- Generalization across models: Live-search accuracy improves across models from 0.18–0.34 with zero searches to 0.44–0.52 at 0.75–1.2 searches per question.These gains are 1.5–2.5×, while settings that add searches without improving accuracy are dominated and excluded from the frontier.
4 Analysis of the Tool-Call Direction
The extracted direction transfers across held-out tools and models, preserves tool routing, and provides strongest, most reliable control when applied in a mid-late layer. Projection tests support a causal role for the direction, while clamping effects vary by model.
- Cross-model reproducibility: Across five models, steering consistently moves call rates from near zero to near one, despite differing distributions across search, calculator, and Python.The models span dense, MoE, and multimodal architectures.
- Tool-general, not tool-specific: The multi-tool direction substantially suppresses calls for six held-out tools, outperforming tool-specific directions for five and coming within 12% for SQL.The held-out tools are translation, weather, unit conversion, e-mail, SQL, and stock-price lookup.
- Routing under steering: Increasing steering strength raises overall call rates while largely preserving task-appropriate tool selection.Across Qwen models, search, calculator, and Python call rates increase together; Gemma and gpt-oss rarely select Python.
- Cost–accuracy trade-off: Non-dominated configurations trace cost–accuracy Pareto frontiers under live search, with each point combining a system prompt and steering coefficient.The number of plotted points differs because only non-dominated configurations are shown for each model.
- Operating layer and steering strength: At the operating layer, propensity spans 37 nats across α ∈ [−4, 4], with about 80% of the range reached at α=±2.Control is strongest and monotone in a mid-late band; later layers show larger but less reliable swings.
- Projection interventions: Clamping monotonically increases tool use in both models, but call rates rise from 0.01 to 1.00 for Gemma-4-E4B versus only 0.01 to 0.16 for Qwen3-4B.Directional ablation produces call rates of 0.11 and 0.90, consistent with corresponding clamp-curve locations.
5 Related Work
Prior work studies tool-use policies, internal representations, and steering, but existing tool-selection approaches largely rely on training or prompting. This paper instead applies training-free inference-time manipulation to whether-to-call behavior while distinguishing it from tool choice.
- Contribution relative to prior work: This paper identifies a shared whether-to-call component alongside tool- and harness-specific structure.The distinction separates deciding whether to call from deciding which tool to select.
- Representation steering: Representation-steering work extracts directions from contrastive data and modulates target behaviors through vector addition or orthogonalization.Earlier studies report steering vectors for refusal and truthfulness, while later work complicates a purely one-dimensional view.
- LLM tool use and deciding when to call: Tool-use research extends language models with external functions, APIs, and environments for information, computation, and domain expertise beyond parametric memory.Dedicated evaluations find unnecessary or incorrect calls, motivating benchmarks for deciding whether and which tool to use.
- LLM tool use and deciding when to call: Prior methods make agents more selective through fine-tuning, retrieval-augmented systems, or prompting, which are costly or difficult to customize at inference time.This paper contrasts those approaches with inference-time manipulation of internal representations.
6 Conclusion
The paper concludes that a single residual-stream direction can continuously control tool calling across models, tools, and baseline usage patterns. It connects this control to knowledge-selective calling and cost–accuracy trade-offs, while noting important execution and formatting limitations.
- Conclusion: A single linear residual-stream direction, extracted from forward passes, moves call rates continuously from near zero to near one without training or prompt changes.The intervention is applied at one mid-to-late layer.
- Conclusion: Positive steering concentrates induced calls on questions the model cannot answer independently and preserves which tool it selects.The method also transfers to six held-out tools at or near the strength of their separately extracted directions.
- Conclusion: Under live tool execution, sweeping the coefficient traces a cost–accuracy Pareto frontier and nearly doubles open-domain QA accuracy.The same procedure transfers to five models from three vendors with baselines ranging from under-use to over-use.
- Limitations: The method controls whether to call but not tool-execution quality, may require bypassing reasoning spans, and can corrupt call formatting under strong positive steering.These limitations constrain deployment beyond the decision to initiate a tool call.
A Prompt Templates
The experiments use fixed system-prompt templates across models, with a neutral prompt as the default and a search-heavy prompt supplying an over-use condition. User turns contain only the evaluation question.
- Prompt templates: All experiments use one of two identical system-prompt strings for every model.The neutral prompt is used unless otherwise stated.
- Neutral prompt: The neutral prompt permits tool use when helpful but does not require it.It frames tool calling as optional assistance for answering the user’s question.
- Search-heavy prompt: The search-heavy prompt encourages web search because long-tail factual questions are easy to get wrong from memory.It supplies the over-use pole for the PopQA cost–accuracy frontier.
- User turns: User turns present the evaluation question verbatim without additional instructions or few-shot examples.This keeps the user-side prompt content minimal.
B Tool Harness Details
The harness presents native tool schemas to the model, including six held-out functions with required arguments and task-specific descriptions.
- Tool schemas: The harness passes tool schemas through the chat template in standard JSON function-calling format.Descriptions are given verbatim, and listed arguments are marked required.
- Held-out tools: The held-out tools cover translation, weather, unit conversion, e-mail, SQL, and stock lookup.The main paper names these tools by task, while the harness exposes their function identifiers.
- Tool schemas: The weather function accepts location and when, and returns current or forecast weather for a location.
- Tool schemas: The unit-conversion function accepts value, from_unit, and to_unit to convert quantities between measurement units.
- Tool schemas: The e-mail function accepts to, subject, and body to send an irreversible message on the user’s behalf.
- Tool schemas: The SQL function runs a read-only query against company tables, while stock lookup retrieves the latest price for a ticker.
C Sampling and Query Construction
The study samples PopQA questions across entity-popularity deciles using a disjoint extraction pool, then compares propensity and realized calls across popularity groups.
- Extraction pool: The extraction pool is uniform over PopQA entity popularities and disjoint from all evaluation sets.
- Popularity-decile study: The popularity study draws 40 questions per decile and renders them in a single-tool search environment under a neutral prompt.
- Popularity-decile study: The first-token panel measures mean log p(t⋆) without generation, while the call-rate panel uses sampled generation grouped into tail, mid, and head.
- Popularity-decile study: The projection study compares 100 rare-tail and 100 head questions in the same environment.
D Hyperparameters and Protocols
The protocol balances contrastive pools by question type, measures calls with fixed sampling settings, and defines malformed or degenerate outputs explicitly.
- Contrastive pools: The top and bottom quantiles are the top and bottom 10% by s(q) within each question type, keeping both pools balanced.
- Generation: Call rates use temperature 0.7, top-p 0.8, at most 384 new tokens, and a fixed seed for sweeps.
- Failure criteria: A call is malformed when its payload names no recognized tool, identified by parsing the tool name from the call payload.
- Failure criteria: A response is degenerate when empty or when it repeats a short text window at least five times near its start.This criterion determines the reported edge of the operating window.
E Projection-Intervention Curves
Projection clamping traces monotonic call-rate curves, while ablation falls on the same intervention dial and removes knowledge selectivity in the over-user.
- Projection interventions: Clamping h′ = h + (c−h·v̂)v̂ fixes the projection onto v̂ at c = µ+tσ, and ablation is the c=0 case.The target uses baseline projection mean µ and standard deviation σ over the evaluation set.
- Projection-intervention curves: The over-user rises steeply and saturates above +2σ, whereas the under-user remains shallow across the same baseline range.
- Knowledge selectivity: Ablation reduces the tail/head call-rate gap from +0.26 to +0.02, with both groups rising near ceiling.Baseline rates are 0.95 for rare-tail and 0.69 for head questions; after ablation they are 0.99 and 0.97.
- Knowledge selectivity: The ablation result indicates that the projection carries the knowledge-dependent modulation of the call decision.