PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Agents Already Know When to Call Tools -- Even Without Reasoning

Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang, Tsui-Wei Weng

arXiv:2605.09252v2cs.CL

TL;DR

Tool-use benchmarks often assume every task requires a tool, leaving the necessity decision under-studied. This paper introduces WHEN2TOOL and uses hidden-state probes to steer tool-use decisions through response prefilling. Probe&Prefill achieves a better accuracy–efficiency tradeoff than the Prompt-only and Reason-then-Act baselines.

  • Problem

    Existing tool-use benchmarks largely test correct tool use while assuming every task needs a tool, leaving the decision of when a tool is necessary systematically unstudied.

    Source: §2 · p. 3

  • Method

    The paper builds WHEN2TOOL, an 18-environment controlled benchmark, then uses a linear probe on pre-generation hidden states to prefill responses with tool-necessity steering.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Probe&Prefill achieves a strictly better accuracy–efficiency tradeoff than Prompt-only and Reason-then-Act; the best baseline reduces tool calls by only 6% at comparable accuracy.

    Source: §1 · p. 3, §7 · p. 9

  • Takeaways & Limitations

    Tool-necessity information is already encoded in hidden states, even when generation fails to translate it into calibrated decisions.

    Source: §1 · p. 2

Abstract

from arXiv · show

Tool-augmented LLM agents tend to call tools indiscriminately, even when the model can answer directly. Each unnecessary call wastes API fees and latency, yet no existing benchmark systematically studies when a tool call is actually needed. We propose When2Tool, a benchmark of 18 environments (15 single-hop, 3 multi-hop) spanning three categories of tool necessity -- computational scale, knowledge boundaries, and execution reliability -- each with controlled difficulty levels that create a clear decision boundary between tool-necessary and tool-unnecessary tasks. We evaluate two families of training-free baselines: Prompt-only (varying the prompt to discourage unnecessary calls) and Reason-then-Act (requiring the model to reason about tool necessity before acting). Both provide limited control: Prompt-only suppresses necessary calls alongside unnecessary ones, and Reason-then-Act still incurs a disproportionate accuracy cost on hard tasks. To understand why these baselines fail, we probe the models' hidden states and find that tool necessity is linearly decodable from the pre-generation representation with AUROC 0.89--0.96 across six models, substantially exceeding the model's own verbalized reasoning. This reveals that models already know when tools are needed, but fail to act on this knowledge during generation. Building on this finding, we propose Probe&Prefill, which uses a lightweight linear probe to read the hidden-state signal and prefills the model's response with a steering sentence. Across all models tested, Probe&Prefill reduces tool calls by 48% with only 1.7% accuracy loss, while the best baseline at comparable accuracy only reduces 6% of tool calls, or achieves a similar tool call reduction but incurs a 5$\times$ higher accuracy loss. Our code is available at https://github.com/Trustworthy-ML-Lab/when2tool

1 Introduction

WHEN2TOOL studies when LLM agents should use tools, and the paper finds that hidden states encode tool necessity even when generated decisions fail to reflect that signal. PROBE&PREFILL uses that signal to improve the accuracy–efficiency tradeoff.

1 Introduction

  • Prior work leaves the overcalling mechanism unresolved: Wu et al. [2025] improve calls that are already needed, while Xu et al. [2025] use oracle tools and SFT.The paper asks whether models overcall because they lack decision-relevant information or fail to act on information they already have.Source: §1 · p. 2
  • Prompt-only and Reason-then-Act provide limited control over tool calls, with hard tasks paying a disproportionate accuracy cost for each saved call.Prompt-only varies system prompts; Reason-then-Act asks the model to reason explicitly about tool necessity before acting.Source: §1 · p. 2
  • A linear classifier on the last input-token hidden state predicts tool necessity with AUROC above 0.9, including for models that fail under Reason-then-Act.The results indicate that the signal is encoded before generation but is not reliably translated into calibrated decisions.Source: §1 · p. 2
  • PROBE&PREFILL reads hidden states with a lightweight linear probe and prefills a steering sentence, with a threshold that controls the accuracy–efficiency tradeoff.The method requires no additional model reasoning and has less than 1 ms overhead; it transfers between tasks and preserves accuracy on hard tasks.Source: §1 · p. 2
  • PROBE&PREFILL reduces tool calls by 48% with only 1.7% accuracy loss; comparable-accuracy baselines reduce calls by 6%, while similar reductions incur 5× more accuracy loss.It also reduces API calls by 20–56% on the Search-o1 agentic benchmark [Li et al., 2025a].Source: §1 · pp. 2–3
  • WHEN2TOOL covers 18 environments—15 single-hop and 3 multi-hop—with 1,080 training and 2,700 test tasks across three necessity categories and three difficulty levels.The categories concern computational scale, knowledge boundaries, and execution reliability; difficulty levels range from easy to hard.Source: §1 · p. 2, Figure 1 · p. 3

2 WHEN2TOOL: A Benchmark for Tool-Call Decisions

WHEN2TOOL benchmarks whether LLMs can recognize when tools are needed, rather than assuming every task requires a tool. It uses controlled difficulty across computational scale, knowledge boundaries, and execution reliability.

2.1 Three categories of tool necessity

  • The benchmark tests three kinds of self-assessment: computational feasibility, whether the model knows the needed facts, and whether it can reliably execute sequential procedures.Hard cases use large computations, fictional or otherwise unavailable information, and lengthy procedures that exceed reliable mental tracking.Source: §2.1 · pp. 3–4

2.2 Difficulty as the decision boundary

  • Easy tasks test overcalling, medium tasks place models near the decision boundary, and hard tasks test whether models recognize when tools are necessary.Difficulty assignments were empirically validated by forcing models to answer without tools.Source: §2.2 · p. 4
  • The benchmark contains 1,080 training tasks and 2,700 test tasks across its single-hop and multi-hop environments.The single-hop environments account for 900 training and 2,250 test tasks; the multi-hop environments account for 180 training and 450 test tasks.Source: §2.2 · p. 4

2 WHEN2TOOL: A Benchmark for Tool-Call Decisions

  • WHEN2TOOL covers 18 environments—15 single-hop and 3 multi-hop—and evaluates tool-call decisions with controlled difficulty, multi-hop tasks, and zero API cost.Its multi-hop tasks require a chain of three dependent tool calls, each step using the previous step’s output.Source: §2 · p. 3, Table 1 · p. 4

3 Failure Analysis: The Limits of Prompting and Explicit Reasoning

Across six models, Prompt-only and Reason-then-Act fail to selectively reduce unnecessary tool calls: discouraging tool use also suppresses needed calls, and prompt-level control remains coarse.

3.1 Experimental setup

  • The evaluation covered Qwen3-1.7B/4B/14B/32B and Llama-3.1-8B/3.3-70B, with each experiment run three times using different seeds and mean results reported.Source: §3.1 · p. 4

3.2 Key findings

  • The accuracy cost per saved call (∆Acc −∆TC) on Qwen3-4B-Instruct worsened from −17.3 on easy tasks to −42.4 on hard tasks when switching from Default to Sparse.Hard tasks therefore lose 2.5× more accuracy per saved call, and the pattern holds across model sizes.Source: Table 2 · p. 5, §3.2 · p. 5, §3.1 · p. 5

3.1 Experimental setup

  • Prompt-only tested Force, Default, Necessary, Sparse, and No Tool instructions, while Reason-then-Act added a necessity-reasoning step before acting under each mode.Source: §3.1 · p. 5

3.2 Key findings

  • Under Default prompting, models made 2,100–4,400 tool calls across 2,250 single-hop tasks, including 864 calls on 750 easy tasks for Qwen3-1.7B.Llama-3.3-70B made 1,482 calls on easy tasks, illustrating default tool overuse even when tasks are simple.Source: §3.2 · p. 5
  • Reason-then-Act improved Qwen3-14B easy-task ∆Acc −∆TC from −14.9 to −6.6, but its hard-task cost remained −34.7.Figure 2 shows partial reduction of unnecessary easy-task calls, while necessary hard-task calls are still suppressed.Source: §3.2 · pp. 5–6, Figure 2 · p. 5
  • Reason-then-Act added generation overhead and sharply reduced accuracy on Llama-3.1-8B (79.5% to 31.2%) and Llama-3.3-70B (83.1% to 47.9%).These models narrated their intent to call tools but failed to produce valid invocations, resulting in near-zero tool calls.Source: §3.2 · p. 6
  • Prompt-only and Reason-then-Act offered few effective operating points: Necessary resembled Default, while Sparse and No Tool nearly collapsed together under Reason-then-Act.Their fixed prompt modes could not smoothly target a desired tool-call budget; Figure 2 shows the coarse tradeoff.Source: Figure 2 · p. 5, §3.2 · p. 6

4 Probing Analysis: Decoding Implicit Tool Necessity

Tool necessity is linearly decodable from pre-generation hidden states across six models, including Llama models whose tool calling collapses under Reason-then-Act.

Probing Analysis: Decoding Implicit Tool Necessity

  • A logistic-regression probe predicts necessity from concatenated last-token hidden states, using labels defined by whether tool-free answers succeed.It uses L2 regularization, trains on 900 examples, evaluates on 2,250 held-out tasks, and takes seconds on CPU.Source: §4 · p. 6
  • Despite Reason-then-Act tool-calling drops on Llama-3.1-8B (79.5% →31.2%) and Llama-3.3-70B (83.1% →47.9%), probes retain AUROC above 0.9.The hidden-state signal remains present even when these models cannot express it during generation.Source: §4 · pp. 6–7
  • AUROC 0.89–0.96 across six models shows necessity is encoded pre-generation; performance is strong at every difficulty, with medium tasks hardest.The authors identify medium tasks as the decision boundary where the model is uncertain.Source: Table 3 · p. 6, §4 · p. 6
  • The authors next use the hidden-state signal to steer tool-call behavior by injecting a short steering sentence into the model’s output.The following section examines whether this can improve tool-call decisions.Source: §4 · p. 7

5 PROBE&PREFILL: Turning Hidden Knowledge into Better Decisions

PROBE&PREFILL uses a linear probe on pre-generation hidden states to steer the model toward direct answers or tool use without fine-tuning or added reasoning. It improves the accuracy–tool-call tradeoff and is evaluated across additional settings.

5.1 Method

  • At inference, the method extracts the last-token hidden states, applies a thresholded linear probe, then prepends a steering sentence that the model may override.Hidden-state extraction adds no forward pass beyond standard prompt encoding; hard prefill instead forces the direct-answer or tool-call output format.Source: §5 · p. 7, §5.1 · p. 7

5.2 Main results

  • PROBE&PREFILL outperforms Reason-then-Act in most cases without extra reasoning tokens; on Llama, reasoning collapses tool calling while the probe retains strong performance.The authors report that the probe’s pre-generation prediction is more accurate than the model’s verbalized assessment.Source: §5.2 · pp. 7–8
  • On Qwen, PROBE&PREFILL has a strictly better accuracy–tool-call tradeoff than both baselines; on Llama, hard prefill restores control when soft prefill is partly ignored.Sweeping threshold τ provides fine-grained control over the operating point.Source: §5.2 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8
  • At τ=0.5, PROBE&PREFILL’s accuracy cost per saved call is −1.6 on easy tasks and −3.4 on hard tasks, the lowest across difficulty levels.Table 4 averages results across six models; more negative ∆Acc −∆TC means each saved call costs more accuracy.Source: Table 4 · p. 8, §5.2 · p. 8

5.3 Additional experiments

  • Additional experiments report up to 75% fewer multi-hop tool calls on Qwen, <1ms inference overhead, and 20–56% fewer Search-o1 API calls while matching or exceeding baseline accuracy.The probe also transfers to held-out environments within the same category; fine-tuning improves accuracy but does not reliably reduce calls and is orders of magnitude more expensive.Source: §5.3 · p. 9

6 Related Work

The related work contrasts WHEN2TOOL’s tool-necessity decision with benchmarks of tool-use correctness and costly efficiency interventions. It positions the method as probing latent tool-use knowledge and steering through output prefilling.

Agentic Tool-use benchmarks

  • Existing tool-use benchmarks assess whether models use tools correctly, assuming tools are needed; WHEN2TOOL instead tests whether to call a tool or solve directly.The benchmark covers the decision given the correct tool.Source: §6 · p. 9

Efficient tool calling

  • Prior efficiency research uses confidence-based fine-tuning [Xu et al., 2025] and instruction/tool-description optimization [Wu et al., 2025]; Yang et al. [2026] survey agent efficiency.Unlike these costly interventions, this work uses a seconds-to-train linear probe to leverage tool-call information in hidden states, adding negligible inference cost.Source: §6 · p. 9

Probing and controlling LLM behavior

  • This work extends probing to agentic tool-use decisions, finding models mostly know when tools are needed but fail to act, and steers by output prefilling rather than activation or weight edits.Prefilling based on a probe prediction requires no forward-pass modification and is compatible with any serving infrastructure.Source: §6 · p. 9

7 Conclusion · H Additional baseline: Compare PROBE&PREFILL with Supervised Fine-Tuning (SFT) 34

LLM agents encode reliable tool-necessity signals in their hidden states, and Probe&Prefill uses them to improve the accuracy–efficiency tradeoff without training.

7 Conclusion

  • A linear probe and response prefilling yield a strictly better accuracy–efficiency tradeoff than Prompt-only and Reason-then-Act, demonstrating a lightweight, training-free way to improve tool-use efficiency.Source: §7 · p. 9

A Benchmark environment details … A.7 Difficulty validation

This appendix documents WHEN2TOOL’s reproducible, offline design and exact-answer evaluation across environments testing computational scale, knowledge boundaries, and execution reliability. It details single-hop and chained tasks with difficulty levels intended to distinguish mental-answerable questions from tool-requiring ones, and validates that distinction with declining no-tool accuracy from easy to hard tasks.

A.1 Design principles

  • WHEN2TOOL is fully offline and deterministic, uses exact-form answers and short task descriptions, and can be extended with self-contained task generators.All tasks are generated from fixed random seeds, and simulated tool responses require no APIs, network access, or external services.Source: §A.1 · p. 13

A.2 Evaluation framework

  • The evaluation measures exact-answer accuracy and total test-set tool calls, reporting mean and standard deviation across three random-seed runs.A deterministic evaluator checks boxed answers against ground truth, with numeric tolerance, case-insensitive strings, and equivalent representations; there is no LLM-based judging.Source: §A.2 · p. 14

A.3 Category A: Computational scale

  • Computational-scale environments cover calculator arithmetic, statistics, combinatorics, matrix operations, and primality or factorization, increasing difficulty as computation grows beyond reliable mental execution.For example, CalculatorEnv varies operand magnitude, while StatisticsEnv adds larger datasets and more complex measures such as correlation.Source: §A.3.1 · p. 14, §A.3.3 · p. 15, §A.3.4 · p. 15

A.4.1 RetrieverEnv

  • RetrieverEnv tests knowledge boundaries through a required search-then-read sequence, with tasks progressing from well-known facts to obscure facts and synthetic corpus-only information.It is the only environment requiring two tool calls: search_corpus identifies a document, then read_doc retrieves its full text.Source: §A.4.1 · pp. 16–17

A.4 Category B: Knowledge boundary

  • HistoricalYearEnv and GameRuleEnv move from familiar facts to obscure or fictional ones, while HashEnv and DecodingEnv contrast potentially memorized or familiar outputs with custom, unknown transformations.Hash tasks progress to non-standard custom algorithms; decoding tasks progress from Morse and ROT13 to custom substitution ciphers with arbitrary mappings.Source: §A.4.2 · p. 17, §A.4.3 · p. 17, §A.4.4 · p. 18

A.5.1 ListManipulationEnv

  • Execution-tracking environments test whether models can reliably carry out known procedures, with ListManipulationEnv scaling from simple list operations to larger lists and 2D arrays.The operations include appending, removing, inserting, sorting, and reversing list elements.Source: §A.5.1 · pp. 18–19, §A.5 · p. 18

A.5.1 ListManipulationEnv; A.5.2 DateTimeEnv

  • List manipulation scales from small one-operation lists to large 2D matrices, while date tasks progress from within-month counting to boundary-crossing arithmetic and arbitrary day-of-week calculations.DateTimeEnv includes date addition, date differences, and day-of-week queries.Source: §A.5.1 · p. 19, §A.5.2 · p. 19

A.5.3 CodeExecutorEnv

  • CodeExecutorEnv tests whether models recognize when loops, recursion, dynamic programming, or long sequences such as Collatz exceed reliable mental tracing.Its tasks range from simple expressions to code involving 10–30+ iterations, and the tool returns captured stdout.Source: §A.5.3 · pp. 19–20

A.5.4 ScheduleEnv; A.5.5 RegexMatchEnv

  • ScheduleEnv increases interval-tracking demands from 2–3 meetings to 15+ constrained meetings, while RegexMatchEnv tests patterns whose overlapping matches and backtracking are difficult to simulate mentally.The environments use tools to find available slots or apply regex operations such as findall, match, search, and substitution.Source: §A.5.4 · p. 20, §A.5.5 · pp. 20–21

A.6 Multi-hop environments

  • The three multi-hop environments require three dependent tool calls, with each hop’s output feeding the next, and require a tool-necessity decision at every hop.They reuse the tools from their single-hop counterparts and retain the same easy-to-hard difficulty scaling.Source: §A.6 · p. 21

A.6.1 ChainedCalculatorEnv (Category A)

  • ChainedCalculatorEnv chains three arithmetic computations, making each expression depend on the preceding result.It uses evaluate_expression and contrasts easy arithmetic with hard computations on large integers and a modulo operation.Source: §A.6.1 · p. 21

A.6.2 ChainedRetrieverEnv (Category B)

  • ChainedRetrieverEnv chains three knowledge lookups, where each answer determines the next query; its corpus includes target documents and 10 distractors.The hard example uses fictional entities that exist only in the provided corpus.Source: §A.6.2 · p. 21

A.6.3 ChainedCodeExecutorEnv (Category C)

  • ChainedCodeExecutorEnv passes each of three scripts’ outputs into the next script, with hard hops involving loops, recursion, or dynamic programming.It uses run_code; the hard example combines coin-change DP, even Fibonacci summation, and longest-increasing-subsequence computation.Source: §A.6.3 · p. 21

A.7 Difficulty validation

  • No-tool accuracy averaged 69.4% on easy tasks, 54.4% on medium tasks, and 15.5% on hard tasks across six models and all 18 environments.The authors use this result to validate the intended difficulty boundary; Table 6 covers 1,080 train and 2,700 test examples.Source: §A.7 · p. 21, Table 6 · p. 22

B Detailed single-hop results

This appendix reports detailed per-model single-hop results, including full accuracy and tool-call outcomes and accuracy cost per saved call. The tables cover six models and the 15 single-hop environments.

B Detailed single-hop results

Accuracy cost per saved call

  • Table 7 compares per-model accuracy cost per saved call for Sparse, Sparse + Reason-then-Act, and PROBE&PREFILL (τ=0.5), relative to Default (Prompt-only).More negative values mean each saved call is more costly; bold marks the best per model in each column.Source: B Detailed single-hop results · p. 23, Table 7 · p. 23

Full single-hop results

C Multi-hop evaluation … E.3 Layer selection

Across chained tasks, Probe&Prefill improves tool-call and accuracy tradeoffs; the probe generalizes to held-out environments, while ablations identify effects of prefill mode, temperature, and layer choice.

C Multi-hop evaluation

  • On multi-hop tasks, the probe beat the best baseline on Qwen3-4B (85.3% vs. 83.9% accuracy; 75% fewer calls vs. 63%) and reduced Qwen3-32B calls 55% vs. 20%.On Llama-3.1-8B and Llama-3.3-70B, accuracy rose from 40.2% to 60.2% and 62.4% to 80.3%, respectively, while tool calls increased.Source: C Multi-hop evaluation · p. 24
  • Multi-hop tool necessity remains linearly decodable, with probe AUROC 0.84–0.97 across models, though Llama-3.3-70B reaches 0.804.Qwen3-4B reaches AUROC 0.966; despite its lower AUROC, Probe&Prefill improves Llama-3.3-70B accuracy from 62.4% to 80.3%.Source: Table 10 · p. 24, C Multi-hop evaluation · p. 24
  • On Qwen3-1.7B, Reason-then-Act helps plan tool chains, while Llama models collapse to near-zero tool calls and Probe&Prefill offers a smooth threshold–accuracy tradeoff.For Qwen3-1.7B, Sparse+R-t-A reaches 60.6% accuracy versus 41.3% for Sparse Prompt-only. On Qwen3-4B, sweeping τ from 0.1 to 0.9 lowers calls from 1287 to 175 as accuracy gradually falls from 82.1% to 79.6%.Source: C Multi-hop evaluation · p. 24, Table 11 · p. 25

D Out-of-distribution generalization

  • In leave-two-out tests, the OOD probe achieves comparable accuracy–efficiency tradeoffs to the in-distribution probe across models.For each category, the probe is trained on three of five environments and evaluated on all five, supporting generalization beyond training environments.Source: Figure 5 · p. 26, D Out-of-distribution generalization · p. 26

E.1 Soft vs. hard prefill

  • Soft prefill generally gives Qwen models higher accuracy than hard prefill at matched call levels, while Llama results depend on model and threshold.Hard prefill achieves 79.9% versus 69.2% for soft on Llama-3.1-8B at τ=0.1; on Llama-3.3-70B, soft stays at 88.4–89.2% across thresholds, while hard ranges from 79.0% at τ=0.1 to 33.9% at τ=0.9.Source: Figure 6 · p. 27, §E.1 · p. 27

E.2 Temperature scaling

  • Temperature changes the probe’s operating range without qualitatively changing Probe&Prefill’s advantage over prompt baselines.T=1.0 offers a wider range, T=3.0 finer control in the middle range, and T=2.0 a balance across models.Source: §E.2 · p. 28, Figure 7 · p. 28

E.3 Layer selection

  • All-layer concatenation performs best, while single-layer probes remain competitive and mid-layer probes generally outperform last-layer probes.This pattern suggests tool-necessity information is distributed across the network and emerges early before persisting through later layers.Source: §E.3 · pp. 28–29, Table 14 · p. 29

E.4 Data efficiency · E.5 Regularization strength · F Inference overhead

Probe performance remains strong with limited training data and is stable across a broad regularization range; Probe&Prefill adds less than 1% inference overhead across six models.

E.4 Data efficiency

  • Probe performance improves steadily with more training data, but marginal gains diminish beyond 50%.The training set was subsampled at 10%, 25%, 50%, 75%, and 100%.Source: §E.4 · p. 29
  • With 90 labeled examples (10%), the probe achieves AUROC above 0.81 on all tested models.Table 15 reports probe test AUROC at training fractions from 10% to 100%.Source: Table 15 · p. 29, §E.4 · p. 29

E.5 Regularization strength

  • The default λ=10000 is near-optimal across all models; performance declines modestly at λ=1 and λ=100000.The sweep uses sklearn C = 1/λ.Source: §E.5 · p. 29
  • Probe performance is stable across four orders of magnitude of regularization, from λ=10 to λ=10000.Table 16 reports probe test AUROC across λ values from 1 to 100000.Source: §E.5 · p. 29, Table 16 · p. 30

F Inference overhead

  • Inference overhead comes from extracting the last-token hidden state and applying the linear probe; the prefill forward pass is shared with standard generation.The probe applies standardization, a dot product, and a sigmoid to the extracted hidden state.Source: F Inference overhead · p. 31, Table 17 · p. 31
  • Across all six models, Probe&Prefill adds under 0.7 ms latency, less than 1% overhead.A typical prefill forward pass takes 10–100 ms and per-token generation takes 5–50 ms.Source: Table 17 · p. 31, F Inference overhead · p. 31

G Generalization to agentic search (Search-o1 benchmarks)

On Search-o1 QA benchmarks, PROBE&PREFILL often reduces search calls without sacrificing accuracy, while probes detect tool necessity with measurable accuracy on held-out data.

Evaluation setup

  • Evaluation used Qwen3-4B-Instruct on six Search-o1 open-domain QA datasets spanning single-hop, two-hop, and complex multi-hop tasks.The setup included five Prompt-only and five Reason-then-Act modes plus PROBE&PREFILL with a threshold sweep; probes trained on a 50/50 split of each dataset in seconds on CPU.Source: G Generalization to agentic search… · p. 32

Generalization results

  • On 4 of 6 datasets, PROBE&PREFILL matched or exceeded accuracy while reducing search calls more than the best baseline.TriviaQA accuracy was 69.2% versus the best baseline’s 68.8%, with 20% fewer searches versus the baseline’s 16%. On HotpotQA and Bamboogle, it exceeded all baselines in accuracy with 50–54% fewer searches; on MuSiQue, PO Force reduced searches by 56% versus PROBE&PREFILL’s 48%.Source: G Generalization to agentic search… · p. 32, Table 18 · p. 32

Probe quality

  • The in-domain probe reached AUROC 0.64–0.84, while the WHEN2TOOL transfer probe reached 0.67–0.80 on Search-o1 benchmarks.The strongest in-domain signals were on TriviaQA (0.84) and 2WikiMultihopQA (0.80), and the authors report that the transfer results confirm partial transfer of the tool-necessity signal.Source: G Generalization to agentic search… · p. 32, Table 19 · p. 32

H Additional baseline: Compare PROBE&PREFILL with Supervised Fine-Tuning (SFT)

Compared with SFT, PROBE&PREFILL reduces tool calls more consistently while avoiding weight updates and requiring far less training compute.

H Additional baseline: Compare PROBE&PREFILL with Supervised Fine-Tuning (SFT)

  • SFT training uses 900 single-hop tasks, targeting direct answers when tools are unnecessary and tool-use trajectories when they are necessary.Full-parameter fine-tuning runs for 2 epochs at learning rate 10−5 across 4 GPUs, masking nonassistant tokens; evaluation covers Qwen3-1.7B, Qwen3-4B-Instruct, and Llama-3.1-8B.Source: H Additional baseline: Compare… · p. 34
  • PROBE&PREFILL (τ=0.5) reduces tool calls by 21–38% across all models, while SFT improves accuracy by 2–3% but meaningfully reduces calls only on Qwen3-1.7B (−18%).SFT tool calls increase on Qwen3-4B and Llama-3.1-8B.Source: Table 21 · p. 34, H Additional baseline: Compare… · p. 34
  • SFT improves answers overall but does not effectively learn the tool-call decision boundary; PROBE&PREFILL works across 1.7B–70B models with seconds of CPU training and a threshold-controlled accuracy–efficiency tradeoff.The comparison is asymmetric: SFT requires hours of full fine-tuning on multiple GPUs, whereas PROBE&PREFILL trains a linear probe in seconds on CPU without modifying weights.Source: H Additional baseline: Compare… · p. 34

Related papers

Loading 2605.09252v2…