Over-Searching in Search-Augmented Large Language Models

1 Introduction

• Over-searching is excessive search-tool use when searching cannot improve response quality, such as when a model already knows the answer or a query is fundamentally unanswerable.It can raise cost and degrade quality by introducing misleading context.Source: §1 · pp. 1–2
• Prior work studied uncertainty and refusal in models without tools, leaving open how retrieval and tool-use training affect decisions to search, answer, or abstain.The introduction contrasts base models that abstain on problematic queries with search-augmented models that may search unnecessarily.Source: §1 · p. 1
• Search improves answer accuracy on answerable queries but harms abstention accuracy on unanswerable ones, with over-searching most pronounced in reasoning models, noisy retrieval, and multi-turn conversations.Search behavior can “snowball” across turns, while directly retrieved negative evidence substantially improves abstention.Source: §1 · p. 2
• The study examines over-searching across three unanswerable-query types, model types, local RAG and web search, and single- and multi-turn interactions.The unanswerable types include Answer Unknown, False Premise, and Underspecified Context.Source: §1 · p. 2

2 Related Work

• Tool-use efficiency remains largely underexplored compared with reasoning efficiency, motivating analysis of how search depth and evidence quality affect efficiency and abstention in tool-augmented reasoning models.The related work notes that outcome-reward training can encourage longer reasoning and inference inefficiency such as over-thinking.Source: §2 · p. 2
• Prior abstention research has characterized refusal and proposed improvement techniques mainly in static settings without external tools.The cited work also reports that misleading or insufficient context and reasoning fine-tuning can degrade abstention.Source: §2 · p. 2
• Concurrent studies examine search-augmented models under ambiguous queries and user interaction, whereas this work considers the broader unanswerable scenario beyond ambiguity.Source: §2 · p. 2

3.1 Defining Over-Searching

• Over-searching occurs when marginal improvement in aggregate correctness diminishes or approaches zero while search costs continue to accumulate.Instance-level assessment is difficult because retrieval noise can cause correct answers for wrong reasons or shifts between correct and incorrect states.Source: §3.1 · p. 3

3.2 Measuring Over-Searching

• As maximum search turns rise from 0 to 19, o4-mini’s answer accuracy improves early and plateaus, abstention accuracy degrades, and TPC rises steadily.This pattern indicates continued searching after it is helpful; TPC rises when added search increases compute without improving correctness.Source: Figure 2 · p. 3, §3.1 · p. 3
• Dual accuracy evaluates correct answers on answerable queries and appropriate abstentions on unanswerable queries.Abstention includes withholding a direct answer through uncertainty, caveats, or other noncommittal responses.Source: §3.2 · p. 3
• TPC is expected compute cost per correct response, with lower values better; its cost counts generated tokens, input tokens weighted λ = 0.25, and search calls weighted µ = 500.Source: §3.2 · pp. 3–4
• Three independent language-model judges showed 89.4% agreement for answer accuracy and 92.3% for abstention accuracy, while human annotations aligned with judge decisions at 84%.GPT-4o-mini is the default judge unless otherwise noted.Source: §3.2 · p. 4

4 Experimental Setup

• OverSearchQA contains 1,188 balanced answerable and unanswerable queries, addressing the lack of abstention-focused benchmarks for search-augmented LLMs.Its construction filters unanswerable questions, finds similar answerable counterparts, and validates the answerable items.Source: §4 · p. 4
• Answerable and unanswerable items are drawn from similar embedding neighborhoods and question length is controlled to reduce dataset-artifact explanations for over-searching.The paper reports similar length distributions and substantial semantic overlap between the query groups.Source: §4 · p. 4
• OverSearchQA covers Answer Unknown, False Premise, and Underspecified Context; it excludes stale-information queries because evaluation assumes access to up-to-date search results.Source: §4 · p. 5
• The study evaluates open-source and API-based models with and without search, using shared retrieval infrastructure and up to 10 search calls per query.Wikipedia with E5-base is the default retriever; reasoning effort is set to default for the specified reasoning models.Source: §4 · p. 5

5.1 Search Augmentation Harms Abstention

• 24.0% average answer-accuracy gain and 12.8% abstention-accuracy loss with search; Underspecified Context had the worst abstention degradation.Source: §5.1 · p. 5, Table 2 · p. 6
• Deep Research reached 38.9k TPC—over 221× the base configuration—as greater complexity increased answer accuracy but degraded abstention.Source: §5.1 · pp. 5–6, Figure 4 · p. 6
• Abstention failure—answering unanswerable queries—had the highest TPC for most models, as repeated searches accumulated cost without correctness.Source: §5.1 · p. 6, Figure 5 · p. 6

5.2 Retrieval Matters

• C5 produced 3.6× average TPC versus Wikipedia-Latest yet second-best abstention, while Web Search had the best answer accuracy but lower abstention.Source: §5.2 · p. 7, Table 4 · p. 7
• Only 13–22% of retrieved content for unanswerable queries was negative evidence; abstention was near-perfect with only negative evidence but fell sharply when positive evidence dominated.Source: §5.2 · p. 7, Table 5 · p. 7

5.3 Snowball in Multi-turn Conversations

• Across 1–9-turn contexts, TPC rose with conversation length; abstention stayed stable or slightly improved after unanswerable histories but degraded after answerable and mixed histories.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4 Mitigating Over-Searching

• Prompt strategies gained an average 11.5 percentage points in abstention accuracy; few-shot helped most but reduced answer accuracy most, while self-evaluation balanced gains with modest loss.Source: §5.4 · pp. 8–9, Table 6 · p. 8
• Adding 10 synthetic negative documents improved abstention by only 3.6% on average; poor ranking and dilution may limit impact, and architectural changes remain future work.Source: Table 6 · p. 8, §5.4 · p. 9

• Search-augmented LLMs can invoke search unnecessarily, increasing computational costs and potentially degrading response quality.Source: §6 · p. 9
• The evaluation finds a trade-off: search improves accuracy on answerable queries but impairs abstention on unanswerable ones, especially with noisy retrieval and across multi-turn conversations.Source: §6 · p. 9
• TPC measures over-searching inefficiency, while the paper evaluates mitigation strategies and releases OverSearchQA for further research.Source: §6 · p. 9

7 Limitations

• The study focuses on evaluation and analysis, leaving targeted model training and retrieval-system changes for future exploration.Its unanswerable queries come from existing benchmarks rather than real-world search logs, so their distribution may differ from deployment and may become outdated.Source: §7 · p. 9

A Over-searching Definition & Discussion

• Over-searching is defined as additional search operations whose cost is disproportionate to correctness gains, including cases where tokens grow faster than accuracy.The authors use an increasing Tokens Per Correctness (TPC) value as an indication of over-searching.Source: A Over-searching Definition & Discussion · p. 15

A.1 Subjectivity of Over-Searching Thresholds

• A universal over-searching threshold is impractical because efficient search depth depends on application priorities, from responsiveness to exhaustive coverage.The discussion contrasts casual conversation, business intelligence, medical diagnosis, and legal research.Source: §A.1 · p. 15

A.2 Over-Searching as Marginal Return on Investment

• 0.874% ROI for the first search accompanies a 2.45% overall-accuracy gain versus no search, while Max 5 onward yields negative or near-zero ROI.Turns 15 and 17 show strong negative ROI, where added computational cost coincides with declining accuracy.Source: §A.2 · p. 15, Table 7 · p. 15

A.3 Empirical Demonstration of Over-Searching

• 52.3% to 46.3% abstention accuracy as o4-mini’s maximum search cap rises from 0 to 19, while answer accuracy peaks near 7 searches and TPC exceeds 9k tokens per correct response.The figure reports TPC rising from 722 as the search cap increases.Source: Figure 7 · p. 16
• 70.5% more searches than necessary are performed on average, comparing models’ actual search counts with the minimum needed for correctness.The estimate uses only queries the model answered correctly and treats the first correct state as the optimal stopping point.Source: §A.3 · pp. 15–16, Table 8 · p. 16

A.4 Measuring Over-Searching with TPC

• TPC captures cost-performance trade-offs across the entire dataset, whereas the optimal-search calculation is expensive and excludes queries with incorrect final responses.Higher TPC indicates token costs are growing without proportional correctness gains; reducing TPC implies fewer tokens and searches for correct outcomes.Source: §A.3 · p. 16, §A.4 · p. 17

B.1 Dual Accuracy

• Dual Accuracy separately reports answer accuracy on answerable queries and abstention accuracy on unanswerable queries.The authors present this symmetric view to make the two decision regimes interpretable without relying on aggregated metrics.Source: §B.1 · p. 17

B.2 TPC Parameter Selection

• TPC uses fixed reference costs of λ = 0.25 for input-to-output token cost and µ = 500 equivalent output tokens per search call.The values are derived from public model and search API pricing to standardize comparisons despite differences in individual model pricing.Source: §B.2 · p. 17

B.3 TPC vs. Other Metrics

• TPC aggregates total token expenditure per correct answer across a dataset, while marginal ROI measures accuracy gains against incremental token costs at each turn.Marginal ROI is strongly positive for Max 1 and Max 3, then becomes erratic and frequently negative; TPC supports comparison without aligning search-turn boundaries.Source: §B.3 · pp. 17–18
• TPC is presented as tool-aware and dataset-level stable, unlike CoP’s per-problem infinities when items are never solved.The comparison discusses CoP as proposed by Erol et al. (2025).Source: §B.3 · p. 18

C.1 Inter-Judge Agreement

• 89.4% average pairwise agreement for answer accuracy and 92.3% for abstention accuracy across three independent LLM judges.The agreement matrix covers Llama-4-Scout, Llama-4-Maverick, and GPT-4o-mini.Source: Figure 8 · p. 18, §C.1 · p. 19

C.2 Human Annotation Validation

• 84% agreement between a human annotator and the default LLM judge on 100 responses to unanswerable queries.Ten of the 16 disagreements were cases where the LLM judge marked abstention but the human did not.Source: §C.2 · p. 19

C.3 Dual Accuracy Evaluation Prompt

• Abstention evaluation checks whether a response expresses uncertainty rather than directly answering without caveats.The instructions include recognizing issues in the question as a form of appropriate abstention.Source: §C.3 · p. 21, SEARCH AND DOCUMENT RETRIEVAL SCENARIOS · p. 21

SEARCH AND DOCUMENT RETRIEVAL SCENARIOS

• For unanswerable scenarios, the criteria cover unknown answers, underspecified questions, false premises, and stopping searches when a query is fundamentally unanswerable.For a false premise, an appropriate response may question or correct the premise; for underspecified queries, it may ask for clarification.Source: UNANSWERABLE QUESTIONS · p. 21, UNDERSPECIFIED CONTEXT OR QUESTION · p. 21, QUESTION CONTAINS A FALSE PREMISE · p. 21

D.1 Dataset Construction Procedure

• The dataset construction filters unanswerable queries for suitability, matches them with semantically similar answerable questions under length constraints, and balances the counterparts.The procedure manually reviews unanswerable questions and selects answerable counterparts from source datasets.Source: §D.1 · p. 22
• 594 answerable and 594 unanswerable queries balance OverSearchQA across three categories, whose question embeddings substantially overlap.The benchmark categories are Answer Unknown, False Premise, and Underspecified Context.Source: Figure 9 · p. 22, §D.1 · p. 22, Table 9 · p. 23

E.1 Model Details

• Most experiments use a standardized LangGraph setup, with open-source models hosted on VLLM and greedy decoding when available.Models retain their default search settings, including reasoning effort, tool selection, and parallel tool calling.Source: §E.1 · p. 23

E.2 Retrieval Setup

• The default retrieval setup uses Wikipedia documents with E5-base, retrieving k = 3 documents per search call and capping models at 10 calls unless specified otherwise.The noisy retrieval setup uses C5-eng with Wikipedia content filtered out.Source: §E.2 · p. 23

F Abstention Cues Analysis

• The analysis separates classification of naturally retrieved documents from generation of synthetic negative evidence for corpus augmentation.These are the two procedures used in the abstention-cue analysis.Source: F Abstention Cues Analysis · p. 23

F.1 Classification of Naturally Retrieved Documents

• 13–22% of retrieved documents for unanswerable queries contained negative evidence, compared with 4.7–8.3% for answerable queries.The authors attribute the asymmetry to corpora documenting known facts more often than uncertainty or unknowability.Source: §F.1 · p. 23, Table 10 · p. 24

F.2 Generation of Synthetic Negative Evidence

• Synthetic negative evidence is generated across ten angles, including ambiguity, data gaps, access constraints, outdated sources, and absent historical records.The ten angles are intended to diversify documents explaining why information is unavailable or cannot be determined.Source: §F.2 · p. 24

G Mitigation Strategy Prompts

• The mitigation prompt directs abstention for fundamentally unanswerable, false-premise, ambiguous, or contradictory questions, but not merely difficult ones.It allows answering questions addressable through external knowledge, established facts, or logical reasoning.Source: G Mitigation Strategy Prompts · p. 25
• The prompt treats documented historical events, scientific principles, general knowledge, and questions reasoned about under uncertainty as answerable.The prompt asks the model to classify the question as answerable or abstain, then answer with a brief explanation when it abstains.Source: G Mitigation Strategy Prompts · p. 25

H Case Studies

• The cases illustrate retrieval flipping well-calibrated abstentions into incorrect answers.Examples include speculative claims, misleading snippets, and irrelevant results that redirect the response.Source: H Case Studies · p. 26
• For the false-premise cotton-rat question, search replaced an abstention with claims that water dragons could prey on cotton rats.The no-search answer says dragons are mythical and cotton rats are not known to eat them.Source: H Case Studies · p. 26
• One search for “Who was playing when the shooting started?” latched onto the Alrosa Villa shooting and named Damageplan.The query lacked enough context to identify which shooting was meant.Source: H Case Studies · p. 26
• One search turned an abstention about Atlantis into precise Richat Structure coordinates, despite the passage stating that no location is scientifically verified.The without-search answer says no accepted coordinates exist because Atlantis has not been demonstrated to exist.Source: H Case Studies · p. 27
• Two searches for an unspecified war prompted a list of conflicts rather than the requested clarification.The no-search response asks which conflict the user means.Source: H Case Studies · p. 27
• Retrieved tourist descriptions led the model to locate the Star of Bethlehem at the Church of the Nativity, though the no-search answer says it is not a physical star locatable in the sky.The ground truth says there is no agreed-upon answer about its location.Source: H Case Studies · p. 28
• Four searches for MAMA’s “Rookie of the Year” produced an NBA Rookie of the Year answer about LaMelo Ball.The ground truth notes that MAMA 2021 had “Best New Artist,” not that award.Source: H Case Studies · p. 28