Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs

1 Introduction

• Closed-source LLMs autonomously decide whether to retrieve and how to search, but no standardized evaluation measures the accuracy, selectivity, or cost-effectiveness of those choices.Unlike always-on retrieval, internal search makes retrieval a conditional decision that balances accuracy, latency, and resource use.Source: §1 · pp. 1–2
• Static and dynamic benchmark splits test whether models retrieve selectively for pre-cutoff facts and whether retrieval helps with time-sensitive questions.The static split compares search-enabled and disabled settings; the dynamic split requires up-to-date information.Source: §1 · p. 2
• The paper introduces a reproducible, versioned protocol for evaluating closed-source web-enabled LLMs without relying on internal configuration.Its stated contributions include examining retrieval benefits and cost, calibration in search-enabled settings, and practical selective-retrieval strategies.Source: §1 · p. 2

2.1 Information Retrieval and LLMs

• Existing IR and RAG benchmarks generally assume retrieval is fixed, leaving conditional, model-initiated retrieval and differences across model scales unevaluated.The paper shifts attention from retrieval quality in fixed pipelines to retrieval decision-making within autonomous search-enabled LLMs.Source: §2.1 · p. 2

2.2 Web-augmented LLMs

• Web-agent benchmarks typically assume retrieval or navigation is initiated, while tool-use benchmarks do not directly assess web-search selectivity, efficiency, or cost-aware invocation.This work treats internal web search as a conditional action in closed-source LLMs and evaluates its effect on answer quality and cost.Source: §2.2 · pp. 2–3

3 Methodology

• The evaluation measures end-to-end answer correctness, selective invocation, and observed failure modes; it omits ranking metrics because internal search does not expose ranked results or their combination.The protocol is reproducible, but live-web retrieval is not exactly temporally repeatable.Source: §3 · p. 3

3.1.1 Dataset.

• The 783-query static split uses pre-cutoff factual questions to test retrieval selectivity when external information is not strictly necessary.Queries are filtered from TempRAGEval and reviewed to remove duplicates, ambiguity, underspecification, and multi-step reasoning beyond factual lookup.Source: §3.1.1 · p. 3

3.1.2 Evaluation and Metrics.

• Static queries are tested with search disabled and enabled, with up to two calls and self-reported confidence, to measure accuracy, calibration, retrieval selectivity, and cost–utility.Models receive the query without an explicit retrieval cue, so invocation reflects their own assessment of information need.Source: §3.1.2 · p. 3

3.2.1 Dataset.

• The 288-query dynamic split rewrites suitable static questions into present-tense queries whose answers changed after the latest evaluated-model cutoff.Questions are manually verified against high-quality sources, and the split is a versioned snapshot with a refresh script to track evolving answers.Source: §3.2.1 · pp. 3–4

3.2.2 Evaluation and Metrics.

• Dynamic queries are evaluated with search enabled and up to two calls, measuring answer accuracy, retrieval invocation, and retrieval efficiency when external information is necessary.Outputs are exact-matched against acceptable semantic answer variants.Source: §3.2.2 · p. 4

3.3 Experimental Setup

• Four OpenAI and Anthropic models are evaluated through their provider APIs at temperature zero, with at most two autonomous search calls per query.Shared prompts are used, and the benchmark assumes correct answers are discoverable within the top one or two standard web-search results.Source: §3.3 · pp. 4–5

4.1 How effective is internal web-based IR in improving LLM accuracy?

• On the static split, web search improves accuracy across models, while dynamic accuracy remains lower, reaching 71.4% for GPT-5 and 66.2% for Claude Sonnet 4.6.The static-split relative gains are 62% for GPT-5-mini and approximately 80% for Claude Haiku 4.5; dynamic comparisons reflect both models and provider-specific search infrastructure.Source: §4.1 · p. 5
• On the dynamic split, the external retrieval pipeline outperforms internal web search across all models by a substantial margin.The comparison is end-to-end, not a controlled comparison of search quality; the pipeline retrieves two Google Custom Search results and supplies their summarized titles and snippets as context.Source: §4.1 · p. 5
• On the static split, accuracy is highest without retrieval and generally declines with additional calls; on the dynamic split, accuracy peaks with one call and declines with more.Repeated calls are associated with unresolved information need rather than additional gains, with steeper declines for Claude models, particularly Claude Haiku 4.5.Source: §4.1 · p. 5

4.2 Does web search effectively change factual outcomes when invoked?

• Across models, retrieval utility exceeds retrieval risk for queries where search was invoked.The analysis compares correctness with and without retrieval on static-split queries that triggered search; the supplied text does not specify the outcome proportions.Source: §4.2 · p. 5

4.3 How well-calibrated are LLMs’ verbalised confidence scores with and without web search?

• The calibration analysis uses standardized verbalized confidence as an observable proxy and tests its empirical alignment with correctness, not as a direct measure of internal uncertainty.The supplied passages identify a calibration-metrics table but do not report its results.Source: §4.3 · p. 6, Table 9 · p. 7

4.4 Do models correctly invoke retrieval when information need is unavoidable?

• On the dynamic split, models invoke search for 87.5%–93.9% of queries, but some queries requiring current information still receive no retrieval.No model achieves perfect retrieval recall.Source: §4.4 · p. 7, Table 10 · p. 7

4.5 Where and why does retrieval fail?

• Query formulation is the largest failure source on the static split, while source-quality failures rise on the dynamic split and evidence-integration failures are least frequent.Query formulation accounts for roughly half of static-split failures; dynamic source-quality failures become comparable to or exceed query formulation, particularly for frontier models.Source: §4.5 · pp. 7–8

4.6 How cost-effective is web search in LLMs?

• Average retrieval cost is around one cent per query, while cost per corrected answer varies by model and is lower for Claude than GPT.On the dynamic split, strict improvement costs decrease across all models, but Claude retains an efficiency advantage; cost-effectiveness depends on retrieval selectivity and efficiency, not model scale alone.Source: §4.6 · p. 8

5.1 Cost–accuracy trade-off

• For stable information needs, a single high-precision retrieval step offers an accuracy–cost trade-off, while repeated calls on rapidly evolving queries show diminishing returns.The authors recommend a one-call upper bound as a verification layer for stable queries and external tools or scaffolding for live web data streams.Source: §5.1 · p. 8

5.2 Inbuilt web search vs external retrieval

• Internal search suits low-latency, responsive use, while external pipelines provide greater control over query formulation, ranking, and source selection.The authors identify external retrieval as better suited to auditable, domain-restricted, or authoritative-corpus settings; even a simple external pipeline outperformed internal search across evaluated models.Source: §5.2 · p. 8

5.3 Selective invocation strategies

• Confidence-gated retrieval achieves favorable accuracy–cost trade-offs, but retrieval degrades confidence calibration and makes verbalized confidence unreliable as a stand-alone signal.The authors recommend combining confidence with signals such as retrieval redundancy, source agreement, or citation quality.Source: §5.3 · p. 8, Table 12 · p. 9

5.4 Provider-specific behaviours

• Claude configurations issue fewer retrieval calls and have lower cost per corrected answer, while GPT configurations search more often and make more iterative calls.These differences may reflect both model-level policies and provider infrastructure; the authors describe the comparison as specific to evaluated deployments.Source: §5.4 · p. 9

6 Conclusion

• The framework combines static and dynamic factual splits, outcome transitions, and confidence-based routing to assess retrieval decisions and factual reliability without access to model internals or provider configuration.It is designed to apply across smaller and frontier systems.Source: §6 · p. 9
• Internal web search yields substantial accuracy gains across models but introduces systematic confidence-calibration failures.The conclusion describes confidence as inflated even when retrieval does not resolve uncertainty.Source: §6 · p. 9
• Retrieval effectiveness is constrained primarily by query formulation and source selection rather than model scale alone.The paper also provides a mechanism to refresh the dynamic split and releases the dataset and evaluation code.Source: §6 · p. 9

• The evaluation covers native, tool-integrated web search; it can be extended to other commercial systems and open-source models with access to internal tool parameters.Source: §7 · p. 9
• Live-web retrieval trajectories may change over time, although the static split is designed to remain stable and the dynamic split is versioned by snapshot date and refreshable.Source: §7 · p. 9
• Retrieval depth is capped at two calls, leaving multi-hop, compositional, and adversarial queries requiring iterative reformulation or multi-source reasoning unevaluated.Source: §7 · p. 9
• Confidence analysis uses only verbalized confidence signals, rather than logits or other possible signals.Source: §7 · p. 9
• The benchmark uses English queries, sources, and responses, so observed behaviors may not generalize to multilingual or crosslingual settings.Source: §7 · p. 9

A.1 Static Split

• The static split aims to retain temporally anchored, fully specified, unambiguous factual queries answerable through a single lookup, excluding cases requiring multi-step reasoning or extensive synthesis.Source: §A.1 · p. 11

A.1.1 Temporal and Factual Filtering.

• Filtering retained explicitly time-grounded facts dated before 1 February 2025, reducing the initial 1,248 samples to 952 candidates.Source: §A.1.1 · p. 11

A.1.2 Manual Validation and De-duplication.

• Manual review removed duplicates and queries lacking specificity, answer uniqueness, completeness, or single-query answerability, leaving 783 samples for final inclusion.Source: §A.1.2 · p. 11

A.1.3 Independent Annotation and Dataset Validation.

• Two independent annotators reviewed 200 static-split candidates, reaching Cohen’s κ=0.86; this quality-control review preceded inclusion of the 783-query static split.Source: §A.1.3 · pp. 11–12

A.2 Dynamic Split

• The dynamic split was designed for current information needs whose answers changed after the models’ knowledge cutoff while remaining direct factual lookups.Source: §A.2 · p. 12

A.2.1 Candidate Selection.

• Annotators selected static queries about potentially changeable roles or attributes that could be rewritten in present tense without changing their meaning; 357 candidates remained after review.Source: §A.2.1 · p. 12

A.2.2 Temporally Neutral Rewriting.

• The 357 candidates were rewritten to express current information needs while preserving the queried entity, requested information, and original intent.Source: §A.2.2 · p. 12

A.2.3 Post-cutoff Validation and Answer Verification.

• The final dynamic split contains 288 queries, each with a temporally verified current answer, supporting sources, and curated acceptable answer variants.Source: §A.2.3 · p. 13
• Rewritten queries were checked against trusted sources for post-cutoff answer changes, then independently validated for query quality and answer completeness, with Cohen’s κ=0.82 agreement.Source: §A.2.3 · pp. 12–13

B Evaluation Protocol

• Models received the same task-level instructions; static queries additionally required a self-assessed confidence score from 0 to 1.Source: B Evaluation Protocol · p. 13
• Confidence was used for calibration and retrieval-selectivity analyses, not correctness; answers were normalized and matched against canonical or validated semantic variants without manual correction.Source: B Evaluation Protocol · p. 13

C Failure Taxonomy Annotation

• A retrieval failure is a searched query with an incorrect final answer, assigned to query formulation, source quality, or evidence integration.Source: C Failure Taxonomy Annotation · p. 13

C.0.1 Annotation Procedure.

• Annotators reviewed the query, issued search, results, answer, and references, assigning the earliest sufficient failure stage and allowing an NA label when categories did not fit.Source: §C.0.1 · p. 13

C.0.2 Inter-Annotator Agreement.

• Cohen’s κ=0.79 indicated high agreement; annotators independently labeled sampled errors, then resolved disagreements jointly by consensus.The samples comprised 150 random incorrect static-split cases per model and 75 cases per model from the dynamic split.Source: §C.0.2 · p. 13

D.1 Prompt sensitivity for verbalised confidence elicitation

• The analysis compared the original confidence instruction with a semantically equivalent request to accompany the answer with a confidence score.It used 100 randomly selected static-split queries and the frontier models GPT-5 and Claude Sonnet.Source: §D.1 · p. 13
• Pearson correlations between confidence scores were assessed separately with retrieval disabled and enabled.Source: §D.1 · p. 14
• r=0.84–0.88 across both models and conditions, indicating highly correlated confidence scores under the two prompt formulations.The correlations remained similarly high with web search enabled.Source: §D.1 · p. 14