PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?

HuiMing Fan, Xiao Wang, Zheng Chu, Qianyu Wang, Zhuoyao Wang, Ming Liu, Bing Qin, XingYu

arXiv:2605.28721v1cs.AI

TL;DR

The paper asks whether search agents genuinely discover answers or mainly verify hypotheses from intrinsic knowledge, and introduces diagnostics plus LiveBrowseComp to test this. LiveBrowseComp largely removes the memory advantage: closed-book accuracy falls below 2%, search-augmented performance drops sharply, and model rankings change.

  • Problem

    The paper examines whether strong search-agent benchmark scores reflect evidence-driven discovery or hypotheses generated from intrinsic knowledge and confirmed through search.

  • Method

    The paper combines evidence-removal diagnostics with LiveBrowseComp, a benchmark of recent, obscure, human-validated questions beyond models’ prior knowledge coverage.

  • Results

    On LiveBrowseComp, every evaluated model achieves below 2% closed-book accuracy, search-augmented scores drop sharply, and static-benchmark rankings no longer reliably predict performance.

  • Takeaways & Limitations

    Intrinsic Knowledge Dependence is a central confound in search-agent evaluation, motivating dynamic, time-sensitive benchmarks that reward evidence-led discovery.

  • Takeaways & Limitations

    Evaluation uses a single search backend, so measured capability may partly reflect index coverage rather than agents’ search strategy.

Abstract

from arXiv · show

Are LLM-based search agents genuinely searching, or using the web to verify what they already know? We study this question on BrowseComp with three diagnostics. Our analysis reveals Intrinsic Knowledge Dependence (IKD): even with tool access, agents often rely on intrinsic knowledge -- information encoded in the model before retrieval -- rather than on external evidence. Agents answer up to 44.5% of BrowseComp questions without tools, generate more than half of their search queries from internally produced hypotheses rather than retrieved leads, and perform worse than closed-book baselines when answer-supporting evidence is removed. These results suggest that static search benchmarks can reward memory-backed verification rather than evidence-driven discovery, conflating what agents already know with what they can find. We then introduce LiveBrowseComp, a deep-search benchmark designed to evaluate agents beyond intrinsic coverage. It contains 335 human-authored questions whose answers depend on facts published within the 90 days preceding benchmark construction, drawn from six updated sources and filtered to exclude globally salient events. On LiveBrowseComp, all evaluated agents fall below 2% closed-book accuracy, search-augmented scores drop by 25-40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance. LiveBrowseComp is available at https://huggingface.co/datasets/Forival/LiveBrowseComp.

1 Introduction

The paper argues that static search benchmarks can reward intrinsic-knowledge-backed verification rather than genuine evidence-driven discovery. It introduces LiveBrowseComp, a temporally grounded benchmark designed to evaluate search beyond models’ current knowledge boundaries.

  • Diagnostic Experiments: The diagnostics progressively remove or perturb retrieved evidence to test whether agents truly need search, whether they can discover without answer-supporting evidence, and how hypotheses arise during browsing.The experiments are designed around three questions concerning tool-free answering, evidence blocking, and multi-step browsing.
  • Intrinsic Knowledge Dependence: 44.5%: closed-book pass@4 reaches up to 44.5% with search tools removed, and every evaluated model scores non-trivially on existing benchmarks.These results indicate that many benchmark questions are already covered by agents’ parametric knowledge.
  • Intrinsic Knowledge Dependence: Intrinsic Knowledge Dependence (IKD) describes agents guessing from memory and using search for confirmation, with search collapsing when facts lie outside their knowledge boundary.The paper distinguishes this failure mode from data contamination, noting that uncontaminated questions can still be solved through broad parametric world knowledge.
  • LiveBrowseComp: 335: LiveBrowseComp contains 335 human-authored questions requiring facts published within the 90 days before benchmark construction and unavailable from earlier information alone.Questions are seeded from six continuously updated sources and filtered to exclude globally salient events.

2 Pilot Study

The pilot study tests whether browsing agents discover answers from retrieved evidence or mainly use search to verify hypotheses generated from intrinsic knowledge. Across multiple diagnostics, agents show substantial closed-book capability, deteriorate when supporting evidence is blocked, and generate searches primarily from their own hypotheses.

  • Pilot design: The study evaluates frontier agents on four challenging benchmarks using closed-book, evidence-blocked, and trajectory-grounding diagnostics.The benchmarks span long-horizon web browsing, multilingual browsing, expert-level knowledge reasoning, and general tool-augmented problem solving.
  • Closed-book coverage: 38.9 average pass@4 across 24 model–benchmark pairs is achieved without retrieval, with scores ranging from 20.4 to 62.0.Notable closed-book scores include 44.5 for MiniMax M2.5 on BrowseComp and 50.2 for Seed 2.0 on HLE.
  • Evidence-blocked search: 6.2 average pass@4 under evidence-blocked search falls below the 26.1 closed-book average for every evaluated model.Answer-supporting and gold documents were removed, leaving only irrelevant and hard-negative documents; all blocked scores remained below 10.
  • Trajectory grounding: More than half of queries are model-originated for every model, rising above 60% in later browsing rounds.Agents generally generate new search directions from internally produced hypotheses rather than extending searches from retrieved leads.
  • Trajectory grounding: 32.2% or less is the evidence-use rate across evaluated models even when answer-supporting evidence is retrieved.Rates were 32.2% for DeepSeek v3.2, 24.7% for GLM-5.1, 30.8% for MiniMax M2.5, and 31.5% for Kimi-K2.5.
  • Interpretation: Intrinsic Knowledge Dependence (IKD) describes agents using parametric knowledge to generate hypotheses and retrieval mainly to confirm them.The diagnostics identify a model-led rather than evidence-led search loop, in which non-supporting retrieval can pull agents away from correct intrinsic answers.

3 LiveBrowseComp: A Deep Search Benchmark Designed to Suppress IKD

LiveBrowseComp is a deep-search benchmark designed to evaluate evidence-driven discovery beyond models’ intrinsic knowledge by combining recent, obscure facts with multi-step search and synthesis. Its construction uses source diversity, stability filtering, human annotation, and independent verification to produce 335 questions with temporally anchored solution paths.

  • Benchmark design: LiveBrowseComp targets information outside intrinsic knowledge by requiring facts from the most recent 90 days while excluding globally salient events.Recency pushes facts beyond likely training-data coverage, while obscurity reduces exposure through widespread reporting.
  • Benchmark design: Six continuously updated sources span global news, entertainment, games, cybersecurity, sports, and earthquakes, reducing dependence on any single domain.The sources are GDELT, TMDB, RAWG, CVE/NVD, SportsDB, and USGS, with timestamped public APIs enabling temporal control.
  • Filtering pipeline: Candidate events pass temporal, long-tail, and answer-stability filters before question construction, excluding facts that are old, widely exposed, or likely to change.Stable candidates must have uniquely determined answers throughout the benchmark’s lifespan.
  • Quality control: Independent human construction and peer review verify question correctness, uniqueness, difficulty, and genuine dependence on recent information.Review includes evidence-chain checking, search-only difficulty calibration, temporality verification, and cross-checking by independent verifiers.
  • Question design: Temporally anchored clues force retrieval of recent evidence while preserving BrowseComp-like multi-hop reasoning depth.Unlike static historical clues, these anchors cannot be resolved from pre-existing knowledge.
  • Benchmark composition: 335 questions span eight topical categories, with category sizes determined by the distribution of events surviving the filtering pipeline.The benchmark aims to cover a broad range of topics while reflecting the available filtered events.

4 Experimental Evaluation on LiveBrowseComp

LiveBrowseComp suppresses intrinsic-knowledge shortcuts: models score substantially lower, rankings transfer less reliably, and search trajectories become longer and more exploratory than on static benchmarks. Closed-book accuracy falls below 2%, indicating that agents must rely on search rather than memory.

  • Evaluation setup: 11 models spanning open- and closed-source families and 230B–1.6T parameters are evaluated with a shared search-agent scaffold.The scaffold standardizes interaction protocol, sampling budget, context limit, answer format, and search, page-retrieval, and code tools.
  • Performance: 28.0–43.2 avg@4 on LiveBrowseComp contrasts with 51–77 points on BrowseComp, demonstrating a sharp performance drop.MiniMax M2.5 scores 28.0, while GPT 5.4 scores 43.2 on LiveBrowseComp.
  • Performance: GLM 5.1 drops from 68.0 on BrowseComp to 33.9 on LiveBrowseComp, while DeepSeek v3.2 rises from 51.4 to 37.6, changing model rankings.The divergence is consistent with unequal intrinsic knowledge dependence across models.
  • Performance: 16.6 points versus 10.3 points: the open-source top-to-bottom spread contracts from BrowseComp to LiveBrowseComp.The static spread is 68.0 versus 51.4; the LiveBrowseComp spread is 38.3 versus 28.0.
  • Intrinsic knowledge suppression: Below 2% closed-book accuracy on LiveBrowseComp, compared with 20–44% on BrowseComp-Plus, shows that memory shortcuts are nearly eliminated.Under the same closed-book configuration as the pilot study, all models require search to score on LiveBrowseComp.
  • Behavioral and ranking transfer: ρ = 0.87 to ρ = 0.74 and r = 0.79 to r = 0.53: correlations between static and live-benchmark performance weaken substantially.On LiveBrowseComp, short-turn verification clusters largely disappear, trajectories shift toward higher turn counts, and queries become more exploratory.

5 Related Work

Related work progresses from static retrieval and web-interaction benchmarks toward deep-search evaluation, while also addressing parametric knowledge contamination and increasingly dynamic benchmark construction.

  • Search-agent benchmarks: Search-agent benchmarks evolved from static-corpus fact extraction to action-level web manipulation and, more recently, deep-search evaluation.Examples include NaturalQuestions, TriviaQA, HotpotQA, WebArena, Mind2Web, WebVoyager, and BrowseComp.
  • Data contamination: Contamination research has expanded beyond literal string overlap to examine models’ reliance on related parametric knowledge.Traditional methods include n-gram overlap, membership inference, temporal cutoff analysis, and behavioral manipulation.
  • Dynamic evaluation: Recent benchmarks increasingly use dynamic data, including monthly refreshes, continuous scraping, temporally aware updates, and periodic difficulty splits.Examples include LiveBench, LiveCodeBench, FreshQA, and LiveMathBench.

6 Discussion and Conclusion

The paper identifies Intrinsic Knowledge Dependence as a central confound in search-agent evaluation and introduces LiveBrowseComp to test agents with recent, long-tail questions beyond their current knowledge coverage. On LiveBrowseComp, closed-book accuracy falls below 2%, search-augmented scores drop sharply, and model rankings change.

  • Discussion and Conclusion: Intrinsic Knowledge Dependence confounds search-agent evaluation when agents generate hypotheses from parametric knowledge and mainly use search for confirmation.This allows agents to score well without relying primarily on externally discovered evidence.
  • Discussion and Conclusion: LiveBrowseComp uses recent, long-tail questions designed to place agents beyond their current knowledge coverage.The benchmark addresses the evaluation confound by testing questions less likely to be answerable from existing model knowledge.
  • Discussion and Conclusion: Below 2% closed-book accuracy is observed on LiveBrowseComp, alongside sharply reduced search-augmented scores and changed model rankings.These outcomes indicate that performance on the new benchmark differs substantially from conventional search-agent evaluation.

A Data Source and Filtering Criteria · B Scoring Prompt

The benchmark constructs long-tail questions from six filtered data sources using source-specific heuristics, thresholds, and quality controls. Model responses are scored by an LLM judge that extracts exact answers, permits reasonable variants, and applies retry-based parsing rules.

  • A Data Source and Filtering Criteria: Six data sources are filtered using API specifications, source-specific thresholds, and long-tail selection criteria.The sources are GDELT, TMDB, RAWG, CVE, Sports, and USGS.
  • A Data Source and Filtering Criteria: 459k GDELT articles are filtered down to 1k using LLM ratings of geographic audience, entity fame, and societal impact.Articles with average scores of 2.0–4.0 are retained, while spam, junk pages, and easily guessable global headlines are excluded; additional filters remove paywalls, bot-detection pages, 404 errors, and other unusable articles.
  • A Data Source and Filtering Criteria: TMDB retains films and television items with long-tail scores ≥2.5, rewarding low popularity, low vote counts, zero box-office revenue, and non-English language.Entries must include an overview, at least three cast members, and an IMDB or Wikidata identifier.
  • A Data Source and Filtering Criteria: RAWG retains games with total long-tail score ≥2.5, emphasizing low ratings_count, low “added” count, absent Metacritic scores, and non-ASCII names.Each entry must have at least one developer or publisher and a non-empty description.
  • A Data Source and Filtering Criteria: CVE entries require English descriptions, non-REJECTED status, valid CVSS scores, and total long-tail score ≥2.0.Scoring rewards CVSS ≥9.0, single-product specificity, exploit availability, recency within 30 days, and reference richness.
  • A Data Source and Filtering Criteria: Sports and USGS filters require long-tail scores ≥1.5, targeting complete and significant athletic events and earthquakes that are felt but not catastrophic.Sports adds +1.5 for non-football events, while USGS considers magnitude, significance, depth, impact indicators, and location annotations.
  • B Scoring Prompt: An LLM-as-judge evaluator extracts a final answer and compares it with the ground truth, accepting reasonable abbreviations or aliases but rejecting imprecise or partial answers.The evaluator is GPT-OSS and uses a structured prompt containing the question, response, correct answer, extracted final answer, and reasoning.
  • B Scoring Prompt: Outputs beginning with “A” are marked correct and those beginning with “B” incorrect; failed parses are retried up to five times before being marked incorrect.The scoring procedure therefore combines answer extraction, categorical judgment, and a fixed failure policy.

C Search Agent Experimental Configuration · C.1 General Configuration

LiveBrowseComp uses a deep-search configuration centered on multi-step, multi-source investigation, with model-specific prompt variants, designated retrieval tools, and bounded context and iteration budgets. Exceeding either budget triggers a forced final-answer response.

  • C.1 General Configuration: The default system prompt applies to DeepSeek, GLM, MiniMax, and Seed, while Kimi uses a cookbook-aligned prompt and Seed follows the Seed1.8 Cookbook format.These prompt variants define the agents’ general operating instructions.
  • C.1 General Configuration: The deep-search assistant is instructed to conduct rigorous, multi-step, multi-source investigations across broad open-domain and specialized academic questions.It must seek, cross-check, and integrate information from credible and diverse sources into comprehensive, accurate, structured, objective responses.
  • C.1 General Configuration: LiveBrowseComp agents can use search(query), which returns up to 10 web-search results with URLs and text snippets via serper.dev.The search tool is one of the core retrieval interfaces provided in the experiments.
  • C.1 General Configuration: Agents can use visit(url, goal) to retrieve full page content summarized toward an information goal.Depending on the model, experiments also provide Python, Google Scholar, and Google Maps.
  • C.1 General Configuration: The maximum context length per sample is 256k tokens, and the maximum iteration budget is 250 steps.Exceeding either limit causes the system to inject a forced-answer prompt.
  • C.1 General Configuration: When a research limit is reached, the forced-answer prompt asks the model to answer using gathered information and place the final response between <answer> and </answer>.This prompt applies after exceeding the maximum research steps or context limit.

C.2 BrowseComp-Plus and Dense Retrieval Experiments

BrowseComp-Plus augments BrowseComp with a categorized document library that supports dense-retrieval evaluation and controlled evidence blocking. Experiments index this library with Qwen3-8B-Embedding, and blocked-evidence settings retain only distractor documents while restricting other search routes.

  • BrowseComp-Plus: BrowseComp-Plus expands BrowseComp with evidence, gold, irrelevant, and hard-negative documents for each question.Evidence documents contain direct answer evidence; gold documents provide high-quality support, while irrelevant and hard-negative documents serve as distractors.
  • Dense Retrieval Experiments: The pilot study constructs a dense retrieval index over the document library using Qwen3-8B-Embedding as the model’s search environment.This follows the official BrowseComp-Plus recommendations.
  • Dense Retrieval Experiments: Evidence-blocking experiments remove evidence and gold documents, retaining only irrelevant and hard-negative documents in the index.Under this condition, models can still issue search queries, but the supplied passage truncates the remaining experimental details.
  • Dense Retrieval Experiments: Google Scholar and Google Maps are disabled, and internet access through the Python interpreter is prohibited in all dense-retrieval experiments.These restrictions apply across the dense-retrieval experiments.

D Limitations

LiveBrowseComp’s 90-day temporal boundary is approximate because facts may have been announced or leaked earlier and models have different training cutoffs. Evaluations also rely on a single search backend, so results may vary with search-index coverage.

  • Temporal boundary: The 90-day temporal window is only heuristic: facts may have been announced or leaked earlier, and models may have different training cutoffs.Therefore, the boundary between intrinsic and newly published information is not perfectly crisp.
  • Search backend: All evaluations use serper.dev, so different search indices may produce different results.Apparent search capability may partly reflect the coverage of the underlying index.

E Closed-Book Answering Configuration … F.2 Annotator Compensation

The paper specifies a tool-free closed-book protocol with standardized prompting and repeated-sample evaluation, then details LiveBrowseComp’s multi-stage annotation, verification, screening, and compensation procedures. Questions are constructed from recent seed events, independently checked for correctness, uniqueness, difficulty, and temporality, with annotators and verifiers paid approximately $9.60 per hour.

  • E Closed-Book Answering Configuration: Closed-book experiments use only parametric knowledge without search or browsing tools, under a unified system prompt.The prompt instructs models to decompose clues, cross-check their knowledge, commit to a candidate, guess when uncertain, and wrap answers in <answer></answer> tags.
  • E Closed-Book Answering Configuration: Architectural differences yield three closed-book implementation categories, including single-round completion and two-phase prefill configurations.DeepSeek V4 and MiniMax M2.5 use single-round calls with max_tokens of 16384 and 8192, respectively; GLM-5, Kimi K2.5/K2.6, and MiroThinker use a thinking-enabled first phase.
  • E Closed-Book Answering Configuration: 4 independent samples are used for all closed-book experiments, with both pass@4 and avg@4 reported.This matches the sampling setup used in tool-use experiments.
  • F Human Annotation Details: Annotators draft questions and reference answers from filtered seed events, while verifiers independently validate each question.All annotators passed the screening and training process described in the main text.
  • F.1 Annotation Workflow and Quality Control: Questions must be multi-step and multi-source, verifiable from definitive sources with a single short-string answer, and anchored by at least one clue from the past 90 days.They should also resist solving from the first three pages of search results for the question text.
  • F.1 Annotation Workflow and Quality Control: Annotators research from seed events, record visited pages, and document the evidence chain linking each question to its answer.This documentation becomes the primary input for subsequent verification stages.
  • F.1 Annotation Workflow and Quality Control: Correctness and uniqueness verification uses full evidence chains and 8 search-enabled rollouts from each of four models to generate candidate answers.Verifiers manually check every candidate against all question constraints and remove questions admitting an alternative answer.
  • F.1 Annotation Workflow and Quality Control: Three independent annotators screen difficulty using web search, while verifiers check temporality and a fourth verifier cross-checks all three verification stages.Difficulty screening excludes questions solved within 30 min; quality control applies to correctness, difficulty, and temporality checks for every question.

G Per-Domain Analysis

LiveBrowseComp accuracy varies substantially across both topical domains and models. Domain-specific rankings can diverge from aggregate rankings, indicating that aggregate scores may conceal meaningful differences in model capability and knowledge coverage.

  • G Per-Domain Analysis: Performance varies considerably across the five LiveBrowseComp topical categories and across evaluated models.Table 5 reports per-domain accuracy using avg@4.
  • G Per-Domain Analysis: Sports and Society & Culture tend to produce higher scores for several models, whereas Sci. & Tech. is more challenging for most.
  • G Per-Domain Analysis: Within-domain rankings can diverge from aggregate rankings, as GLM 5.0 leads Entertainment despite a below-average overall score.GLM 5.0 scores 52.1% in Entertainment and 28.5% overall.
  • G Per-Domain Analysis: Open-source models are competitive in Entertainment and Sports, while domain-specific knowledge coverage varies across model families.These patterns show why aggregate scores alone can mask meaningful capability differences.
Loading 2605.28721v1…