PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge

Yunhan Wang, Jiaan Wang, Lianzhe Huang, Xianfeng Zeng, Fandong Meng

arXiv:2606.13120v1cs.CL

TL;DR

Static search-agent benchmarks can become vulnerable to contamination and parametric memorization, limiting evaluation of genuine browsing. EvoBrowseComp automatically synthesizes fresh, complex questions from live-web traversal, and experiments show frontier models achieve only modest tool-assisted accuracy that collapses without tools.

  • Problem

    Existing search-agent benchmarks rely on static knowledge, making their fixed question–answer content vulnerable to test-set contamination and parametric memorization.

  • Method

    EvoBrowseComp automatically synthesizes 400 English and 400 Chinese complex questions from live-web traversal through a collaborative three-agent framework.

  • Results

    44.8% accuracy with tools fell to 6.0% without tools for Claude-Opus-4.6, indicating the benchmark challenges retrieval and multi-hop reasoning over fresh knowledge.

  • Takeaways & Limitations

    EvoBrowseComp provides a fully automated, regularly updateable paradigm for contamination-resistant, future-proof evaluation of search agents.

  • Takeaways & Limitations

    The synthesis agents use DeepSeek-V3.2, potentially inheriting its biases and toxic behaviors, while judging only final answers cannot distinguish correct reasoning from accidental success.

Abstract

from arXiv · show

Search Agents -- large language models augmented with search tools -- have intensified the need for future-proof evaluation benchmarks. Existing benchmarks such as BrowseComp rely on static knowledge, making them vulnerable to test-set contamination and parametric memorization. Consequently, models can achieve high scores through fact recall rather than genuine retrieval, obscuring true browsing competence via reasoning shortcuts. In this paper, we introduce EvoBrowseComp, an evolving benchmark of 400 English and 400 Chinese contamination-free complex questions synthesized via live-web traversal. To collect these questions, we design a three-agent collaborative framework: (1) a QA synthesis agent that retrieves fresh knowledge from the live web to synthesize QA pairs; (2) an information filtering agent that filters retrieved knowledge in terms of credibility and popularity to block parametric shortcuts; and (3) a high-level guidance agent that formalizes questions into reasoning graphs to reduce logical redundancy and shortcuts in synthesized QA pairs. Because the framework supports fully automated synthesis, EvoBrowseComp can be regularly updated to prevent data contamination and maintain temporal freshness. Extensive experiments confirm its great difficulty, requiring broad horizontal search. It establishes a scalable paradigm for auto-updatable, high-difficulty benchmarking that keeps pace with both evolving world knowledge and advancing agent capabilities.

1 Introduction

EvoBrowseComp addresses the limitations of static web-browsing benchmarks with an automatically synthesized, continuously refreshable benchmark built from fresh live-web knowledge. Its questions remain difficult for advanced models, especially without search tools, highlighting the need for genuine retrieval.

  • Motivation: Existing benchmarks typically anchor questions, documents, webpages, or files to static or time-fixed knowledge, enabling potential contamination and memorization shortcuts.BrowseComp and BrowseComp-ZH use manually curated pairs, BrowseComp-Plus freezes document snapshots, and GAIA uses immutable web pages or files.
  • Benchmark: EvoBrowseComp contains 400 English and 400 Chinese complex questions automatically synthesized through live-web traversal and fresh-knowledge validation.Its questions are designed to prevent exploitation of parametric memorization.
  • Construction: A three-agent collaborative framework synthesizes QA pairs by retrieving fresh knowledge, filtering information, and formalizing questions into reasoning guidance.The framework combines QA synthesis, information filtering, and high-level guidance to construct questions automatically.
  • Construction: Full automation removes costly manual annotation and enables continuous, low-cost regeneration that retires over-exposed questions and incorporates newly emerged facts.The pipeline supports regular refreshing as knowledge evolves.
  • Evaluation: 44.8% accuracy was achieved by Claude-Opus-4.6 with tools, versus 6.0% without tools, showing that EvoBrowseComp requires difficult retrieval of temporally fresh, structurally complex information.The results were obtained under tool-based and tool-free settings.

2 EvoBrowseComp

EvoBrowseComp is designed around temporally fresh knowledge and a fully automated, continuously evolvable construction pipeline. Its three-agent framework synthesizes, filters, and structurally guides complex QA pairs, producing a contamination-resistant benchmark that can be periodically regenerated.

  • Foundational principles: EvoBrowseComp prioritizes knowledge emerging after model training cutoffs to reduce parametric memorization and supports periodic replacement of over-exposed questions.This design aims to preserve long-term benchmark validity without expensive human curation.
  • Three-agent framework: The pipeline uses three collaborative agents: live-web QA synthesis, credibility and popularity filtering, and reasoning-graph guidance to detect redundancy and shortcuts.The agents operate as an iterative feedback loop beginning from seed entities and candidate QA pairs.
  • Seed entity collection: About 50K seed entities are collected through live-web retrieval of recently surfaced entities mentioned on high-coverage news or official websites.The seed entities are denoted as E and are intended to support temporally fresh and logically complex QA synthesis.
  • QA synthesis: For each seed entity, the QA synthesis agent iteratively gathers evidence through Google search and page visits before constructing a complex QA pair.The process encourages gathering fresh knowledge, while the benchmark’s timestamp can be adjusted to other timestamps.
  • Information filtering: The information filtering agent cross-validates fresh evidence for credibility and judges non-fresh evidence for excessive popularity or coverage, addressing reliability and predictability risks.These risks include rumor propagation and over-covered knowledge that makes reasoning paths too predictable.
  • Benchmark composition: 400 English and 400 Chinese QA pairs constitute EvoBrowseComp after quality filtering and domain balancing.The resulting benchmark is intended to provide contamination-free, complex questions generated through the evolving pipeline.

3 Experiments

Experiments evaluate diverse LLMs on EvoBrowseComp under controlled tool-use and judging settings. Results show that the benchmark limits tool-free performance, is harder than BrowseComp, and exposes failures caused by excessive tool use.

  • Experimental setup: Evaluations cover multiple recent LLMs, using maximum thinking mode, up to 40 tool calls, GLM-5-Chat as judge, and three independent runs averaged.All three data-collection agents use DeepSeek-V3.2, and the information-filtering threshold k is 5.
  • Tool-Free Setting: Without tools, most LLMs score below 5%; Kimi-K2.6 reaches 0% English and 1.6% Chinese, while DeepSeek-V3.2 reaches 6.3% and 10.3%.These limited tool-free results suggest EvoBrowseComp prevents reliance on parameterized memory by introducing fresh knowledge.
  • Difficulty comparison: Model performance on EvoBrowseComp is significantly lower than on BrowseComp and BrowseComp-ZH, indicating that EvoBrowseComp questions are difficult.Table 3 compares three example LLMs across BrowseComp, BrowseComp-ZH, and EvoBrowseComp-EN/ZH.
  • The Effect of Reasoning Effort: DeepSeek-V4-Pro and DeepSeek-V4-Flash underperform DeepSeek-V3.2 because many samples exceed the maximum 40 tool calls.The authors inspect DeepSeek-V4 predictions and evaluate DeepSeek-V4-Flash under three reasoning-effort configurations to study this effect.

4 Related Work

Prior benchmarks evaluate LLM browsing, reasoning, and retrieval through single-hop or multi-hop fact retrieval, but many are now manageable for cutting-edge LLMs. BrowseComp was introduced to raise the difficulty ceiling with reverse-engineered questions.

  • Early Benchmarking: Benchmarks such as NaturalQuestions, TriviaQA, and HotpotQA evaluate LLM capabilities through single-hop or multi-hop fact retrieval.These datasets are cited as early examples in browsing, reasoning, and retrieval evaluation.
  • Early Benchmarking: Many early fact-retrieval datasets are effectively handled by cutting-edge LLMs.The passage motivates developing more challenging benchmarks because existing datasets no longer provide a sufficiently high difficulty ceiling.
  • BrowseComp: BrowseComp introduces reverse-engineered questions to raise the difficulty ceiling for browsing evaluation.It is presented as a later benchmark designed to increase challenge beyond earlier retrieval-focused datasets.

5 Conclusion

EvoBrowseComp introduces an automatically synthesized, evolving benchmark with 400 English and 400 Chinese contamination-free complex QA pairs. Its live-web framework supports quality-controlled, regularly updateable data that remains temporally fresh and challenging for cutting-edge LLMs.

  • Benchmark and framework: EvoBrowseComp contains 400 English and 400 Chinese contamination-free complex QA pairs synthesized through a three-agent framework that discovers fresh knowledge from the live web.The framework is designed for search-agent benchmarking.
  • Data quality: Quality controls target reasoning redundancy, shortcuts, textual quality, answer uniqueness, and question difficulty.Human analyses indicate that the synthesized data achieves a high level of quality.
  • Sustainable updating: The fully automatic framework requires no costly manual annotation, enabling regular updates that prevent data contamination and ensure temporal freshness.Its updateability is presented as a way to keep the benchmark current.
  • Benchmark significance: Experiments on cutting-edge LLMs underscore the challenges posed by EvoBrowseComp, establishing a sustainable paradigm for future benchmarking.The conclusion links the benchmark’s difficulty and automated maintenance to long-term use.

Limitations

The framework has limitations stemming from its DeepSeek-V3.2 backbone and from evaluating only agents’ final answers rather than their complete reasoning trajectories.

  • Model and evaluation limitations: The synthesized data may inherit biases and toxic behaviors from the DeepSeek-V3.2 backbone used by all three agents.The backbone is DeepSeek-V3.2 (DeepSeek-AI, 2025).
  • Model and evaluation limitations: Evaluation judges only final answers, making it difficult to distinguish correct reasoning from an agent that obtained the correct answer through another process.

Ethical Considerations

EvoBrowseComp addresses licensing and privacy by releasing synthesized data under a noncommercial share-alike license and filtering potential private information from publicly available web sources.

  • Licenses: Synthesized data will be released under the CC-BY-NC-SA 4.0 license.This license governs the release of the benchmark’s synthesized data.
  • Privacy Information: The benchmark extracts knowledge from publicly available web pages.Its source material is drawn from web pages that are publicly accessible.
  • Privacy Information: LLMs filter out potential privacy information from the extracted knowledge.Privacy filtering is performed as part of the data-processing workflow.

A Construction Details … A.4 Examples of High-level Guidance Agent Prompt for Question Graph Parsing

The construction prompts define an automated pipeline that retrieves fresh evidence, verifies reliability and popularity, and parses or expands QA reasoning graphs. Across its stages, the prompts enforce live-web search, multi-hop evidence chains, strict logical fidelity, and adversarial graph operations.

  • A.2 Examples of the QA Synthesis Agent Prompt for Evidence Collection: A.2 Evidence collection iteratively follows search results and visited pages, prioritizes time-aware queries, requires page-level verification, and links items through shared entities.Evidence collection targets events, relations, and attribute updates occurring on or after January 1, 2026.
  • A.3 Examples of Information Filtering Agent Prompt for Fresh Knowledge Reliability Assessment: A.3 Reliability filtering extracts each triple’s key fact, seeks authoritative corroboration, requires at least two independent sources, and labels the evidence list credible, not credible, or unclear.The provided source URL may be consulted but cannot count as independent corroboration.
  • A.3 Examples of Information Filtering Agent Prompt for Fresh Knowledge Reliability Assessment: A.3 Popularity filtering labels an evidence list popular only when most triples describe widely known entities or facts; otherwise it labels the list non-popular.Evidence beyond the assessor’s knowledge scope is also labeled non-popular.
  • A.4 Examples of High-level Guidance Agent Prompt for Question Graph Parsing: A.4 Question-graph parsing represents literal question semantics with nodes and projection, intersection, or complement edges while preserving local connectivity without forcing unrelated subgraphs together.The graph schema records known entities, reference entities, root status, and structured node and edge identifiers.
  • A.4 Examples of High-level Guidance Agent Prompt for Question Graph Parsing: A.4 Graph expansion proposes one added operation—complement, inverse projection, or intersection expansion—to increase adversarial reasoning difficulty.Complement creates exclusionary hard negatives, inverse projection reverses effect-to-cause reasoning, and intersection placement controls difficulty distribution along the path.

B Low Quality QA pairs filtering

The filtering agent evaluates each QA pair and returns only “pass” or “fail,” rejecting any pair that violates a single inspection criterion.

  • Filtering procedure: The evaluator inspects each QA pair and outputs only “pass” or “fail.”Failing any single inspection item results in “fail.”
  • Content integrity: Both question and answer must be non-empty, nonwhitespace, and free of truncation or garbled text.
  • Question language quality: Questions must be fluent, grammatical, clearly structured, and free of machine-generated patchwork or excessive nested modifiers.
  • Question clarity: Questions must be semantically unique and unambiguous, with explicit inquiry targets, definite scope, and no equally valid alternative answers.
  • Evaluability: Questions should be concise and definite enough for automatic evaluation, such as asking for a specific entity, number, or date.

C Human Analyses on Data Quality · D Prompt of LLM-as-a-judge

Human analyses use two bilingual data experts who independently assess evidence correctness, question quality, and answer inferability. The LLM judge compares responses with supplied correct answers using a structured correctness decision and explanation format.

  • C Human Analyses on Data Quality: Two bilingual data experts independently verify whether each evidence item is consistent with or inferable from its source web pages.They are proficient in both English and Chinese.
  • C Human Analyses on Data Quality: The experts independently judge whether each synthesized question matches its evidence and is unambiguous.This assessment addresses consistency between the question and its corresponding evidence set.
  • C Human Analyses on Data Quality: They also assess whether each answer can be inferred from the corresponding evidence.The judgments are conducted independently by both data experts.
  • D Prompt of LLM-as-a-judge: The LLM judge is prompted to determine whether a response to the original question is correct relative to the provided correct answer.The prompt frames the task as judging the response against the supplied reference answer.
  • D Prompt of LLM-as-a-judge: The input template supplies the original question, correct answer, and response as explicit fields for evaluation.These fields are represented as [Original Question], [Correct Answer], and [Response].
  • D Prompt of LLM-as-a-judge: The judge must extract a final answer from the response or return ’None’ when no clear final answer appears.It must explain correctness based on the correct answer while focusing on substantial differences rather than re-solving the problem.
  • D Prompt of LLM-as-a-judge: The evaluation section reports Spearman correlation between the LLM judge and human judge.This correlation is presented in Table 5.
  • D Prompt of LLM-as-a-judge: The judge labels an answer ’Correct’ when it matches the reference or falls within an acceptable small numerical error range.Any inconsistency, ambiguity, inequivalence, or incorrect extracted answer yields ’Incorrect’.

E Selection of the Judge Model

The study selects a reliable judge model by evaluating nine cutting-edge LLMs on 800 randomly sampled predictions from the main experiments.

  • Judge-model selection: 800 randomly sampled predictions were judged using nine candidate LLMs, including GPT-4.1, DeepSeek, Kimi, and GLM variants.The candidates span GPT-4.1; DeepSeek-V4-Flash-Chat, DeepSeek-V4-Flash-Max, DeepSeek-V3.2-Chat, and DeepSeek-V3.2-Think; Kimi-K2.6-Chat and Kimi-K2.6-Think; and GLM-5-Chat and GLM-5-Think.
Loading 2606.13120v1…