Source-linked AI summary
WideSearch: Benchmarking Agentic Broad Info-Seeking
Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao, Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, Wenhao Huang, Yang Wang, Ke Wang
TL;DR
Large-scale information-seeking tasks are repetitive but require reliable and complete collection of many verifiable facts, a capability that existing benchmarks do not adequately test. WideSearch introduces a manually curated benchmark and evaluates more than 10 agentic search systems on structured collection tasks. Most systems achieve success rates near 0%, with the best reaching only about 5%, while the analysis identifies deficiencies in planning, reflection, evidence use, and factual grounding.
Problem
Existing benchmarks do not adequately evaluate reliable and complete large-scale information seeking, where agents must collect many atomic facts and organize them into a structured output.
Method
WideSearch is a benchmark built through a multi-stage human-centered collection and validation process and used to evaluate more than 10 single-agent, multi-agent, and commercial search systems.
Results
Most systems achieve table-level success rates near 0%, with the top performer reaching only 5%, despite item-level F1 scores approaching 80% with sufficient retries.
Takeaways & Limitations
Current search agents remain critically deficient on large-scale information seeking, especially in producing comprehensive and faithful outputs at scale.
Takeaways & Limitations
Agents exhibit incomplete query decomposition, weak reflection and iterative refinement, evidence-utilization failures, and hallucination or factual inconsistency when search results are insufficient.
Abstract
from arXiv · showhide
From professional research to everyday planning, many tasks are bottlenecked by wide-scale information seeking, which is more repetitive than cognitively complex. With the rapid development of Large Language Models (LLMs), automated search agents powered by LLMs offer a promising solution to liberate humans from this tedious work. However, the capability of these agents to perform such "wide-context" collection reliably and completely remains largely unevaluated due to a lack of suitable benchmarks. To bridge this gap, we introduce WideSearch, a new benchmark engineered to evaluate agent reliability on these large-scale collection tasks. The benchmark features 200 manually curated questions (100 in English, 100 in Chinese) from over 15 diverse domains, grounded in real user queries. Each task requires agents to collect large-scale atomic information, which could be verified one by one objectively, and arrange it into a well-organized output. A rigorous five-stage quality control pipeline ensures the difficulty, completeness, and verifiability of the dataset. We benchmark over 10 state-of-the-art agentic search systems, including single-agent, multi-agent frameworks, and end-to-end commercial systems. Most systems achieve overall success rates near 0\%, with the best performer reaching just 5\%. However, given sufficient time, cross-validation by multiple human testers can achieve a near 100\% success rate. These results demonstrate that present search agents have critical deficiencies in large-scale information seeking, underscoring urgent areas for future research and development in agentic search. Our dataset, evaluation pipeline, and benchmark results have been publicly released at https://widesearch-seed.github.io/
1 Introduction
WideSearch targets a largely unevaluated class of large-scale information-seeking tasks where agents must collect and organize comprehensive atomic information. The benchmark addresses the need to measure reliability at scale, where efficiency gains coexist with stringent completeness and fidelity requirements.
- Problem: WideSearch tasks require agents to acquire all large-scale atomic information meeting specified criteria and arrange it in a well-organized output.Examples include identifying qualifying companies or matching job vacancies across multiple constraints.
- Motivation: Existing agent benchmarks do not adequately evaluate this class of comprehensive, large-scale information-seeking tasks.The paper positions WideSearch as a systematic evaluation of search reliability in authentic user scenarios.
- Problem: These tasks are operationally overwhelming rather than primarily cognitively difficult, making them distinct from benchmarks focused on locating or deeply investigating a specific topic.WideSearch addresses the “I could do it, but the sheer volume is overwhelming” problem.
- Benchmark: WideSearch combines a multi-stage data collection and verification framework with hybrid automated evaluation to assess these tasks objectively.The benchmark is introduced specifically to quantify agent performance and reliability at scale.
- Core challenge: The benchmark exposes a large gap between finding individual facts and producing a complete, accurate large-scale result.A single omission, error, or extra datum can cause total task failure, even when item-level F1 approaches 80% with sufficient retries.
2 Related Work
Prior search-agent benchmarks progressed from single-query fact retrieval to multi-hop reasoning, deep investigation, and report synthesis. WideSearch extends this landscape by evaluating broad information gathering across multiple entities in a structured table, while prior search-agent systems remain insufficiently evaluated on this capability.
- Benchmark evolution: Early benchmarks such as Natural Questions and TriviaQA primarily tested single-query retrieval or information already present in parametric knowledge.Multi-hop datasets later increased difficulty by requiring agents to connect multiple pieces of evidence.
- DeepSearch: DeepSearch benchmarks evaluate intensive, vertical investigations into a single complex topic through multi-hop reasoning, tool use, and uncertainty reduction.Examples include GAIA, Xbench-DeepSearch, and BrowseComp-en/zh.
- DeepResearch: DeepResearch Bench evaluates whether agents can answer PhD-level questions and synthesize findings into detailed, accurate reports.This differs from WideSearch’s structured collection across many parallel entities.
- WideSearch: WideSearch evaluates broad information gathering across multiple parallel entities by requiring agents to populate a structured table.Its focus is distinct from deep reasoning on a single query.
- Search agents: Existing search-agent systems include proprietary and open-source model-centric and architecture-centric efforts, but their broad information-seeking performance has not been thoroughly evaluated.WideSearch is presented as the first benchmark specifically designed to assess this capability.
3 WideSearch Benchmark
WideSearch is a manually curated benchmark for evaluating agents that gather and verify broad, multi-entity information into structured outputs. Its 200 tasks span two languages and 18 topics, with human studies showing substantial time and source breadth requirements.
- Task Definition: WideSearch tasks require agents to populate a predefined table by identifying all entities satisfying a query and filling each entity’s requested attributes from the live web.The resulting table is evaluated against a ground-truth table for completeness and factual accuracy.
- Task Construction Methodology: Each task is manually curated by domain experts and designed for high search volume, objective verifiability, public accessibility, external-tool reliance, and scenario diversity.These principles distinguish broad, multi-step collection from singular deep investigations and support reproducible evaluation.
- Task Construction Methodology: A five-stage curation and validation pipeline transforms real-world queries into standardized tasks, accepting them only after all filtering stages and an iterative validation loop.The pipeline is intended to ensure that tasks are realistic, complex, verifiable, resistant to simple memorization, and aligned with human judgment.
- Benchmark Composition and Statistics: 200 tasks are balanced equally between English and Chinese and distributed across 18 diverse topics to support cross-lingual evaluation and broad applicability.The topic balance is intended to mitigate domain-specific bias.
- Evaluation Framework and Metrics: 2.33 hours is the overall average human completion time, with English tasks averaging 2.29 hours and Chinese tasks 2.37 hours.This measure covers query comprehension, multi-step searching, information synthesis, and final validation.
- Evaluation Framework and Metrics: 44.10 unique web pages per task is the average research breadth, comprising 48.74 pages for Chinese tasks and 39.46 for English tasks.The count reflects actual research, including cross-verification across multiple sources.
4 Experiments
Experiments compare single-agent, multi-agent, commercial, and human performance on WideSearch, alongside validation of the automated evaluation pipeline. Results show severe agent weaknesses on broad collection, a multi-agent advantage, and high agreement with human judgments.
- Experimental Setup: The experiments evaluate single-agent, multi-agent, commercial end-to-end, and human systems on WideSearch.The modular agents use search and webpage-reading tools; the single-agent completes the lifecycle, while the multi-agent divides queries into parallel subtasks.
- Main Findings: Current models struggle with planning, reflection, dynamic search adjustment, and evidence use, producing extremely low success rates.Failures include incomplete sub-query decomposition, abandoning unsuccessful searches, and misinterpreting or misattributing retrieved content.
- Main Findings: Multi-agent mode consistently outperforms single-agent mode, especially on F1 scores measuring partial correctness.Its planner decomposes broad queries into parallel sub-tasks, improving search breadth and efficiency through division of labor.
- Human Performance: 20% is the success rate for a single human completing tasks independently, even with ample time and access to any tools.The benchmark’s dense outputs can contain thousands of facts, and one extra, missing, or incorrect item causes task failure under strict criteria.
5 Analysis
The analysis identifies inadequate recall and several advanced and basic failure modes as central obstacles to reliable WideSearch performance.
- Inadequate recall is the primary bottleneck, with performance remaining far from optimal at both row and item levels.The analysis frames the core challenge as comprehensively capturing information rather than merely retrieving isolated facts.
- Challenges in Advanced Agentic Capabilities: Incomplete query decomposition causes agents to miss constraints or inquiry scopes during multifaceted searches.This reflects weaknesses in complex task planning and structured decomposition.
- Challenges in Advanced Agentic Capabilities: Agents often abandon searches after insufficient tool results instead of reformulating queries or adjusting search criteria.The reported failure reflects missing reflection and iterative refinement.
- Challenges in Advanced Agentic Capabilities: Failure in evidence utilization occurs when agents misinterpret, disregard, or inadequately validate retrieved sources before using them.This creates a gap between information retrieval and generation.
- Challenges in Advanced Agentic Capabilities: Knowledge hallucination and factual inconsistency arise when agents substitute internal knowledge for relevant externally retrieved information.The analysis emphasizes the need to ground outputs in externally verifiable sources.
- Basic Failure Modes: Basic failures include tool invocation, output formatting, context-length exceedance, and response refusal errors.These failures can directly terminate the task workflow.
6 Test-time Scaling
The test-time scaling experiments repeatedly sample a single agent with search tools and evaluate success and F1 metrics as compute increases. Multi-agent parallel search and cross-validation are identified as an important future direction.
- Test-time Scaling: N expands from 1 to 128 in repeated single-agent trials using Kimi K2 with Search and Web Browse tools.
- Test-time Scaling: The experiments record Success Rate with Pass@N, plus Row-level F1 Score and Item-level F1 Score with Max@N.
- Test-time Scaling: Multi-agent architectures are presented as an important future research direction because agents can search in parallel and cross-validate one another.The authors relate this process to human annotation.
7 Conclusion
WideSearch evaluates agents on extensive structured web-information gathering and integration. Across more than 10 systems, table-level success is extremely low, with failures tied chiefly to comprehensive atomic-information collection and several advanced agentic deficiencies.
- Conclusion: WideSearch is a benchmark for gathering and integrating extensive structured information from the web, termed wide information seeking.
- Conclusion: Most systems score near 0% on table-level tasks, while the top performer reaches only 5%.
- Conclusion: The principal failure is finding all atomic information accurately and comprehensively, rather than finding individual facts.Item-level F1 scores can nevertheless be high.
- Conclusion: Key deficiencies include incomplete query decomposition, insufficient reflection after search failures, and incorrect use of retrieved evidence.
A Models and API Identifiers
The paper documents how benchmark aliases correspond to model API identifiers and notes configuration distinctions for similarly identified models.
- Models and API Identifiers: Table 4 lists each model’s Benchmark Alias alongside its corresponding API Identifier.The alias is the shorthand used for the model within the paper.
- Models and API Identifiers: Doubao-Seed-1.6 Thinking and Non-Thinking configurations share an API identifier but differ in whether thinking is enabled.
- Models and API Identifiers: Claude Sonnet 4 Thinking denotes a configuration with the thinking feature enabled.
B Detailed Experiments
The experiments report multiple row- and item-level metrics across WideSearch subsets, supported by single-agent and multi-agent search frameworks with iterative tool use and optional sub-agents.
- WideSearch results include Success Rate, row-level precision, recall, and F1, plus item-level precision, recall, and F1.
- Multi-Agent Framework: The multi-agent framework adds a Create Sub-Agent tool for delegating specific tasks based on input prompts.
- Single-Agent Framework: The single-agent framework is instructed to decompose complex user queries, use search tools, and reassess incomplete or inaccurate results.
- Search Tools: Search queries should be concise and decomposed into multiple steps, with revised queries used when initial results are unhelpful.
- Search Tools: A link-reading tool opens webpages or PDFs and summarizes relevant information from valuable links returned by search or prior browsing.
D Evaluation Details
The evaluation maps semantically equivalent schema values before matching tables, then combines rule-based checks with column-wise LLM judging for harder fields.
- Semantic Alignment: Semantic mapping aligns equivalent column names and primary-key entities before inner-joining model and reference tables.
- Semantic Alignment: The alignment prompt preserves each original value while transforming matched expressions into the corresponding reference vocabulary.
- LLM-as-Judge Evaluation: For non-primary-key columns that rule-based methods cannot evaluate, the benchmark uses annotated LLM-as-judge criteria applied column by column.
- LLM-as-Judge Evaluation: The judge scores each response-reference pair on a 0-to-1 scale, with only integer scores of 0 or 1 allowed.
- LLM-as-Judge Evaluation: An example accepts semantically equivalent expressions such as “1mtokens” for “1m” while assigning binary scores to individual items.
E Error Analysis and Examples
The error analysis uses Gemini-2.5-pro trajectories to illustrate failures in decomposition, iterative refinement, evidence attribution, and factual grounding.
- The examples analyze user tasks, agent actions, retrieved evidence, and the specific error associated with each trajectory.
- Incomplete Query Decomposition: Incomplete query decomposition omits follow-up searches for university websites, deadlines, and fees after identifying the universities and rankings.
- Lack of Reflection and Iterative Refinement: Lack of reflection and iterative refinement occurs when a broad World Bank query returns regional GDP data but the agent does not pursue country-specific information.
- Failure in Evidence Utilization: Failure in evidence utilization arises when the agent reports a correct GPA value from the University of Houston as though it came from Harvard.
- Knowledge Hallucination and Factual Inconsistency: Knowledge hallucination and factual inconsistency occur when the agent invents a $15 future entrance fee after the search returns no evidence, instead of reporting unavailability.
F Domain-Specific Performance Analysis
Domain-specific analysis compares models and agent frameworks across English and Chinese domains using row-level F1, revealing broad multi-agent gains alongside domain and model differences.
- The analysis evaluates English and Chinese datasets across domains to examine model strengths, weaknesses, and language-dependent performance variation.
- Figure F.1: Figure F.1 heatmaps row-level F1 scores, placing English domains left and Chinese domains right, with single-agent results above multi-agent results.
- Framework Comparison: The multi-agent framework consistently improves performance across nearly all models and domains.
- Domain Comparison: Academics and Transportation are challenging across models, while model aptitudes differ across Healthcare, Automotive, and Law.
- Reported Metrics: Table 5 is presented as the detailed experiment-results table for the WideSearch benchmark.