PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking

Tian Lan, Bin Zhu, Qianghuai Jia, Junyang Ren, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, Kaifu Zhang

arXiv:2510.20168v1cs.CL

TL;DR

Current benchmarks leave high-width, high-depth information seeking unaddressed despite its relevance to market analysis and business development. DeepWideSearch evaluates this combined capability by converting established datasets through two methods. State-of-the-art agents achieve only 2.39% average success rate on the benchmark.

  • Problem

    Current benchmarks leave high-width, high-depth information seeking unaddressed, despite its relevance to comprehensive market analysis and business development.

    Source: §1 · p. 2

  • Method

    DeepWideSearch evaluates joint deep-and-wide information seeking by converting established datasets through Deep2Wide and Wide2Deep methods with human validation.

    Source: §1 · p. 2

  • Results

    State-of-the-art agents achieve only 2.39% average success rate on DeepWideSearch, underscoring the difficulty of jointly handling deep reasoning and wide-scale information collection.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    The results identify the combinatorial complexity of deep and wide search as a key frontier for future information-seeking agent research.

    Source: §7 · p. 11

    The dataset relies heavily on human annotation, which limits scalability.

    Source: §8 · p. 12

Abstract

from arXiv · show

Current search agents fundamentally lack the ability to simultaneously perform \textit{deep} reasoning over multi-hop retrieval and \textit{wide}-scale information collection-a critical deficiency for real-world applications like comprehensive market analysis and business development. To bridge this gap, we introduce DeepWideSearch, the first benchmark explicitly designed to evaluate agents to integrate depth and width in information seeking. In DeepWideSearch, agents must process a large volume of data, each requiring deep reasoning over multi-hop retrieval paths. Specifically, we propose two methods to converse established datasets, resulting in a curated collection of 220 questions spanning 15 diverse domains. Extensive experiments demonstrate that even state-of-the-art agents achieve only 2.39% average success rate on DeepWideSearch, highlighting the substantial challenge of integrating depth and width search in information-seeking tasks. Furthermore, our error analysis reveals four failure modes: lack of reflection, overreliance on internal knowledge, insufficient retrieval, and context overflow-exposing key limitations in current agent architectures. We publicly release DeepWideSearch to catalyze future research on more capable and robust information-seeking agents.

1. Introduction

DeepWideSearch targets the unbenchmarked combination of large-scale information collection and multi-hop reasoning, and the paper reports substantial difficulty for current agents on these tasks.

Motivation

  • Existing benchmarks address low-width/high-depth, low/low, or high-width/low-depth search, leaving high-width/high-depth information gathering with deep reasoning unaddressed.The paper identifies this combination as important for applications such as comprehensive market analysis and business development.Source: §1 · p. 2

Benchmark construction

  • DeepWideSearch converts established datasets through Deep2Wide table-schema expansion and Wide2Deep complex subquestions, with human validation.The benchmark contains 220 questions across 15 domains, including Chinese and English queries with human-verified ground truths; 85 derive from Deep2Wide and 135 from Wide2Deep. The benchmark, datasets, and evaluation codebase are publicly released.Source: §1 · p. 2

Experimental findings

  • Agents achieve only 2.39% average success rate; frameworks improve Core Entity Accuracy by +15.91 absolute percentage points but often underperform LLM counterparts in wide-scale collection.The reported failures indicate that stronger core-entity identification does not translate into reliable broad information gathering.Source: §1 · p. 2

Error analysis

  • The error analysis identifies four failure modes: ineffective reflection, overreliance on internal knowledge, insufficient retrieval despite relevant webpages, and context overflow.These findings expose limitations in current agent architectures for deep and wide information seeking.Source: §1 · p. 3

2. Related Work

Related work spans LLM-based search-agent design and benchmarks that separately assess multi-hop reasoning or broad information collection. DeepWideSearch addresses the gap in evaluating both capabilities together.

2.1. LLM-based Search Agents

  • LLM-based search-agent research addresses system architecture, synthetic data, retrieval efficiency, multi-hop knowledge management, and performance evaluation.Systems range from closed-source implementations to open-source platforms and support applications such as computer use, deep research, and tool-based retrieval.Source: §2.1 · p. 3

2.2. Benchmarks for LLM-based Agents

  • Existing benchmarks assess depth or width separately, leaving real-world tasks combining extensive exploration and multi-step reasoning unevaluated; DeepWideSearch is designed to fill this gap.Depth examples include GAIA and BrowseComp; width examples include WideSearch, PaSa, and SPAR.Source: §2.2 · p. 3

3. Task Formulation

DeepWideSearch defines tasks that combine broad information collection with deep reasoning and verification. Agents receive a complex natural-language question and schema columns, then produce a structured table of verified candidates.

Task Formulation

  • The framework captures the combined demands of deep reasoning and wide-scale information collection, with Column F1, Row F1, Item F1, and Success Rate described in Section 4.4.Source: §3 · p. 3
  • Each task input is a tuple of a complex natural-language question and schema columns defining attributes and constraints to collect and verify.Examples include EV price and MoM sales growth.Source: §3 · p. 4
  • Agents must return a structured table by broadly collecting numerous candidates and deeply verifying each candidate.Source: §3 · p. 4

4. Methodology of Dataset Construction

DeepWideSearch is constructed by converting established deep- and wide-search datasets into questions that combine multi-hop reasoning with broad information collection, supported by human annotation. Its evaluation measures depth, width, and efficiency.

4.1. Convert Deep Search Datasets (Deep2Wide)

  • Deep2Wide expands deep-search questions through core-entity filtering, table-schema design, and exhaustive web annotation.It samples 80 Chinese BrowseComp-zh questions and 20 English BrowseComp questions [Zhou et al., 2025b; Wei et al., 2025]; each instance takes about 30 minutes to annotate, and timestamps are added to keep answers invariant over time.Source: §4.1 · pp. 4–5

4.2. Convert Wide Search Datasets (Wide2Deep)

  • Wide2Deep adds entity-identification complexity to WideSearch queries by synthesizing deep sub-questions and fusing them with the original queries.The human-in-the-loop pipeline extracts entities from 160 English and Chinese questions, requires a unique answer needing at least one additional web-search step, then has seven master’s-level annotators validate and refine questions; annotation takes about 40 minutes per instance [Wong et al., 2025; Wu et al., 2025b; Li et al., 2025; Tao et al., 2025].Source: §4.2 · pp. 5–6

4.3. Data Statistics

  • DeepWideSearch contains 220 instances across 15 domains in English and Chinese, with 85 from Deep2Wide and 135 from Wide2Deep.It averages 414.10 information units per table and requires 4.21 search steps to identify core entities, compared with 1.24 for WideSearch.Source: §4.3 · p. 6, Table 1 · p. 6

4.4. Evaluation Metrics of DeepWideSearch

  • Evaluation covers depth, width, and efficiency using entity-identification, table-retrieval, and token-and-cost metrics.Depth uses Column-F1 and Core Entity Accuracy; width uses exact-match Success Rate, Row-level F1, and Item-level F1. Efficiency measures input/output tokens and estimated API cost. Each baseline receives four runs per question, reported as Avg@4 and Max@4, plus Pass@4 for Success Rate.Source: §4.4 · pp. 6–7

5. Experiments

Experiments show near-zero success across most baselines, while Gemini 2.5 Pro leads several metrics; tool-using agents can improve entity identification but often fail to improve broad collection or column-level precision.

5.1. Experimental Setup

  • The evaluation compares closed- and open-source LLMs without tools against open-source agents using GPT-5, Claude Sonnet 4, or Gemini 2.5 Pro backbones.All agents used Google Search and webpage-visit tools, and HTML summarization costs counted toward efficiency metrics.Source: §5.1 · p. 7

5.2. Main Results

  • Most baselines achieve near-zero success; Gemini 2.5 Pro leads LLMs with Column F1 45.27% (Avg@4), Core Entity Accuracy 73.98% (Avg@4), and Pass@4 Success Rate 1.82%.The authors attribute Gemini's performance to entity identification and internal knowledge for filling result tables without external search.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • WebSailor raises GPT-5 Core Entity Accuracy (Avg@4) from 58.41% to 74.32% (+15.91 percentage points) and Claude Sonnet 4 from 57.95% to 70.91%.The paper links these depth gains to iterative tool calls and multi-step reasoning in complex information retrieval.Source: §5.2 · p. 8
  • Gemini agent runs show invalid tables, incorrect tool APIs, and incomplete solving in 24.24% of cases on average, versus GPT-5 16.36% and Claude Sonnet 4 17.80%.The authors describe Gemini's output formatting as brittle under multi-step tool orchestration.Source: §5.2 · p. 8
  • Despite depth gains, agents do not consistently improve column precision: Claude Sonnet 4 Column F1 (Avg@4) falls from 32.63% to 30.08% in OWL and 21.60% in Smolagents.The paper concludes that agents may identify core entities through multi-hop reasoning yet fail to collect complete entities reliably.Source: §5.2 · p. 8
  • Only OWL (Claude Sonnet 4), WebSailor (Claude Sonnet 4), and WebSailor (GPT-5) improve consistently across width metrics; Smolagents' minimal pre-tool reasoning yields imprecise queries and poor coverage.Most other agent frameworks show substantial degradation against their base LLM counterparts.Source: §5.2 · pp. 8–9

6. Analysis

The analysis examines operational costs and tool use, compares dataset construction methods, characterizes topic-level performance, and identifies recurring agent failure modes.

6.1. Efficiency Analysis

  • $2.75 per question for OWL (GPT-5) and $1.40 for WebSailor (Claude Sonnet 4), with OWL exceeding $6.8 under retries, reveal high operating costs.Many queries remain unresolved despite the cost; network instability and tool-call errors trigger retries, underscoring scalability concerns and motivating efficient planning, memory reuse, and adaptive resource allocation.Source: §6.1 · p. 9, Table 3 · p. 9

6.2. Tool Calls Analysis

  • WebSailor (Claude Sonnet 4) averages 23.23 Search calls, versus 4.77 for Gemini 2.5 Pro and 8.72 for GPT-5, aligning with its superior performance.The analysis suggests that scaling Search calls improves performance.Source: §6.2 · p. 9, Table 4 · p. 9

6.3. Differences in Dataset Construction Methods

  • 0.0% Avg@4 for LLMs and 0.15% Avg@4 for agents on Deep2Wide, versus 1.17% and 1.23% on Wide2Deep, show Deep2Wide is harder.Entity Accuracy is 33.29% on Deep2Wide versus 88.84% on Wide2Deep; Wide2Deep column-F1 remains below 51%, so comprehensive entity collection is still challenging.Source: §6.3 · pp. 9–10, Table 5 · p. 10

6.4. Per-topic Performance Analysis

  • Politics leads width (35% Item-F1; 19% Row-F1) and, with Finance, depth; Finance, Travel, and Education show weak width despite strong depth, while History and Games underperform.History reaches only 5% Column-F1. The five most frequent topics are Film & Movies, Politics, Finance, Technology, and Sports.Source: §6.4 · p. 10, Figure 6 · p. 11

6.5. Error Analysis

  • Agents fail through absent reflection, reliance on internal knowledge, incomplete retrieval, and context overflow, which occurred in 24.96% of cases.Agents may abandon wrong trajectories or tool errors, produce tables from parametric knowledge that is outdated or inaccurate, and omit details through incomplete visits or summaries. Long multi-step reasoning and numerous search calls expand context; the analysis points to question-aware webpage summarization and specialized architectures as needs.Source: §6.5 · pp. 10–11

7. Conclusion

DeepWideSearch is introduced as the first benchmark to assess deep reasoning and wide-scale information collection together. State-of-the-art agents achieve only 2.39% average success, underscoring the challenge as a frontier for future research.

  • DeepWideSearch is the first benchmark designed to assess deep reasoning and wide-scale information collection simultaneously.Source: §7 · p. 11
  • 2.39% average success rate by state-of-the-art agents reveals fundamental limitations on this challenging benchmark.Source: §7 · p. 11
  • The combinatorial complexity of deep and wide search marks a key frontier for research toward more capable information-seeking agents.Source: §7 · p. 11

8. Limitations and Future Work

The authors identify limitations in question difficulty, real-world solution-path alignment, and human-dependent dataset construction, and propose refinements to address them.

Limitations and Future Work

  • Wide2Deep produces easier questions than Deep2Wide, as indicated by substantially higher CE Accuracy; the authors plan to refine sub-questions while preserving natural language quality.The comparison is reported in Table 5.Source: §8 · p. 12
  • The dataset’s solution paths differ slightly from real-world deep and wide search questions, motivating iterative refinement toward better application alignment.The paper points to cases in Appendix B.Source: §8 · p. 12
  • Human annotation limits dataset-construction scalability, so future work should explore automated data generation and reference-free evaluation to support efficient expansion and model optimization across domains.The proposed metrics would avoid complex, human-verified tabular answers.Source: §8 · p. 12

A. Details of Datasets

This section defines how DeepWideSearch measures question volume and search effort, then illustrates the dataset with science and entertainment questions requiring multi-hop retrieval and structured outputs.

A. Details of Datasets

  • Table volume is the product of rows and columns in the searched information table.Table 1’s average search-entity steps count reasoning steps and tool calls.Source: A. Details of Datasets · p. 15
  • Average search steps use GAIA reference trajectories and annotations from three human raters for WideSearch.The measure counts reasoning steps and tool calls.Source: A. Details of Datasets · p. 15
  • Dataset cases span a Nobel-prize-related scientist question and an entertainment question about a singer’s TV dramas, both requiring tabular answers.The former asks for scientists awarded in the same year as a 2023 prize recipient; the latter asks for drama titles, directors, and main cast.Source: A. Details of Datasets · pp. 15–16, Figure 7 · p. 15, Figure 8 · p. 16

B. Differences between Our Dataset and Real-world Questions · C. Prompts for DeepWideSearch Data Construction

DeepWideSearch differs from many real-world tasks in search order: it emphasizes deep search before wide collection, while real-world tasks often gather candidates widely before verifying them deeply. Its Wide2Deep construction uses prompts for core-entity extraction, deep sub-question synthesis, and question fusion.

B. Differences between Our Dataset and Real-world Questions

  • A representative dataset question identifies an artist through multi-hop clues, then gathers solo-exhibition titles, venues, cities, and dates.The stated solution path first identifies the artist through deep/wide search, then performs wide search for information collection.Source: B. Differences between Our Dataset and… · p. 17
  • DeepWideSearch emphasizes deep search followed by wide attribute collection, whereas real-world tasks often gather candidates widely before deeply verifying each one.The dataset’s initial deep phase also involves listing and reasoning over candidates, so it retains a wide-search characteristic.Source: Figure 9 · p. 17, B. Differences between Our Dataset and… · p. 17
  • During its initial deep search, DeepWideSearch requires candidate listing and reasoning with systematic verification, thereby incorporating a characteristic of wide search.This qualification explains why the dataset retains deep-and-wide traits despite its different solution path from many real-world tasks.Source: B. Differences between Our Dataset and… · p. 17

C. Prompts for DeepWideSearch Data Construction

  • The Wide2Deep construction section presents prompts for core-entity extraction, deep sub-question synthesis, and question fusion.The section associates the three prompts with Figures 10, 11, and 12, respectively.Source: C. Prompts for DeepWideSearch Data… · p. 17
  • The core-entity extraction prompt prioritizes one central, specific, concrete entity mentioned in the query.The supplied prompt passage begins a further condition for cases where no such entity is present, but does not provide its completion.Source: Figure 10 · p. 18, C. Prompts for DeepWideSearch Data… · p. 18
  • The deep sub-question synthesis prompt requires a multi-hop query whose only unambiguous answer is the target entity, without overly specific information.The supplied prompt frames this as gathering available information about the entity and synthesizing a query from it.Source: Figure 11 · p. 19, C. Prompts for DeepWideSearch Data… · p. 19

D. Error Cases in DeepWideSearch

The section identifies four representative agent error types: lack of reflection, overreliance on internal knowledge, context overflow, and insufficient retrieval.

D. Error Cases in DeepWideSearch

Related papers

Loading 2510.20168v1…