PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking

Tian Lan, Bin Zhu, Qianghuai Jia, Junyang Ren, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, Kaifu Zhang

arXiv:2510.20168v1cs.CL

TL;DR

Existing benchmarks do not simultaneously test deep multi-hop reasoning and wide-scale information collection, despite the importance of this combination for real-world information seeking. DeepWideSearch addresses the gap by converting established datasets into a human-verified benchmark of 220 questions across 15 domains. State-of-the-art agents achieve only 2.39% average success, while the authors identify architectural failure modes and release the benchmark for future research.

  • Problem

    Existing benchmarks leave high-width, high-depth information-seeking tasks unaddressed, although they matter for applications such as comprehensive market analysis and business development.

  • Method

    DeepWideSearch converts established deep- and wide-search datasets into human-validated tasks requiring broad candidate collection and deep candidate verification.

  • Results

    2.39% average success rate is achieved by state-of-the-art agents on DeepWideSearch.

  • Takeaways & Limitations

    DeepWideSearch exposes the difficulty of integrating deep reasoning with wide information collection and provides a benchmark for developing more capable information-seeking agents.

  • Takeaways & Limitations

    The dataset construction relies heavily on human annotation, limiting scalability, and Wide2Deep produces significantly easier questions than Deep2Wide.

Abstract

from arXiv · show

Current search agents fundamentally lack the ability to simultaneously perform \textit{deep} reasoning over multi-hop retrieval and \textit{wide}-scale information collection-a critical deficiency for real-world applications like comprehensive market analysis and business development. To bridge this gap, we introduce DeepWideSearch, the first benchmark explicitly designed to evaluate agents to integrate depth and width in information seeking. In DeepWideSearch, agents must process a large volume of data, each requiring deep reasoning over multi-hop retrieval paths. Specifically, we propose two methods to converse established datasets, resulting in a curated collection of 220 questions spanning 15 diverse domains. Extensive experiments demonstrate that even state-of-the-art agents achieve only 2.39% average success rate on DeepWideSearch, highlighting the substantial challenge of integrating depth and width search in information-seeking tasks. Furthermore, our error analysis reveals four failure modes: lack of reflection, overreliance on internal knowledge, insufficient retrieval, and context overflow-exposing key limitations in current agent architectures. We publicly release DeepWideSearch to catalyze future research on more capable and robust information-seeking agents.

1. Introduction

Existing benchmarks evaluate search depth or width separately, leaving high-width, high-depth information seeking largely unaddressed. DeepWideSearch fills this gap with a curated benchmark, where advanced agents achieve only 2.39% average success and exhibit four recurring failure modes.

  • 1. Introduction: High-width, high-depth tasks require extensive candidate collection plus deep verification, exceeding the scope or scale of existing width- and depth-focused benchmarks.The paper motivates this setting with applications such as comprehensive market analysis and business development.
  • 1. Introduction: DeepWideSearch converts established datasets through Deep2Wide and Wide2Deep methods, producing 220 human-verified questions across 15 domains in Chinese and English.The benchmark contains 85 Deep2Wide and 135 Wide2Deep instances.
  • 1. Introduction: 2.39% average success rate is achieved by state-of-the-art agents on DeepWideSearch, demonstrating the difficulty of jointly handling deep reasoning and wide-scale collection.Agent frameworks improve core entity identification by 15.91 absolute percentage points in one example but remain weak at broad information collection.
  • 1. Introduction: Four failure modes recur: insufficient reflection, overreliance on internal knowledge, inadequate retrieval, and context overflow.These findings expose limitations in current agent architectures for deep and wide information seeking.

2. Related Work

Prior information-seeking benchmarks primarily target either multi-hop reasoning depth or broad information collection width. DeepWideSearch introduces a benchmark explicitly designed to evaluate both capabilities together.

  • 2. Related Work: Existing benchmarks assess depth through multi-hop reasoning or width through comprehensive retrieval, but do not jointly evaluate the two capabilities.Examples include GAIA and BrowseComp for depth, and WideSearch, PaSa, and SPAR for width.
  • 2. Related Work: DeepWideSearch is presented as the first benchmark explicitly designed for deep and wide information-seeking tasks.Its purpose is to address the evaluation gap surrounding complex real-world information seeking.

3. Task Formulation

DeepWideSearch defines each task as a complex query paired with a table schema, requiring agents to collect candidates broadly and verify them deeply. Evaluation covers depth, width, and efficiency.

  • 3. Task Formulation: The benchmark evaluates Depth, Width, and Efficiency using Column F1, Row F1, Item F1, Success Rate, token consumption, and cost.Column F1 measures F1 over unique columns representing core entity attributes; width metrics assess comprehensive retrieval at multiple granularities.
  • 3. Task Formulation: Each task is defined as a tuple (Q, C), where Q is a complex query and C is a set of table columns specifying attributes and constraints to collect and verify.Examples include EV price and month-over-month sales growth.
  • 3. Task Formulation: Agents must produce a structured table by combining wide search for numerous candidates with deep search to verify each candidate.The output is a tabular response R.

4. Methodology of Dataset Construction

The dataset construction methodology converts existing deep and wide benchmarks into joint deep-and-wide tasks through human-designed schemas, synthesized sub-questions, and validation. The resulting benchmark emphasizes substantial search volume and multi-step entity identification.

  • 4.1. Convert Deep Search Datasets (Deep2Wide): Deep2Wide expands deep-search questions by selecting suitable core entities, defining table schemas, and exhaustively annotating associated information.The pipeline is inspired by WideSearch and includes human validation; each instance takes approximately 30 minutes to annotate.
  • 4.2. Convert Wide Search Datasets (Wide2Deep): Wide2Deep transforms wide-search queries by replacing explicit entities with synthesized complex sub-questions requiring additional web-search steps.A human-in-the-loop process fuses sub-questions with the original query, then validates uniqueness, complexity, and linguistic naturalness.
  • 4.3. Data Statistics: 414.10 average information units and 4.21 average search steps characterize DeepWideSearch, with 220 instances spanning 15 domains and two languages.The benchmark contains 85 Deep2Wide and 135 Wide2Deep instances; its average entity-search steps are nearly four times WideSearch’s 1.24.
  • 4.4. Evaluation Metrics: The evaluation protocol measures depth, width, and efficiency, and reports Avg@4 and Max@4 across four independent runs, plus Pass@4 for Success Rate.Efficiency uses total input/output tokens and estimated inference cost.

5. Experiments

Experiments evaluate closed-source LLMs, open-source LLMs, and agent systems on DeepWideSearch, showing strong gains in core-entity identification but limited and sometimes degraded wide information collection.

  • 5.2. Main Results: Gemini 2.5 Pro achieves the highest Column F1 (45.27%, Avg@4), Core Entity Accuracy (73.98%, Avg@4), and Pass@4 Success Rate (1.82%) among evaluated LLMs.It even outperforms several agent systems.
  • Depth Metrics: GPT-5’s Core Entity Accuracy (Avg@4) rises from 58.41% as a base LLM to 74.32% in WebSailor, a +15.91 percentage point gain.Claude Sonnet 4 similarly improves from 57.95% to 70.91% under WebSailor.
  • Depth Metrics: Agent systems generally improve deep-search core-entity identification through iterative tool calls and multi-step reasoning, but they do not consistently improve column-level precision.Claude Sonnet 4’s Column F1 (Avg@4) declines from 32.63% in the base LLM to 30.08% in OWL and 21.60% in Smolagents.
  • Depth Metrics: Gemini 2.5 Pro frequently fails in agent systems through invalid markdown tables, incorrect tool-call APIs, and incomplete task solving from inference errors.Inference errors occur in 24.24% of cases on average, versus 16.36% for GPT-5 and 17.80% for Claude Sonnet 4.
  • Width Metrics: Most agent frameworks do not significantly improve wide-search capabilities; only OWL with Claude Sonnet 4, WebSailor with Claude Sonnet 4, and WebSailor with GPT-5 improve across all width metrics.The remaining agents show substantial degradation relative to their counterpart base LLMs.

6. Analysis

DeepWideSearch remains costly and difficult across construction methods, topics, and agent architectures. Analysis identifies persistent weaknesses in wide-scale collection, context management, and reflection despite some gains in entity identification.

  • Efficiency: $2.75 average cost per question for OWL (GPT-5) and $1.40 for WebSailor (Claude Sonnet 4) show substantial resource overhead.Under retry conditions, OWL (GPT-5) exceeds $6.8 average cost per question.
  • Tool calls: 23.23 Search tool calls for WebSailor with Claude Sonnet 4 exceed 4.77 for Gemini 2.5 Pro and 8.72 for GPT-5, aligning with its superior performance.The analysis links higher Search tool-call volume with better WebSailor performance in the reported comparison.
  • Dataset construction: 0.0% Avg@4 for LLMs and 0.15% Avg@4 for agents on Deep2Wide contrast with 1.17% and 1.23% on Wide2Deep, respectively.Deep2Wide also has 33.29% overall Entity Accuracy versus 88.84% for Wide2Deep, while Wide2Deep column-F1 remains below 51%.
  • Per-topic performance: Politics reaches 35% Item F1 and 19% Row F1, whereas History reaches 5% Column-F1, revealing heterogeneous topic difficulty.Finance, Travel, and Education show comparatively strong depth but weaker width, while History and Games underperform across metrics.
  • Error analysis: Four failure patterns recur: ineffective reflection, reliance on internal knowledge, insufficient retrieval, and context overflow affecting 24.96% of cases.Context overflow arises from extensive multi-step reasoning and search calls; incomplete visits and summaries also cause information omissions.

7. Conclusion

The paper introduces DeepWideSearch to evaluate deep reasoning and wide-scale information collection together. State-of-the-art agents achieve only 2.39% average success, exposing substantial limitations and motivating further research.

  • Conclusion: 2.39% average success rate on DeepWideSearch reveals fundamental limitations in current agents handling combined deep and wide information seeking.The conclusion frames the benchmark’s combinatorial complexity as a frontier for future information-seeking research.

8. Limitations and Future Work

The paper identifies construction and evaluation limitations that future work should address. It calls for harder, more realistic, more scalable data and evaluation procedures.

  • Limitations: Wide2Deep produces significantly easier questions than Deep2Wide, as shown by substantially higher CE Accuracy.Future work will refine sub-questions while maintaining natural-language quality.
  • Future Work: Automated data generation and reference-free evaluation metrics could support efficient dataset expansion and model optimization across diverse domains.These approaches would avoid complex human-verified tabular answers.

A. Details of Datasets

The dataset measures search volume and reasoning effort through table-based questions, with Deep2Wide examples spanning scientific and entertainment domains. The appendix illustrates these instances with concrete cases.

  • Dataset measurements: Table volume is defined as the product of table rows and columns, representing the amount of information searched in each question.Average entity-search steps count reasoning steps and tool calls; GAIA uses reference trajectories, while WideSearch uses annotations from three human raters.
  • Dataset cases: Deep2Wide_62 asks about scientists connected through doctoral supervision, Nobel recognition, and a 2023 physics prize.The example requests award titles, birth dates, and nationalities for other award recipients.
  • Dataset cases: Deep2Wide_26 asks for television dramas acted in by a Chinese singer, including titles, directors, and main cast.The case requires identifying the singer and song through layered clues before compiling the requested table.

B. Differences between Our Dataset and Real-world Questions

DeepWideSearch questions combine deep multi-hop identification with wide information collection, while real-world tasks often reverse that order. The dataset’s construction and prompts operationalize this combined search process.

  • Dataset Example: A representative question asks the agent to identify an artist through multiple educational and mentorship clues, then compile information about the artist’s solo exhibitions.The requested exhibition fields include title, venue, city, and dates.
  • Differences between Dataset and Real-world Questions: DeepWideSearch first performs deep search to identify a target entity, then wide search to collect its relevant attributes.Real-world tasks often begin with broad candidate collection and then deeply verify each candidate.
  • Differences between Dataset and Real-world Questions: The dataset’s initial deep phase still requires listing and reasoning over candidates before applying constraint-based verification.This gives the first stage an inherent wide-search characteristic despite its overall deep-then-wide order.
  • Wide2Deep Method: Wide2Deep uses prompts for core-entity extraction, deep sub-question synthesis, and fusion of the deep and wide questions.The method’s extraction prompt prioritizes one specific, concrete, central entity from the query.

D. Error Cases in DeepWideSearch

The error analysis identifies four recurring failure modes in DeepWideSearch agents: insufficient reflection, internal-knowledge overreliance, context overflow, and insufficient retrieval. Representative cases show these failures across complex table-generation tasks and multi-turn tool use.

  • Error Taxonomy: The analysis defines four representative errors: lack of reflection, overreliance on internal knowledge, context overflow, and insufficient retrieval.These categories are illustrated through Figures 13–17 and accompanying agent cases.
  • Lack of Reflection: Lack of reflection appears when agents continue along wrong reasoning trajectories or make incorrect tool calls without correcting course.The examples separately illustrate wrong trajectories and wrong tool calls.
  • Representative Cases: The cases span Wide2Deep and Deep2Wide instances requiring long, structured outputs, including smartphone product histories, competition results, and scientist award information.These examples demonstrate the error analysis in complex multi-hop, wide-collection settings.
  • Overreliance on Internal Knowledge: Overreliance on internal knowledge is illustrated by agents depending on LLM knowledge rather than adequately grounding answers through search.One case is explicitly labeled “Overreliance on Internal Knowledge.”
  • Context Overflow: Context overflow arises during multi-turn tool calls and reasoning, interrupting agents before they finish outputting the requested table.The failure is tied to the accumulation of context across the interaction.
  • Insufficient Retrieval: Insufficient retrieval occurs when complete webpage information is not passed to agents, leaving retrieved evidence incomplete for the requested answer.The associated cases involve broad tables such as athlete results, designated areas, and institutional information.
Loading 2510.20168v1…