Source-linked AI summary
DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, Hidekazu Oiwa, Elena Gribovskaya, Jan Ackermann, John Blitzer, Sasha Goldshtein, Dipanjan Das
TL;DR
DeepSearchQA addresses the limited coverage of single-answer evaluations for open-ended, exhaustive information seeking. It introduces a 900-prompt benchmark centered on multi-source answer-set generation and evaluates leading agents, finding persistent difficulty balancing recall and precision. The benchmark is intended to diagnose these comprehensiveness failures, while its outcome-only and static-web design impose important limits.
Problem
Existing single-answer benchmarks prioritize precise retrieval, leaving exhaustive multi-source research, entity resolution, and stopping-criterion reasoning insufficiently evaluated.
Method
DeepSearchQA uses 900 curated prompts with single- and set-answer formats to evaluate final answer sets for completeness and correctness.
Results
Even advanced agents struggle to balance recall and precision, while deep research agents outperform standalone reasoning models on the benchmark.
Takeaways & Limitations
DeepSearchQA provides a measurable diagnostic framework for systematic exploration, multi-source synthesis, and search completion in web agents.
Takeaways & Limitations
Outcome-only evaluation cannot distinguish correct reasoning from inefficient or accidental success, and the static-web assumption limits volatile breaking-news retrieval.
Abstract
from arXiv · showhide
We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks that target single answer retrieval or broad-spectrum factuality, DeepSearchQA features a dataset of challenging, handcrafted tasks designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.
1. Introduction
DeepSearchQA targets a comprehensiveness gap left by single-answer benchmarks: real-world research often requires exhaustive, multi-source answer sets. It evaluates systematic collation, entity resolution, and stopping-criterion reasoning rather than only precise retrieval.
- Introduction: DeepSearchQA is positioned as a diagnostic response to the need for realistic evaluation of agents reasoning over external, unstructured web information.The broader motivation is to identify failure modes in increasingly autonomous, multi-step research systems.
- The Prevailing Paradigm: Single-Answer Verification: Single-answer benchmarks emphasize precision-first search, whereas many real-world information needs require recall-oriented exhaustive research.Traditional benchmarks helped isolate navigation and search strategy, but do not fully represent tasks requiring complete answer lists.
- The Comprehensiveness Gap: Beyond Finding an Answer: The Comprehensiveness Gap arises because tasks such as identifying all qualifying companies or clinical trials require more than retrieving one data point.These tasks involve open-ended information aggregation under explicit constraints.
- The Comprehensiveness Gap: Beyond Finding an Answer: DeepSearchQA isolates systematic collation, entity resolution, and stopping-criteria reasoning as higher-order capabilities missing from existing evaluations.Agents must aggregate fragmented sources, deduplicate varying entity forms, and decide when search is complete under uncertainty.
- DeepSearchQA Core Contributions: The benchmark shifts evaluation from precision-based retrieval to exhaustive answer-set generation judged by final-set recall and precision.Its outcome-based protocol permits architectural diversity while requiring agents to balance exploration with candidate verification.
2. DeepSearchQA: Dataset and Taxonomy
DeepSearchQA is a diverse benchmark of curated, objectively answerable web questions organized by domain, answer type, and task complexity. Its examples emphasize multi-step retrieval, context management, reasoning, and cross-source constraint satisfaction.
- Dataset Statistics and Domains: The dataset contains 900 prompts spanning diverse domains and is curated to represent realistic, high-value information-seeking queries with definitive ground truth.Its domain breadth tests generalization across varying web structures and content types.
- Answer Types: DeepSearchQA distinguishes single-answer tasks from set-answer tasks, including enumeration and composite responses.Set answers require collections of items rather than a unique entity or value.
- Quality Verification Protocol: Ground-truth quality is supported by independent research, verification against curator answers, and conflict resolution for discrepancies or ambiguity.This protocol can update the original answer when review identifies an error or flawed prompt.
- Representative Tasks: Representative financial tasks require retrieving data from disparate sources, calculating annual growth, and intersecting multiple thresholds.The workflow combines source synthesis with mathematical constraint satisfaction.
- Task Complexity Taxonomy: The benchmark taxonomy diagnoses structured retrieval, context management, and logical reasoning demands placed on agents.These categories cover search strategy, synthesis under context limits, deduction, planning, and conflict resolution.
3. Evaluation Methodology
DeepSearchQA evaluates submitted answer sets with information-retrieval metrics and categorical correctness labels. The methodology rewards complete, accurate sets while penalizing omissions, extraneous guesses, and unresolved ambiguity.
- Formal Evaluation Metrics: Submitted sets S_i are compared with ground-truth sets G_i using a two-tier evaluation of continuous metrics and categorical success classes.The framework supports granular performance analysis alongside strict response classification.
- Continuous Metrics: Precision, Recall, and F1 are computed per prompt and averaged, with F1 serving as the primary ranking metric for balancing accuracy and exhaustiveness.F1 is equivalent to exact-match accuracy for single-answer tasks and balances completeness with accuracy for set-answer tasks.
- Categorical Classification: A fully correct response exactly matches the ground-truth set, requiring all correct items and no incorrect additions.For set answers, this means both Recall and Precision equal 1.0.
- Categorical Classification: Partial correctness captures retrieving some but not all ground-truth items, while fully incorrect responses retrieve no valid item for set-answer tasks.These categories distinguish under-retrieval severity from total search failure.
- Categorical Classification: Correct-with-extraneous-answers identifies the hedging failure mode: all correct items are found, but incorrect items reduce Precision below 1.0.For single-answer tasks, multiple candidates indicate failure to disambiguate; for set answers, they indicate failure to stop.
- Semantic Evaluation: An automated LLM judge assesses semantic equivalence between submitted and ground-truth items using Gemini 2.5 Flash in a zero-shot setting.The judge checks each submitted item against the ground-truth set.
4. Results and Analysis
DeepSearchQA results show that leading deep-research agents outperform standalone reasoning models, while top systems still exhibit distinct comprehensiveness trade-offs and failure patterns.
- Evaluation Metrics: Table 4 reports Fully Correct, Fully Incorrect, Correct with Extraneous Answers, and F1-Score for state-of-the-art agents and reasoning models.The table defines Fully Correct as exact ground-truth-set retrieval and Fully Incorrect as finding no relevant items; F1 balances precision and recall.
- State-of-the-Art in Deep Research: Deep Research Agents outperform standalone reasoning models, establishing a clear hierarchy on DeepSearchQA.The results are described as confirming that iterative agent loops are required alongside reasoning ability to address the Comprehensiveness Gap.
- Comparative Performance: 66.09% and 65.18% fully correct rates are statistically comparable for GPT-5 Pro High Reasoning and Gemini Deep Research Agent, respectively.Gemini records the lower Fully Incorrect rate: 9.95% versus 14.13% for GPT-5 Pro High Reasoning.
- Failure Analysis: Table 5 categorizes common failures by eliciting prompts, reasoning or tool limitations, and exact model excerpts.The failure analysis focuses on Gemini Deep Research Agent and GPT-5 Pro High Reasoning.
5. Future Work
The paper identifies outcome-only evaluation and static-web assumptions as limitations, while proposing process diagnostics and dynamic tasks as extensions and outlining capabilities needed for stronger agents.
- Limitations: Outcome-based evaluation treats agents as black boxes and cannot distinguish correct reasoning from inefficient or accidental success without trajectory data.The proposed remedy is to collect auxiliary trajectory information such as pages visited and query sequences.
- Limitations: The Static Web Assumption limits evaluation of breaking-news retrieval because websites and ground truths can change over time.The dataset therefore requires periodic manual reviews and updates when sources are removed or substantially altered.
- Methodological Extensions: Future benchmark versions could add process-based metrics to distinguish retrieval failures, reasoning errors, and synthesis issues.Scoring would remain outcome-based while trajectory categories provide diagnostic insight.
- Methodological Extensions: Dynamic and time-sensitive lists could test real-time retrieval under volatile ground-truth conditions.The proposed live questions would address information that changes over time, such as current institutional memberships.
- Implications for Advancements in Agent Architecture: High benchmark performance will likely require systematic exploration, advanced information synthesis, and dynamic stopping criteria.These capabilities cover exhaustive navigation, entity resolution and deduplication, and deciding when further search is unproductive.
6. Conclusion
DeepSearchQA targets the comprehensiveness gap by evaluating systematic, multi-source research and balancing exhaustive retrieval against precision. Its evaluation shows that advanced models still struggle with this recall–precision trade-off.
- DeepSearchQA addresses the Comprehensiveness Gap with a set-based framework that penalizes both under-retrieval and hedging.
- The benchmark measures systematic exploration, multi-source synthesis, and search completion using Precision, Recall, and F1-score.
- Even advanced models struggle to balance recall and precision when producing exhaustive answer sets.
7. Contributions and Acknowledgements
The paper credits contributors across experimental design, organization, early experimentation, sponsorship, authorship, and external support.
- Contributions: Experimental design and reported experiments were led by Nikita Gupta, Riju Chatterjee, and Hidekazu Oiwa.
- Contributions: Connie Tao, Dipanjan Das, Lukas Haas, and John Blitzer managed the effort from start to completion.
- Contributions: Emily Ingebricson and eight collaborators contributed ideas, data collection, and experiments during early experimentation.
- Contributions: Srinivasan Venkatachary, Koray Kavukcuoglu, Slav Petrov, Ya Xu, and Yossi Matias sponsored the effort and provided technical guidance.
- Acknowledgements: All authors wrote parts of the report, while the Gemini team, Kaggle team, expert annotators, and reviewers provided additional support.
Grader Prompt
The grader determines whether an AI response matches the expected answer under a specified prompt type, then returns a structured JSON assessment with explanations and answer-level details.
- The task evaluates whether an AI response provides the correct answer based on the prompt type and correct answer.
- Prompt Types: For a Single Answer, the response must address the user’s question without requiring exact wording.
- Prompt Types: For a Set Answer, correctness requires checking each ground-truth item, while order may be irrelevant and extra answers are separately assessed.
- Output Fields: The grader provides an explanation, per-answer correctness details, and a list of excessive answers.
- Output Format: The output must be a nested JSON dictionary with an Answer Correctness key, and NULL is required when any required input is empty.
Grader Partial Output Example
The example illustrates a correctness assessment in which expected answers are marked individually and an unlisted response is identified as excessive.
- The example explains that Belgium and France were correctly identified, while Italy was an excessive answer.
- Its correctness details mark Belgium and France as true.
- The example is presented after instructions to evaluate the supplied prompt, response, and correct answer.