Source-linked AI summary
DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, Zhendong Mao
TL;DR
DeepResearch Bench addresses the absence of a comprehensive benchmark for evaluating Deep Research Agents and the difficulty of judging their open-ended reports. It introduces a 100-task, 22-domain benchmark with RACE and FACT, whose reliability is supported by human studies and whose protocols are publicly released.
Problem
Existing evaluation frameworks often assess isolated capabilities, while complex DRA reports lack definitive ground-truth answers.
Method
The paper constructs a 100-task benchmark across 22 domains and introduces RACE for report quality and FACT for citation accuracy and effective supported citations.
Results
Human studies show high consistency between the RACE and FACT evaluation frameworks and human judgments.
Takeaways & Limitations
DeepResearch Bench provides a comprehensive evaluation resource for DRA report generation and web retrieval capabilities.
Takeaways & Limitations
The authors cannot confirm whether Deep Research model iterations occurred during the reference collection period.
Abstract
from arXiv · showhide
Deep Research Agents are a prominent category of LLM-based agents. By autonomously orchestrating multistep web exploration, targeted retrieval, and higher-order synthesis, they transform vast amounts of online information into analyst-grade, citation-rich reports--compressing hours of manual desk research into minutes. However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we present DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. Evaluating DRAs is inherently complex and labor-intensive. We therefore propose two novel methodologies that achieve strong alignment with human judgment. The first is a reference-based method with adaptive criteria to assess the quality of generated research reports. The other framework is introduced to evaluate DRA's information retrieval and collection capabilities by assessing its effective citation count and overall citation accuracy. We have open-sourced DeepResearch Bench and key components of these frameworks at https://github.com/Ayanami0730/deep_research_bench to accelerate the development of practical LLM-based agents.
1 Introduction
DeepResearch Bench addresses the lack of comprehensive, end-to-end evaluation for Deep Research Agents, whose complex reports lack definitive ground truth and whose existing benchmarks assess isolated capabilities. It introduces a 100-task, 22-domain benchmark and the RACE and FACT frameworks, validated through human studies.
- Existing DRA evaluation is difficult because internal reasoning and retrieval are opaque, while complex research queries rarely have definitive ground-truth answers.
- Current benchmarks commonly assess isolated browsing, retrieval, or generation capabilities rather than end-to-end deep research.
- DeepResearch Bench contains 100 expert-crafted tasks spanning 22 domains, with domain task counts guided by analysis of over 96,000 real-world user queries.
- RACE evaluates final report quality, while FACT evaluates information retrieval effectiveness and citation accuracy.
- Human studies validate the reliability of the proposed evaluation frameworks, and the benchmark and protocols are publicly released.
2 DeepResearch Bench: Topic Distribution Analysis & Data Collection
The benchmark is built from anonymized real-world search queries, filtered and categorized into 22 domains, then proportionally compressed into 100 balanced tasks. Domain experts propose challenging tasks that undergo manual quality screening.
- Topic Distribution Analysis & Data Collection: The authors collect 96,147 anonymized raw user queries from interactions with a web-search-enabled LLM chatbot.
- Topic Distribution Analysis & Data Collection: Deep research tasks are defined as requiring multiple web searches, information gathering, analysis, and production of high-quality reports.
- Topic Distribution Analysis & Data Collection: 44,019 queries remain after filtering for alignment with the deep research task definition.
- Topic Distribution Analysis & Data Collection: The filtered queries are classified across 22 topic domains, whose aggregated distribution represents real-world user demand.
- Benchmark Task Collection: The observed domain distribution is proportionally compressed into 100 benchmark tasks, split evenly between Chinese and English while preserving topical balance.
- Benchmark Task Collection: Ph.D. researchers or senior practitioners propose challenging tasks, which are manually screened for quality, clarity, complexity, and alignment with the task definition.
3 Evaluation Methodology
The evaluation methodology uses complementary frameworks for report quality and web retrieval. RACE adapts criteria and weights around a high-quality reference, while FACT extracts cited claims and verifies their source support.
- RACE: RACE addresses the limitations of fixed checklists and static rubrics by providing adaptive evaluation for diverse tasks, domains, and quality dimensions.
- RACE: RACE derives task-specific weights from four expert-defined dimensions: Comprehensiveness, Insight/Depth, Instruction-Following, and Readability.
- RACE: For each dimension, the Judge LLM generates tailored criteria and normalized criterion weights for evaluating the report.
- RACE: RACE compares each target report with a high-quality reference to improve score discrimination across generated reports.
- RACE: RACE combines weighted criterion scores into dimension scores, then weights those scores to produce an intermediate overall score and a reference-relative final score.
- FACT: FACT extracts and deduplicates statement–URL pairs, checks webpage support with a Judge LLM, and computes Citation Accuracy and Average Effective Citations per Task.
4 Experiments
Experiments evaluate DeepResearch Bench with RACE and FACT across specialized Deep Research Agents and LLMs with search tools, then test RACE’s alignment with human judgments and judge-model cost. Results show differentiated model capabilities, strong human consistency for full RACE, and a performance–cost trade-off among judge LLMs.
- Experimental Setup: DeepResearch Bench evaluates four early-released Deep Research Agents alongside LLMs with built-in web search tools, using standardized configurations.The evaluated DRA systems include Gemini-2.5-Pro-based Deep Research, OpenAI Deep Research, Grok Deeper Search, and Perplexity Deep Research.
- RACE Evaluation: Gemini-2.5-Pro Deep Research leads overall RACE performance, while OpenAI Deep Research surpasses it on Instruction-Following and both outperform the other two DRAs.Perplexity Deep Research slightly outperforms Grok Deeper Search, and the top agents’ scores are relatively close under reference-based relative scoring.
- RACE Evaluation: Claude-3.7-Sonnet with search exceeds Grok Deeper Search, possibly because it is allowed multi-turn web searches; Perplexity-Sonar-Reasoning-Pro(high) performs similarly.Across topics, individual models maintain relatively stable performance, while Chinese transportation tasks fall below models’ respective averages.
- FACT Evaluation: Gemini-2.5-Pro Deep Research averages 111.21 effective citations, while Perplexity Deep Research shows stronger citation accuracy than Gemini-2.5-Pro and OpenAI Deep Research.Except for Grok, DRAs tend to include more effective citations than LLMs with search tools; Claude-3.7-Sonnet with search has the second-highest effective-citation count among those models and strong citation accuracy.
- Human Consistency: RACE is evaluated against human judgments from reports generated for 50 Chinese tasks, with three domain-expert annotators scoring each task’s reports.More than 70 annotators with relevant expertise evaluate reports across four dimensions and overall performance, while four alignment metrics quantify consistency.
- Human Consistency: RACE(Full) significantly outperforms the vanilla prompt baseline and other variants, and its Pairwise Agreement Rate exceeds agreement among human experts.The results support reliable and accurate evaluation of deep research reports with high human consistency.
- Judge LLM Comparison: Gemini 2.5 Pro Preview achieves the best overall judge performance at a competitive average cost of $0.13 per query, higher only than o4-mini.The authors therefore select Gemini 2.5 Pro Preview as the final framework’s Judge LLM.
5 Related Work
Related work situates DeepResearch Bench within broader evaluation of LLM-based agents and the rapidly developing Deep Research Agent field. It identifies a remaining need for standardized evaluation that supports meaningful comparisons of DRA capabilities.
- LLM-based Agent Evaluation: Prior work evaluates LLM-based agents for real-world scenarios, emphasizing realistic task design and novel evaluation methods for practical systems.These evaluations target diverse agent capabilities as LLMs advance.
- Deep Research Agent: After OpenAI and Google released Deep Research Agents, related DRA frameworks quickly followed, but standardized evaluation methodology remained lacking.The absence of standardization prevents meaningful comparative analysis of DRA capabilities.
6 Conclusion
DeepResearch Bench is presented as a comprehensive benchmark for Deep Research Agents, covering report generation and web retrieval across 100 tasks in 22 domains. Its RACE and FACT frameworks show high consistency with human judgments and are intended to support more human-centric agent development.
- DeepResearch Bench evaluates both report generation and web retrieval capabilities of Deep Research Agents.
- 100 high-quality research tasks span 22 distinct domains and are curated to reflect authentic user needs.
- RACE and FACT demonstrate high consistency with human judgments, affirming their reliability.
- The benchmark is intended to guide development of more powerful and human-centric AI agent systems addressing genuine user requirements.
A Limitations
The benchmark and its evaluation frameworks have limitations involving scale, domain representation, human-evaluation throughput, and broader societal risks. The authors identify expansion, additional review, and larger annotation campaigns as future priorities while noting risks from misinformation, over-reliance, and bias.
- Benchmark scale: The 100-task benchmark is limited in scale because expert development, refinement, and screening require substantial effort.The authors state that expansion could improve statistical robustness and topic coverage.
- Domain coverage bias: Curation may introduce domain coverage bias despite multiple review layers and explicit acceptance criteria.Future iterations plan to add external reviewers with specialized domain knowledge for more balanced field representation.
- Human evaluation throughput: Human evaluation remains modest: three experts assessed 50 tasks across four agents, totaling 600 reports.Thorough review typically requires 30–60 minutes per report, and the sample remains susceptible to individual variability.
- Broader impact: Powerful Deep Research Agents may generate misinformation, encourage over-reliance, and amplify biases in models and web data.The authors frame FACT and RACE as steps toward more reliable, transparent, and responsible agents, alongside continued safety and ethics research.
B Definitions of Evaluation Dimensions in RACE
The paper describes RACE and FACT as complementary evaluation frameworks for report quality and information retrieval. FACT uses a Judge LLM for statement-URL processing, and Gemini-2.5-Flash shows strong agreement with human judgments while offering cost advantages over Gemini-2.5-Pro.
- RACE dimensions: RACE evaluates research reports across four top-level dimensions.Their definitions are provided in Table 4.
- FACT evaluation: FACT uses a Judge LLM for statement-URL pair extraction, deduplication, and support judgment.The Judge LLM selection balances evaluation accuracy with the token and operational costs of these steps.
- Judge LLM validation: Gemini-2.5-Flash aligned with human support judgments in 96% of cases and not-support judgments in 92%.
- Judge LLM selection: Gemini-2.5-Flash was selected for FACT because its accuracy was very close to Gemini-2.5-Pro at a more advantageous cost.This choice is intended to maintain evaluation reliability while managing operational costs.
D Data Collection Timeframes for Evaluated Models
The supplied passages identify model-output collection timeframes from 2025 and define citation-accuracy calculation components. Citation Accuracy averages per-task supported statement-URL proportions, assigning zero when a task has no citable statements.
- Data collection timeframes: Commercial-model outputs used in the experiments were collected during specified timeframes in 2025.Table 5 provides the model categories, provider groups, and date ranges.
- Citation metric inputs: Citation metrics are calculated from the set of benchmark tasks and unique statement-URL pairs remaining after deduplication.For each task, support judgments classify pairs as support or not support.
- Citation Accuracy: Citation Accuracy is the average across tasks of each task’s proportion of supported statement-URL pairs.The per-task accuracy is defined using supported pairs divided by unique pairs when the denominator is positive.
- Citation Accuracy: Tasks with no citable statements contribute an accuracy of 0 to the overall average.
E.2 Average Effective Citations per Task (E. Cit.)
Average Effective Citations per Task measures how much useful, relevant information an agent retrieves and correctly supports with evidence, averaged across benchmark tasks.
- Average Effective Citations per Task divides total support statement-URL pairs by the benchmark’s total number of tasks.The metric captures the average quantity of useful, relevant, and correctly evidenced information retrieved per task.
- E. Cit. evaluates both information usefulness and relevance and whether the agent correctly supports that information with evidence.
- The metric provides a direct measure of average verifiably supported statements generated per task.
F Detailed Calculation of Human Consistency Metrics
The human-consistency analysis compares automated evaluation with expert preferences, scores, and rankings, filtering unreliable tasks before computing per-task correlations.
- Pairwise Agreement Rate: Pairwise Agreement Rate measures how often automated evaluation preferences match human preferences across report pairs and tasks.Human preferences are established from experts’ average overall scores for each report pair.
- Overall Pearson Correlation: Overall Pearson Correlation measures the linear relationship between method-derived and human-derived average model scores aggregated across 50 tasks.The correlation is computed between vectors of average scores for the evaluated models.
- Filtered Correlations: Tasks with ICC < 0 are excluded because negative intraclass correlation indicates poor inter-rater reliability, leaving 37 tasks in the experiments.ICC(1,1) is computed from scores assigned by 3 experts to 4 reports per task.
- Filtered Correlations: Filtered Average Pearson Correlation averages per-task Pearson correlations between method scores and mean human scores over the reliable-task subset.The filtering is intended to provide a more robust assessment of absolute-score correlation.
- Filtered Correlations: Filtered Average Spearman Correlation averages per-task rank correlations to assess whether automated evaluation preserves human model ordering on consistent tasks.
G.1 Human Evaluation Effort
Human evaluation required substantial expert time, while standardized model settings were used to reduce configuration-driven variability and support fair comparison.
- Human Evaluation Effort: Each expert annotator spent approximately 1.5 hours per query evaluating reports from four agents, totaling 225 person-hours across all tasks and annotators.The resulting human judgments served as the basis for analyzing human consistency.
- Standardized Evaluation Settings: Models with configurable reasoning resources received a high setting equivalent to 16,000 tokens where applicable.This provided ample processing capacity for complex queries.
- Standardized Evaluation Settings: The maximum number of permitted web-search turns was standardized to five for models exposing that configuration.The setting was intended to make web-exploration depth comparable across models.
- Standardized Evaluation Settings: Maximum output length was set to 36,000 tokens, unless a model’s native maximum was lower.
- Standardized Evaluation Settings: Citation presentation was standardized by parsing citations according to official APIs and inserting markers such as [1][2] for downstream factual evaluation.The uniform citation structure supported equitable and rigorous verification.
- Standardized Evaluation Settings: These settings were applied to minimize variability from differing defaults and enable more direct comparison of model capabilities.